[2017-05-04 10:40:02,560: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-05-04 10:40:02,560: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-05-04 10:40:02,560: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-05-04 10:40:02,560: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-05-04 10:40:02,560: INFO/job_scheduler] Stopping...
[2017-05-04 10:40:02,561: INFO/queue] Stopping ServiceQueue job_scheduler_notifications
[2017-05-04 10:40:02,561: INFO/job_scheduler] Joining...
[2017-05-04 10:40:03,365: INFO/job_scheduler] Complete.
[2017-05-04 10:40:03,366: INFO/job_scheduler] Cancelling outstanding jobs...
[2017-05-04 10:40:03,365: INFO/chroma_service] Joining job_scheduler
[2017-05-04 10:41:47,967: INFO/queue] Purged 0 messages from 'job_scheduler_notifications' queue
[2017-05-04 10:41:48,021: INFO/queue] Purged 0 messages from 'agent_action_runner_rx' queue
[2017-05-04 10:45:42,397: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 10:45:42,398: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-04 10:45:42,415: INFO/command_plan] add_jobs: created Job 1 (Test for host connectivity)
[2017-05-04 10:45:42,453: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-04 10:45:42,461: INFO/job_scheduler] Job 1: RunJobThread.run
[2017-05-04 10:45:42,502: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 10:45:42,510: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-04 10:45:42,523: INFO/command_plan] add_jobs: created Job 2 (Test for host connectivity)
[2017-05-04 10:45:42,567: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 10:45:42,568: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-04 10:45:42,582: INFO/command_plan] add_jobs: created Job 3 (Test for host connectivity)
[2017-05-04 10:45:42,590: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-05-04 10:45:42,596: INFO/job_scheduler] Job 2: RunJobThread.run
[2017-05-04 10:45:42,597: INFO/job_scheduler] Job 3: RunJobThread.run
[2017-05-04 10:45:42,629: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-04 10:45:42,678: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 10:45:42,678: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-04 10:45:42,725: INFO/command_plan] add_jobs: created Job 4 (Test for host connectivity)
[2017-05-04 10:45:42,782: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-05-04 10:45:42,784: INFO/job_scheduler] Job 4: RunJobThread.run
[2017-05-04 10:45:46,065: INFO/job_scheduler] Job 2 finished 1 steps successfully
[2017-05-04 10:45:46,131: INFO/job_scheduler] Job 2: complete_job: Updating cache
[2017-05-04 10:45:46,131: INFO/job_scheduler] Job 2 completing (errored=False, cancelled=False)
[2017-05-04 10:45:46,164: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-04 10:45:46,194: INFO/job_scheduler] Job 3 finished 1 steps successfully
[2017-05-04 10:45:46,206: INFO/job_scheduler] Job 3: complete_job: Updating cache
[2017-05-04 10:45:46,206: INFO/job_scheduler] Job 3 completing (errored=False, cancelled=False)
[2017-05-04 10:45:46,239: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-05-04 10:45:46,289: INFO/job_scheduler] Job 4 finished 1 steps successfully
[2017-05-04 10:45:46,314: INFO/job_scheduler] Job 4: complete_job: Updating cache
[2017-05-04 10:45:46,314: INFO/job_scheduler] Job 4 completing (errored=False, cancelled=False)
[2017-05-04 10:45:46,342: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-04 10:45:48,013: INFO/job_scheduler] Job 1 finished 1 steps successfully
[2017-05-04 10:45:48,027: INFO/job_scheduler] Job 1: complete_job: Updating cache
[2017-05-04 10:45:48,027: INFO/job_scheduler] Job 1 completing (errored=False, cancelled=False)
[2017-05-04 10:45:48,060: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 10:45:51,881: INFO/job_scheduler] Getting FQDN for 'lotus-32vm5': lotus-32vm5
lotus-32vm5.lotus.hpdd.lab.intel.com

[2017-05-04 10:45:51,907: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 10:45:51,911: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-1 transitioning from undeployed to unconfigured
[2017-05-04 10:45:51,929: INFO/command_plan] Created command 5 (Deploying agent to host lotus-32vm5.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-05-04 10:45:51,981: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-04 10:45:52,010: INFO/job_scheduler] Job 5: RunJobThread.run
[2017-05-04 10:45:52,414: INFO/job_scheduler] Getting FQDN for 'lotus-32vm6.lotus.hpdd.lab.intel.com': lotus-32vm6
lotus-32vm6.lotus.hpdd.lab.intel.com

[2017-05-04 10:45:52,441: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 10:45:52,445: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-2 transitioning from undeployed to unconfigured
[2017-05-04 10:45:52,466: INFO/command_plan] Created command 6 (Deploying agent to host lotus-32vm6.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-05-04 10:45:52,477: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-05-04 10:45:52,502: INFO/job_scheduler] Job 6: RunJobThread.run
[2017-05-04 10:45:52,947: INFO/job_scheduler] Getting FQDN for 'lotus-32vm7.lotus.hpdd.lab.intel.com': lotus-32vm7
lotus-32vm7.lotus.hpdd.lab.intel.com

[2017-05-04 10:45:52,974: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 10:45:52,976: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-3 transitioning from undeployed to unconfigured
[2017-05-04 10:45:52,993: INFO/command_plan] Created command 7 (Deploying agent to host lotus-32vm7.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-05-04 10:45:53,013: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 10:45:53,035: INFO/job_scheduler] Job 7: RunJobThread.run
[2017-05-04 10:45:53,042: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-04 10:45:53,079: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-04 10:45:53,526: INFO/job_scheduler] Getting FQDN for 'lotus-32vm8.lotus.hpdd.lab.intel.com': lotus-32vm8
lotus-32vm8.lotus.hpdd.lab.intel.com

[2017-05-04 10:45:53,549: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 10:45:53,553: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-4 transitioning from undeployed to unconfigured
[2017-05-04 10:45:53,570: INFO/command_plan] Created command 8 (Deploying agent to host lotus-32vm8.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-05-04 10:45:53,618: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-05-04 10:45:53,652: INFO/job_scheduler] Job 8: RunJobThread.run
[2017-05-04 10:45:53,659: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-04 10:45:54,448: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-04 10:45:54,469: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:45:54,470: INFO/job_scheduler]   Job 6 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-04 10:45:54,470: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-04 10:45:54,507: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-04 10:45:54,507: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-04 10:45:55,042: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:45:55,042: INFO/job_scheduler]   Job 7 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-04 10:45:55,042: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-04 10:45:55,082: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-04 10:45:55,082: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-04 10:45:55,508: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 359 seconds remain
[2017-05-04 10:45:55,508: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 359 seconds remain
[2017-05-04 10:45:55,815: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:45:55,815: INFO/job_scheduler]   Job 8 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-04 10:45:55,815: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-04 10:45:55,853: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-04 10:45:55,853: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-04 10:45:56,083: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 359 seconds remain
[2017-05-04 10:45:56,083: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 359 seconds remain
[2017-05-04 10:45:56,510: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 358 seconds remain
[2017-05-04 10:45:56,510: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 358 seconds remain
[2017-05-04 10:45:56,854: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 359 seconds remain
[2017-05-04 10:45:56,854: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 359 seconds remain
[2017-05-04 10:45:57,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 358 seconds remain
[2017-05-04 10:45:57,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 358 seconds remain
[2017-05-04 10:45:57,511: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 357 seconds remain
[2017-05-04 10:45:57,511: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 357 seconds remain
[2017-05-04 10:45:57,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 358 seconds remain
[2017-05-04 10:45:57,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 358 seconds remain
[2017-05-04 10:45:58,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 357 seconds remain
[2017-05-04 10:45:58,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 357 seconds remain
[2017-05-04 10:45:58,407: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:45:58,407: INFO/job_scheduler]   Job 5 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-04 10:45:58,408: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-04 10:45:58,512: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 356 seconds remain
[2017-05-04 10:45:58,512: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 356 seconds remain
[2017-05-04 10:45:58,517: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-04 10:45:58,517: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-04 10:45:58,857: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 357 seconds remain
[2017-05-04 10:45:58,857: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 357 seconds remain
[2017-05-04 10:45:59,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 356 seconds remain
[2017-05-04 10:45:59,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 356 seconds remain
[2017-05-04 10:45:59,513: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 355 seconds remain
[2017-05-04 10:45:59,513: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 355 seconds remain
[2017-05-04 10:45:59,518: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 359 seconds remain
[2017-05-04 10:45:59,518: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 359 seconds remain
[2017-05-04 10:45:59,858: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 356 seconds remain
[2017-05-04 10:45:59,858: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 356 seconds remain
[2017-05-04 10:46:00,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 355 seconds remain
[2017-05-04 10:46:00,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 355 seconds remain
[2017-05-04 10:46:00,515: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 354 seconds remain
[2017-05-04 10:46:00,515: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 354 seconds remain
[2017-05-04 10:46:00,519: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 358 seconds remain
[2017-05-04 10:46:00,519: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 358 seconds remain
[2017-05-04 10:46:00,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 355 seconds remain
[2017-05-04 10:46:00,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 355 seconds remain
[2017-05-04 10:46:01,089: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 354 seconds remain
[2017-05-04 10:46:01,089: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 354 seconds remain
[2017-05-04 10:46:01,516: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 353 seconds remain
[2017-05-04 10:46:01,516: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 353 seconds remain
[2017-05-04 10:46:01,521: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 357 seconds remain
[2017-05-04 10:46:01,521: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 357 seconds remain
[2017-05-04 10:46:01,861: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 354 seconds remain
[2017-05-04 10:46:01,861: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 354 seconds remain
[2017-05-04 10:46:02,090: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 353 seconds remain
[2017-05-04 10:46:02,090: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 353 seconds remain
[2017-05-04 10:46:02,518: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 352 seconds remain
[2017-05-04 10:46:02,518: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 352 seconds remain
[2017-05-04 10:46:02,521: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 356 seconds remain
[2017-05-04 10:46:02,521: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 356 seconds remain
[2017-05-04 10:46:02,862: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 353 seconds remain
[2017-05-04 10:46:02,862: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 353 seconds remain
[2017-05-04 10:46:03,092: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 352 seconds remain
[2017-05-04 10:46:03,092: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 352 seconds remain
[2017-05-04 10:46:03,519: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 351 seconds remain
[2017-05-04 10:46:03,519: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 351 seconds remain
[2017-05-04 10:46:03,522: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 355 seconds remain
[2017-05-04 10:46:03,522: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 355 seconds remain
[2017-05-04 10:46:03,864: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 352 seconds remain
[2017-05-04 10:46:03,864: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 352 seconds remain
[2017-05-04 10:46:04,093: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 351 seconds remain
[2017-05-04 10:46:04,093: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 351 seconds remain
[2017-05-04 10:46:04,520: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 350 seconds remain
[2017-05-04 10:46:04,520: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 350 seconds remain
[2017-05-04 10:46:04,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 354 seconds remain
[2017-05-04 10:46:04,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 354 seconds remain
[2017-05-04 10:46:04,647: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:46:04,647: INFO/job_scheduler]   Job 7 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-04 10:46:04,647: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-04 10:46:04,761: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:46:04,762: INFO/job_scheduler]   Job 7 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-04 10:46:04,762: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-04 10:46:04,865: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 351 seconds remain
[2017-05-04 10:46:04,865: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 351 seconds remain
[2017-05-04 10:46:04,987: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:46:04,987: INFO/job_scheduler]   Job 6 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-04 10:46:04,988: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-04 10:46:05,081: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:46:05,081: INFO/job_scheduler]   Job 6 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-04 10:46:05,082: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-04 10:46:05,093: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 350 seconds remain
[2017-05-04 10:46:05,093: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 350 seconds remain
[2017-05-04 10:46:05,335: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:46:05,335: INFO/job_scheduler]   Job 8 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-04 10:46:05,335: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-04 10:46:05,432: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:46:05,432: INFO/job_scheduler]   Job 8 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-04 10:46:05,432: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-04 10:46:05,522: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 349 seconds remain
[2017-05-04 10:46:05,522: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 349 seconds remain
[2017-05-04 10:46:05,524: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 353 seconds remain
[2017-05-04 10:46:05,524: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 353 seconds remain
[2017-05-04 10:46:05,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 350 seconds remain
[2017-05-04 10:46:05,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 350 seconds remain
[2017-05-04 10:46:06,095: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 349 seconds remain
[2017-05-04 10:46:06,095: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 349 seconds remain
[2017-05-04 10:46:06,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 348 seconds remain
[2017-05-04 10:46:06,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 348 seconds remain
[2017-05-04 10:46:06,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 352 seconds remain
[2017-05-04 10:46:06,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 352 seconds remain
[2017-05-04 10:46:06,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 349 seconds remain
[2017-05-04 10:46:06,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 349 seconds remain
[2017-05-04 10:46:06,890: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:46:06,891: INFO/job_scheduler]   Job 5 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-04 10:46:06,891: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-04 10:46:07,004: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:46:07,004: INFO/job_scheduler]   Job 5 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-04 10:46:07,004: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-04 10:46:07,095: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 348 seconds remain
[2017-05-04 10:46:07,095: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 348 seconds remain
[2017-05-04 10:46:07,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 347 seconds remain
[2017-05-04 10:46:07,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 347 seconds remain
[2017-05-04 10:46:07,527: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 351 seconds remain
[2017-05-04 10:46:07,527: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 351 seconds remain
[2017-05-04 10:46:07,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 348 seconds remain
[2017-05-04 10:46:07,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 348 seconds remain
[2017-05-04 10:46:08,097: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 347 seconds remain
[2017-05-04 10:46:08,097: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 347 seconds remain
[2017-05-04 10:46:08,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 346 seconds remain
[2017-05-04 10:46:08,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 346 seconds remain
[2017-05-04 10:46:08,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 350 seconds remain
[2017-05-04 10:46:08,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 350 seconds remain
[2017-05-04 10:46:08,870: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 347 seconds remain
[2017-05-04 10:46:08,870: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 347 seconds remain
[2017-05-04 10:46:09,098: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 346 seconds remain
[2017-05-04 10:46:09,098: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 346 seconds remain
[2017-05-04 10:46:09,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 345 seconds remain
[2017-05-04 10:46:09,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 345 seconds remain
[2017-05-04 10:46:09,529: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 349 seconds remain
[2017-05-04 10:46:09,529: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 349 seconds remain
[2017-05-04 10:46:09,871: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 346 seconds remain
[2017-05-04 10:46:09,871: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 346 seconds remain
[2017-05-04 10:46:10,099: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 345 seconds remain
[2017-05-04 10:46:10,099: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 345 seconds remain
[2017-05-04 10:46:10,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 344 seconds remain
[2017-05-04 10:46:10,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 344 seconds remain
[2017-05-04 10:46:10,530: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 348 seconds remain
[2017-05-04 10:46:10,530: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 348 seconds remain
[2017-05-04 10:46:10,873: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 345 seconds remain
[2017-05-04 10:46:10,873: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 345 seconds remain
[2017-05-04 10:46:11,100: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 344 seconds remain
[2017-05-04 10:46:11,100: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 344 seconds remain
[2017-05-04 10:46:11,530: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 343 seconds remain
[2017-05-04 10:46:11,530: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 343 seconds remain
[2017-05-04 10:46:11,531: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 347 seconds remain
[2017-05-04 10:46:11,531: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 347 seconds remain
[2017-05-04 10:46:11,874: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 344 seconds remain
[2017-05-04 10:46:11,874: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 344 seconds remain
[2017-05-04 10:46:12,101: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 343 seconds remain
[2017-05-04 10:46:12,101: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 343 seconds remain
[2017-05-04 10:46:12,531: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 342 seconds remain
[2017-05-04 10:46:12,531: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 342 seconds remain
[2017-05-04 10:46:12,532: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 346 seconds remain
[2017-05-04 10:46:12,532: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 346 seconds remain
[2017-05-04 10:46:12,876: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 343 seconds remain
[2017-05-04 10:46:12,876: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 343 seconds remain
[2017-05-04 10:46:13,103: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 342 seconds remain
[2017-05-04 10:46:13,103: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 342 seconds remain
[2017-05-04 10:46:13,533: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 341 seconds remain
[2017-05-04 10:46:13,533: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 341 seconds remain
[2017-05-04 10:46:13,533: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 345 seconds remain
[2017-05-04 10:46:13,533: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 345 seconds remain
[2017-05-04 10:46:13,876: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 342 seconds remain
[2017-05-04 10:46:13,876: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 342 seconds remain
[2017-05-04 10:46:14,104: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 341 seconds remain
[2017-05-04 10:46:14,104: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 341 seconds remain
[2017-05-04 10:46:14,534: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 344 seconds remain
[2017-05-04 10:46:14,534: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 344 seconds remain
[2017-05-04 10:46:14,534: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 340 seconds remain
[2017-05-04 10:46:14,534: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 340 seconds remain
[2017-05-04 10:46:14,877: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 341 seconds remain
[2017-05-04 10:46:14,877: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 341 seconds remain
[2017-05-04 10:46:15,106: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 340 seconds remain
[2017-05-04 10:46:15,106: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 340 seconds remain
[2017-05-04 10:46:15,535: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 339 seconds remain
[2017-05-04 10:46:15,535: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 339 seconds remain
[2017-05-04 10:46:15,535: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 343 seconds remain
[2017-05-04 10:46:15,535: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 343 seconds remain
[2017-05-04 10:46:15,879: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 340 seconds remain
[2017-05-04 10:46:15,879: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 340 seconds remain
[2017-05-04 10:46:16,107: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 339 seconds remain
[2017-05-04 10:46:16,107: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 339 seconds remain
[2017-05-04 10:46:16,536: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 342 seconds remain
[2017-05-04 10:46:16,536: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 342 seconds remain
[2017-05-04 10:46:16,537: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 338 seconds remain
[2017-05-04 10:46:16,537: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 338 seconds remain
[2017-05-04 10:46:16,880: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 339 seconds remain
[2017-05-04 10:46:16,880: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 339 seconds remain
[2017-05-04 10:46:17,109: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 338 seconds remain
[2017-05-04 10:46:17,109: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 338 seconds remain
[2017-05-04 10:46:17,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 341 seconds remain
[2017-05-04 10:46:17,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 341 seconds remain
[2017-05-04 10:46:17,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 337 seconds remain
[2017-05-04 10:46:17,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 337 seconds remain
[2017-05-04 10:46:17,880: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 338 seconds remain
[2017-05-04 10:46:17,880: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 338 seconds remain
[2017-05-04 10:46:18,110: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 337 seconds remain
[2017-05-04 10:46:18,110: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 337 seconds remain
[2017-05-04 10:46:18,540: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 340 seconds remain
[2017-05-04 10:46:18,540: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 340 seconds remain
[2017-05-04 10:46:18,540: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 336 seconds remain
[2017-05-04 10:46:18,540: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 336 seconds remain
[2017-05-04 10:46:18,882: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 337 seconds remain
[2017-05-04 10:46:18,882: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 337 seconds remain
[2017-05-04 10:46:19,110: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 336 seconds remain
[2017-05-04 10:46:19,110: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 336 seconds remain
[2017-05-04 10:46:19,541: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 339 seconds remain
[2017-05-04 10:46:19,541: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 339 seconds remain
[2017-05-04 10:46:19,541: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 335 seconds remain
[2017-05-04 10:46:19,541: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 335 seconds remain
[2017-05-04 10:46:19,883: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 336 seconds remain
[2017-05-04 10:46:19,883: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 336 seconds remain
[2017-05-04 10:46:20,112: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 335 seconds remain
[2017-05-04 10:46:20,112: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 335 seconds remain
[2017-05-04 10:46:20,544: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 334 seconds remain
[2017-05-04 10:46:20,544: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 334 seconds remain
[2017-05-04 10:46:20,544: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 338 seconds remain
[2017-05-04 10:46:20,544: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 338 seconds remain
[2017-05-04 10:46:20,884: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 335 seconds remain
[2017-05-04 10:46:20,884: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 335 seconds remain
[2017-05-04 10:46:21,113: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 334 seconds remain
[2017-05-04 10:46:21,113: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 334 seconds remain
[2017-05-04 10:46:21,546: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 333 seconds remain
[2017-05-04 10:46:21,546: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 333 seconds remain
[2017-05-04 10:46:21,546: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 337 seconds remain
[2017-05-04 10:46:21,546: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 337 seconds remain
[2017-05-04 10:46:21,886: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 334 seconds remain
[2017-05-04 10:46:21,886: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 334 seconds remain
[2017-05-04 10:46:22,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 333 seconds remain
[2017-05-04 10:46:22,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 333 seconds remain
[2017-05-04 10:46:22,547: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 332 seconds remain
[2017-05-04 10:46:22,547: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 332 seconds remain
[2017-05-04 10:46:22,547: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 336 seconds remain
[2017-05-04 10:46:22,547: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 336 seconds remain
[2017-05-04 10:46:22,887: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 333 seconds remain
[2017-05-04 10:46:22,887: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 333 seconds remain
[2017-05-04 10:46:23,116: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 332 seconds remain
[2017-05-04 10:46:23,116: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 332 seconds remain
[2017-05-04 10:46:23,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 331 seconds remain
[2017-05-04 10:46:23,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 331 seconds remain
[2017-05-04 10:46:23,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 335 seconds remain
[2017-05-04 10:46:23,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 335 seconds remain
[2017-05-04 10:46:23,889: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 332 seconds remain
[2017-05-04 10:46:23,889: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 332 seconds remain
[2017-05-04 10:46:24,117: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 331 seconds remain
[2017-05-04 10:46:24,117: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 331 seconds remain
[2017-05-04 10:46:24,551: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 330 seconds remain
[2017-05-04 10:46:24,551: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 330 seconds remain
[2017-05-04 10:46:24,551: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 334 seconds remain
[2017-05-04 10:46:24,551: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 334 seconds remain
[2017-05-04 10:46:24,890: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 331 seconds remain
[2017-05-04 10:46:24,890: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 331 seconds remain
[2017-05-04 10:46:25,119: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 330 seconds remain
[2017-05-04 10:46:25,119: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 330 seconds remain
[2017-05-04 10:46:25,552: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 333 seconds remain
[2017-05-04 10:46:25,552: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 333 seconds remain
[2017-05-04 10:46:25,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 329 seconds remain
[2017-05-04 10:46:25,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 329 seconds remain
[2017-05-04 10:46:25,891: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 330 seconds remain
[2017-05-04 10:46:25,891: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 330 seconds remain
[2017-05-04 10:46:26,120: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 329 seconds remain
[2017-05-04 10:46:26,120: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 329 seconds remain
[2017-05-04 10:46:26,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 328 seconds remain
[2017-05-04 10:46:26,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 328 seconds remain
[2017-05-04 10:46:26,554: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 332 seconds remain
[2017-05-04 10:46:26,554: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 332 seconds remain
[2017-05-04 10:46:26,892: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 329 seconds remain
[2017-05-04 10:46:26,892: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 329 seconds remain
[2017-05-04 10:46:27,122: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 328 seconds remain
[2017-05-04 10:46:27,122: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 328 seconds remain
[2017-05-04 10:46:27,555: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 327 seconds remain
[2017-05-04 10:46:27,555: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 327 seconds remain
[2017-05-04 10:46:27,555: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 331 seconds remain
[2017-05-04 10:46:27,555: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 331 seconds remain
[2017-05-04 10:46:27,893: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 328 seconds remain
[2017-05-04 10:46:27,893: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 328 seconds remain
[2017-05-04 10:46:28,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 327 seconds remain
[2017-05-04 10:46:28,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 327 seconds remain
[2017-05-04 10:46:28,556: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 326 seconds remain
[2017-05-04 10:46:28,556: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 326 seconds remain
[2017-05-04 10:46:28,557: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 330 seconds remain
[2017-05-04 10:46:28,557: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 330 seconds remain
[2017-05-04 10:46:28,895: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 327 seconds remain
[2017-05-04 10:46:28,895: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 327 seconds remain
[2017-05-04 10:46:29,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 326 seconds remain
[2017-05-04 10:46:29,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 326 seconds remain
[2017-05-04 10:46:29,558: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 325 seconds remain
[2017-05-04 10:46:29,558: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 325 seconds remain
[2017-05-04 10:46:29,558: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 329 seconds remain
[2017-05-04 10:46:29,558: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 329 seconds remain
[2017-05-04 10:46:29,896: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 326 seconds remain
[2017-05-04 10:46:29,896: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 326 seconds remain
[2017-05-04 10:46:30,126: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 325 seconds remain
[2017-05-04 10:46:30,126: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 325 seconds remain
[2017-05-04 10:46:30,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 324 seconds remain
[2017-05-04 10:46:30,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 324 seconds remain
[2017-05-04 10:46:30,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 328 seconds remain
[2017-05-04 10:46:30,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 328 seconds remain
[2017-05-04 10:46:30,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 325 seconds remain
[2017-05-04 10:46:30,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 325 seconds remain
[2017-05-04 10:46:31,127: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 324 seconds remain
[2017-05-04 10:46:31,127: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 324 seconds remain
[2017-05-04 10:46:31,560: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 323 seconds remain
[2017-05-04 10:46:31,560: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 323 seconds remain
[2017-05-04 10:46:31,561: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 327 seconds remain
[2017-05-04 10:46:31,561: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 327 seconds remain
[2017-05-04 10:46:31,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 324 seconds remain
[2017-05-04 10:46:31,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 324 seconds remain
[2017-05-04 10:46:32,128: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 323 seconds remain
[2017-05-04 10:46:32,128: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 323 seconds remain
[2017-05-04 10:46:32,562: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 322 seconds remain
[2017-05-04 10:46:32,562: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 322 seconds remain
[2017-05-04 10:46:32,562: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 326 seconds remain
[2017-05-04 10:46:32,562: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 326 seconds remain
[2017-05-04 10:46:32,919: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 323 seconds remain
[2017-05-04 10:46:32,919: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 323 seconds remain
[2017-05-04 10:46:33,130: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 322 seconds remain
[2017-05-04 10:46:33,130: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 322 seconds remain
[2017-05-04 10:46:33,564: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 321 seconds remain
[2017-05-04 10:46:33,564: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 325 seconds remain
[2017-05-04 10:46:33,564: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 321 seconds remain
[2017-05-04 10:46:33,564: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 325 seconds remain
[2017-05-04 10:46:33,919: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 322 seconds remain
[2017-05-04 10:46:33,919: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 322 seconds remain
[2017-05-04 10:46:34,131: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 321 seconds remain
[2017-05-04 10:46:34,131: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 321 seconds remain
[2017-05-04 10:46:34,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 320 seconds remain
[2017-05-04 10:46:34,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 320 seconds remain
[2017-05-04 10:46:34,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 324 seconds remain
[2017-05-04 10:46:34,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 324 seconds remain
[2017-05-04 10:46:34,921: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 321 seconds remain
[2017-05-04 10:46:34,921: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 321 seconds remain
[2017-05-04 10:46:35,132: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 320 seconds remain
[2017-05-04 10:46:35,132: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 320 seconds remain
[2017-05-04 10:46:35,567: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 319 seconds remain
[2017-05-04 10:46:35,567: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 319 seconds remain
[2017-05-04 10:46:35,568: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 323 seconds remain
[2017-05-04 10:46:35,568: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 323 seconds remain
[2017-05-04 10:46:35,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 320 seconds remain
[2017-05-04 10:46:35,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 320 seconds remain
[2017-05-04 10:46:36,133: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 319 seconds remain
[2017-05-04 10:46:36,133: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 319 seconds remain
[2017-05-04 10:46:36,569: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 318 seconds remain
[2017-05-04 10:46:36,569: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 322 seconds remain
[2017-05-04 10:46:36,569: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 318 seconds remain
[2017-05-04 10:46:36,569: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 322 seconds remain
[2017-05-04 10:46:36,924: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 319 seconds remain
[2017-05-04 10:46:36,924: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 319 seconds remain
[2017-05-04 10:46:37,135: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 318 seconds remain
[2017-05-04 10:46:37,135: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 318 seconds remain
[2017-05-04 10:46:37,570: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 321 seconds remain
[2017-05-04 10:46:37,570: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 321 seconds remain
[2017-05-04 10:46:37,571: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 317 seconds remain
[2017-05-04 10:46:37,571: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 317 seconds remain
[2017-05-04 10:46:37,935: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 318 seconds remain
[2017-05-04 10:46:37,935: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 318 seconds remain
[2017-05-04 10:46:38,136: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 317 seconds remain
[2017-05-04 10:46:38,136: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 317 seconds remain
[2017-05-04 10:46:38,571: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 316 seconds remain
[2017-05-04 10:46:38,571: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 316 seconds remain
[2017-05-04 10:46:38,571: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 320 seconds remain
[2017-05-04 10:46:38,571: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 320 seconds remain
[2017-05-04 10:46:38,936: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 317 seconds remain
[2017-05-04 10:46:38,936: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 317 seconds remain
[2017-05-04 10:46:39,138: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 316 seconds remain
[2017-05-04 10:46:39,138: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 316 seconds remain
[2017-05-04 10:46:39,573: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 315 seconds remain
[2017-05-04 10:46:39,573: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 315 seconds remain
[2017-05-04 10:46:39,573: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 319 seconds remain
[2017-05-04 10:46:39,573: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 319 seconds remain
[2017-05-04 10:46:39,938: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 316 seconds remain
[2017-05-04 10:46:39,938: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 316 seconds remain
[2017-05-04 10:46:40,139: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 315 seconds remain
[2017-05-04 10:46:40,139: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 315 seconds remain
[2017-05-04 10:46:40,574: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 314 seconds remain
[2017-05-04 10:46:40,574: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 314 seconds remain
[2017-05-04 10:46:40,574: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 318 seconds remain
[2017-05-04 10:46:40,574: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 318 seconds remain
[2017-05-04 10:46:40,940: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 315 seconds remain
[2017-05-04 10:46:40,940: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 315 seconds remain
[2017-05-04 10:46:41,140: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 314 seconds remain
[2017-05-04 10:46:41,140: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 314 seconds remain
[2017-05-04 10:46:41,575: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 313 seconds remain
[2017-05-04 10:46:41,575: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 317 seconds remain
[2017-05-04 10:46:41,575: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 313 seconds remain
[2017-05-04 10:46:41,575: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 317 seconds remain
[2017-05-04 10:46:41,942: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 314 seconds remain
[2017-05-04 10:46:41,942: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 314 seconds remain
[2017-05-04 10:46:42,141: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 313 seconds remain
[2017-05-04 10:46:42,141: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 313 seconds remain
[2017-05-04 10:46:42,576: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 316 seconds remain
[2017-05-04 10:46:42,576: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 316 seconds remain
[2017-05-04 10:46:42,578: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 312 seconds remain
[2017-05-04 10:46:42,578: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 312 seconds remain
[2017-05-04 10:46:42,948: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 313 seconds remain
[2017-05-04 10:46:42,948: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 313 seconds remain
[2017-05-04 10:46:43,142: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 312 seconds remain
[2017-05-04 10:46:43,142: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 312 seconds remain
[2017-05-04 10:46:43,577: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 315 seconds remain
[2017-05-04 10:46:43,577: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 315 seconds remain
[2017-05-04 10:46:43,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 311 seconds remain
[2017-05-04 10:46:43,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 311 seconds remain
[2017-05-04 10:46:43,950: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 312 seconds remain
[2017-05-04 10:46:43,950: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 312 seconds remain
[2017-05-04 10:46:44,144: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 311 seconds remain
[2017-05-04 10:46:44,144: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 311 seconds remain
[2017-05-04 10:46:44,579: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 314 seconds remain
[2017-05-04 10:46:44,579: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 314 seconds remain
[2017-05-04 10:46:44,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 310 seconds remain
[2017-05-04 10:46:44,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 310 seconds remain
[2017-05-04 10:46:44,951: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 311 seconds remain
[2017-05-04 10:46:44,951: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 311 seconds remain
[2017-05-04 10:46:45,145: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 310 seconds remain
[2017-05-04 10:46:45,145: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 310 seconds remain
[2017-05-04 10:46:45,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 313 seconds remain
[2017-05-04 10:46:45,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 313 seconds remain
[2017-05-04 10:46:45,582: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 309 seconds remain
[2017-05-04 10:46:45,582: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 309 seconds remain
[2017-05-04 10:46:45,952: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 310 seconds remain
[2017-05-04 10:46:45,952: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 310 seconds remain
[2017-05-04 10:46:46,146: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 309 seconds remain
[2017-05-04 10:46:46,146: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 309 seconds remain
[2017-05-04 10:46:46,581: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 312 seconds remain
[2017-05-04 10:46:46,581: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 312 seconds remain
[2017-05-04 10:46:46,583: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 308 seconds remain
[2017-05-04 10:46:46,583: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 308 seconds remain
[2017-05-04 10:46:46,954: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 309 seconds remain
[2017-05-04 10:46:46,954: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 309 seconds remain
[2017-05-04 10:46:47,148: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 308 seconds remain
[2017-05-04 10:46:47,148: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 308 seconds remain
[2017-05-04 10:46:47,583: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 311 seconds remain
[2017-05-04 10:46:47,583: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 311 seconds remain
[2017-05-04 10:46:47,584: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 307 seconds remain
[2017-05-04 10:46:47,584: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 307 seconds remain
[2017-05-04 10:46:48,064: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 308 seconds remain
[2017-05-04 10:46:48,064: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 308 seconds remain
[2017-05-04 10:46:48,148: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 307 seconds remain
[2017-05-04 10:46:48,148: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 307 seconds remain
[2017-05-04 10:46:48,540: WARNING/chroma_core.services.job_scheduler.agent_rpc] Unable to reset action_runner sessions
[2017-05-04 10:46:48,540: WARNING/chroma_core.services.job_scheduler.agent_rpc] Unable to reset action_runner sessions
[2017-05-04 10:46:48,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: None/None
[2017-05-04 10:46:48,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: None/None
[2017-05-04 10:46:48,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/3f887698-ee4e-40ba-a714-0bf922db9925
[2017-05-04 10:46:48,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/3f887698-ee4e-40ba-a714-0bf922db9925
[2017-05-04 10:46:48,554: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/e504a1be-d450-4293-91a6-bab4bc2762e2
[2017-05-04 10:46:48,554: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/e504a1be-d450-4293-91a6-bab4bc2762e2
[2017-05-04 10:46:48,554: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/839cc874-0126-46e3-8ee9-e2e3f3508ca5
[2017-05-04 10:46:48,554: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/839cc874-0126-46e3-8ee9-e2e3f3508ca5
[2017-05-04 10:46:48,555: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e87b6272-1037-46fa-a66b-9f8af8ed61e4
[2017-05-04 10:46:48,555: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e87b6272-1037-46fa-a66b-9f8af8ed61e4
[2017-05-04 10:46:48,584: INFO/job_scheduler] Job 5 finished 1 steps successfully
[2017-05-04 10:46:48,585: INFO/job_scheduler] Job 6 finished 1 steps successfully
[2017-05-04 10:46:48,640: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 139993318334160
[2017-05-04 10:46:48,645: INFO/job] Job 5: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:48,646: INFO/job] Job 5: not_deleted=True
[2017-05-04 10:46:48,646: INFO/job_scheduler] Job 5: complete_job: Updating cache
[2017-05-04 10:46:48,646: INFO/job_scheduler] Job 5: locked_item 139993318334160 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-04 10:46:48,649: INFO/job_scheduler] Job 5 completing (errored=False, cancelled=False)
[2017-05-04 10:46:48,702: INFO/job_scheduler] Replaying 3 buffered notifications for ManagedHost-1
[2017-05-04 10:46:48,702: INFO/job_scheduler] _notify: Updating .boot_time of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com) from None to 2017-05-03 16:42:07+00:00
[2017-05-04 10:46:48,707: INFO/job_scheduler] _notify: Set boot_time=2017-05-03 16:42:07+00:00 on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-1) and saved
[2017-05-04 10:46:48,713: INFO/job_scheduler] _notify: Updating .properties of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-05-04 10:46:48,719: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-1) and saved
[2017-05-04 10:46:48,736: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-04 10:46:48,751: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 139993318330768
[2017-05-04 10:46:48,755: INFO/job] Job 6: StateChangeJob complete, setting state unconfigured on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:48,755: INFO/job] Job 6: not_deleted=True
[2017-05-04 10:46:48,756: INFO/job_scheduler] Job 6: complete_job: Updating cache
[2017-05-04 10:46:48,756: INFO/job_scheduler] Job 6: locked_item 139993318330768 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-04 10:46:48,758: INFO/job_scheduler] Job 6 completing (errored=False, cancelled=False)
[2017-05-04 10:46:48,807: INFO/job_scheduler] Replaying 3 buffered notifications for ManagedHost-2
[2017-05-04 10:46:48,807: INFO/job_scheduler] _notify: Updating .boot_time of item 2 (lotus-32vm6.lotus.hpdd.lab.intel.com) from None to 2017-04-25 07:09:19+00:00
[2017-05-04 10:46:48,812: INFO/job_scheduler] _notify: Set boot_time=2017-04-25 07:09:19+00:00 on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-2) and saved
[2017-05-04 10:46:48,818: INFO/job_scheduler] _notify: Updating .properties of item 2 (lotus-32vm6.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-05-04 10:46:48,824: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-2) and saved
[2017-05-04 10:46:48,837: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-05-04 10:46:49,065: INFO/job_scheduler] Job 8 finished 1 steps successfully
[2017-05-04 10:46:49,117: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 139993318467024
[2017-05-04 10:46:49,123: INFO/job] Job 8: StateChangeJob complete, setting state unconfigured on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:49,123: INFO/job] Job 8: not_deleted=True
[2017-05-04 10:46:49,123: INFO/job_scheduler] Job 8: complete_job: Updating cache
[2017-05-04 10:46:49,123: INFO/job_scheduler] Job 8: locked_item 139993318467024 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-04 10:46:49,126: INFO/job_scheduler] Job 8 completing (errored=False, cancelled=False)
[2017-05-04 10:46:49,150: INFO/job_scheduler] Job 7 finished 1 steps successfully
[2017-05-04 10:46:49,175: INFO/job_scheduler] Replaying 3 buffered notifications for ManagedHost-4
[2017-05-04 10:46:49,175: INFO/job_scheduler] _notify: Updating .boot_time of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com) from None to 2017-04-25 07:09:56+00:00
[2017-05-04 10:46:49,185: INFO/job_scheduler] _notify: Set boot_time=2017-04-25 07:09:56+00:00 on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-4) and saved
[2017-05-04 10:46:49,190: INFO/job_scheduler] _notify: Updating .properties of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-05-04 10:46:49,194: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-4) and saved
[2017-05-04 10:46:49,208: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-04 10:46:49,225: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 139993318359760
[2017-05-04 10:46:49,230: INFO/job] Job 7: StateChangeJob complete, setting state unconfigured on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:49,230: INFO/job] Job 7: not_deleted=True
[2017-05-04 10:46:49,230: INFO/job_scheduler] Job 7: complete_job: Updating cache
[2017-05-04 10:46:49,230: INFO/job_scheduler] Job 7: locked_item 139993318359760 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-04 10:46:49,233: INFO/job_scheduler] Job 7 completing (errored=False, cancelled=False)
[2017-05-04 10:46:49,277: INFO/job_scheduler] Replaying 3 buffered notifications for ManagedHost-3
[2017-05-04 10:46:49,277: INFO/job_scheduler] _notify: Updating .boot_time of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com) from None to 2017-04-25 07:09:31+00:00
[2017-05-04 10:46:49,282: INFO/job_scheduler] _notify: Set boot_time=2017-04-25 07:09:31+00:00 on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-3) and saved
[2017-05-04 10:46:49,287: INFO/job_scheduler] _notify: Updating .properties of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-05-04 10:46:49,292: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-3) and saved
[2017-05-04 10:46:49,306: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 10:46:54,150: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 10:46:54,157: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-1 transitioning from unconfigured to managed
[2017-05-04 10:46:54,294: INFO/command_plan] Created command 9 (Setup managed host lotus-32vm5.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-05-04 10:46:54,319: INFO/job_scheduler] run_next: 1 runnable jobs of (11 pending, 0 tasked)
[2017-05-04 10:46:54,339: INFO/job_scheduler] Job 9: RunJobThread.run
[2017-05-04 10:46:54,340: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-05-04 10:46:54,353: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc aac3a269-ad2f-4e1e-90c0-858a39b00a22
[2017-05-04 10:46:54,353: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc aac3a269-ad2f-4e1e-90c0-858a39b00a22
[2017-05-04 10:46:54,423: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 10:46:54,428: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-2 transitioning from unconfigured to managed
[2017-05-04 10:46:54,542: INFO/command_plan] Created command 10 (Setup managed host lotus-32vm6.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-05-04 10:46:54,599: INFO/job_scheduler] run_next: 1 runnable jobs of (21 pending, 1 tasked)
[2017-05-04 10:46:54,617: INFO/job_scheduler] Job 20: RunJobThread.run
[2017-05-04 10:46:54,618: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-05-04 10:46:54,636: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3b869ae4-d17f-4bc9-8592-0a4f63a8f4dc
[2017-05-04 10:46:54,636: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3b869ae4-d17f-4bc9-8592-0a4f63a8f4dc
[2017-05-04 10:46:54,665: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 10:46:54,668: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-3 transitioning from unconfigured to managed
[2017-05-04 10:46:54,783: INFO/command_plan] Created command 11 (Setup managed host lotus-32vm7.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-05-04 10:46:54,893: INFO/job_scheduler] run_next: 1 runnable jobs of (31 pending, 2 tasked)
[2017-05-04 10:46:54,925: INFO/job_scheduler] Job 31: RunJobThread.run
[2017-05-04 10:46:54,927: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-05-04 10:46:54,945: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 50d2d1f5-0b05-4c76-b8c9-a2a87e85f668
[2017-05-04 10:46:54,945: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 50d2d1f5-0b05-4c76-b8c9-a2a87e85f668
[2017-05-04 10:46:54,958: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 10:46:54,970: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-4 transitioning from unconfigured to managed
[2017-05-04 10:46:55,081: INFO/command_plan] Created command 12 (Setup managed host lotus-32vm8.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-05-04 10:46:55,270: INFO/job_scheduler] run_next: 1 runnable jobs of (41 pending, 3 tasked)
[2017-05-04 10:46:55,288: INFO/job_scheduler] Job 42: RunJobThread.run
[2017-05-04 10:46:55,289: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-05-04 10:46:55,304: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 419f4d25-79a5-443b-96e4-2ad32bc77784
[2017-05-04 10:46:55,304: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 419f4d25-79a5-443b-96e4-2ad32bc77784
[2017-05-04 10:46:55,706: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/3f887698-ee4e-40ba-a714-0bf922db9925
[2017-05-04 10:46:55,706: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/3f887698-ee4e-40ba-a714-0bf922db9925
[2017-05-04 10:46:55,707: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/3f887698-ee4e-40ba-a714-0bf922db9925
[2017-05-04 10:46:55,707: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/3f887698-ee4e-40ba-a714-0bf922db9925
[2017-05-04 10:46:55,707: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3b869ae4-d17f-4bc9-8592-0a4f63a8f4dc
[2017-05-04 10:46:55,707: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3b869ae4-d17f-4bc9-8592-0a4f63a8f4dc
[2017-05-04 10:46:55,753: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3b869ae4-d17f-4bc9-8592-0a4f63a8f4dc
[2017-05-04 10:46:55,753: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3b869ae4-d17f-4bc9-8592-0a4f63a8f4dc
[2017-05-04 10:46:55,753: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-05-04 10:46:55.753368+00:00:
[2017-05-04 10:46:55,753: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-05-04 10:46:55.753368+00:00:
[2017-05-04 10:46:55,753: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-04 10:46:55,753: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-04 10:46:55,766: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:46:55,766: INFO/job_scheduler]   Job 20 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-04 10:46:55,767: INFO/job_scheduler]   Job 21 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:55,767: INFO/job_scheduler]   Job 22 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:55,767: INFO/job_scheduler]   Job 23 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:55,767: INFO/job_scheduler]   Job 24 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:55,767: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:55,767: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:46:55,768: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-04 10:46:55,769: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-05-04 10:46:55.769368+00:00:
[2017-05-04 10:46:55,769: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-05-04 10:46:55.769368+00:00:
[2017-05-04 10:46:55,769: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-04 10:46:55,769: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-04 10:46:55,785: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com restart_agent {}
[2017-05-04 10:46:55,793: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:46:55,793: INFO/job_scheduler]   Job 20 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-04 10:46:55,794: INFO/job_scheduler]   Job 21 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:55,794: INFO/job_scheduler]   Job 22 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:55,794: INFO/job_scheduler]   Job 23 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:55,794: INFO/job_scheduler]   Job 24 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:55,794: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:55,794: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:46:55,795: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-04 10:46:55,800: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8ef08cfd-f294-4812-ad95-d6b004f42df1
[2017-05-04 10:46:55,800: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8ef08cfd-f294-4812-ad95-d6b004f42df1
[2017-05-04 10:46:55,876: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/3f887698-ee4e-40ba-a714-0bf922db9925
[2017-05-04 10:46:55,876: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/3f887698-ee4e-40ba-a714-0bf922db9925
[2017-05-04 10:46:55,876: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/3f887698-ee4e-40ba-a714-0bf922db9925
[2017-05-04 10:46:55,876: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/3f887698-ee4e-40ba-a714-0bf922db9925
[2017-05-04 10:46:55,876: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8ef08cfd-f294-4812-ad95-d6b004f42df1
[2017-05-04 10:46:55,876: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8ef08cfd-f294-4812-ad95-d6b004f42df1
[2017-05-04 10:46:55,916: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8ef08cfd-f294-4812-ad95-d6b004f42df1
[2017-05-04 10:46:55,916: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8ef08cfd-f294-4812-ad95-d6b004f42df1
[2017-05-04 10:46:55,917: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm6.lotus.hpdd.lab.intel.com (old 3f887698-ee4e-40ba-a714-0bf922db9925)
[2017-05-04 10:46:55,917: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm6.lotus.hpdd.lab.intel.com (old 3f887698-ee4e-40ba-a714-0bf922db9925)
[2017-05-04 10:46:56,047: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/e504a1be-d450-4293-91a6-bab4bc2762e2
[2017-05-04 10:46:56,047: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/e504a1be-d450-4293-91a6-bab4bc2762e2
[2017-05-04 10:46:56,047: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/e504a1be-d450-4293-91a6-bab4bc2762e2
[2017-05-04 10:46:56,047: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/e504a1be-d450-4293-91a6-bab4bc2762e2
[2017-05-04 10:46:56,047: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 50d2d1f5-0b05-4c76-b8c9-a2a87e85f668
[2017-05-04 10:46:56,047: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 50d2d1f5-0b05-4c76-b8c9-a2a87e85f668
[2017-05-04 10:46:56,059: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 50d2d1f5-0b05-4c76-b8c9-a2a87e85f668
[2017-05-04 10:46:56,059: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 50d2d1f5-0b05-4c76-b8c9-a2a87e85f668
[2017-05-04 10:46:56,060: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-05-04 10:46:56.060219+00:00:
[2017-05-04 10:46:56,060: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-05-04 10:46:56.060219+00:00:
[2017-05-04 10:46:56,060: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-04 10:46:56,060: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-04 10:46:56,072: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-05-04 10:46:56.072739+00:00:
[2017-05-04 10:46:56,072: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-05-04 10:46:56.072739+00:00:
[2017-05-04 10:46:56,073: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-04 10:46:56,074: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:46:56,073: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-04 10:46:56,074: INFO/job_scheduler]   Job 31 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-04 10:46:56,082: INFO/job_scheduler]   Job 32 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:56,082: INFO/job_scheduler]   Job 33 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:56,084: INFO/job_scheduler]   Job 34 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:56,084: INFO/job_scheduler]   Job 35 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:56,084: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:56,084: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:46:56,085: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-04 10:46:56,091: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:46:56,092: INFO/job_scheduler]   Job 31 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-04 10:46:56,092: INFO/job_scheduler]   Job 32 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:56,092: INFO/job_scheduler]   Job 33 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:56,095: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com restart_agent {}
[2017-05-04 10:46:56,092: INFO/job_scheduler]   Job 34 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:56,104: INFO/job_scheduler]   Job 35 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:56,104: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:56,104: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:46:56,104: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-04 10:46:56,110: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0246dc51-c662-4d98-9a42-71c3cc0bd316
[2017-05-04 10:46:56,110: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0246dc51-c662-4d98-9a42-71c3cc0bd316
[2017-05-04 10:46:56,170: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/e504a1be-d450-4293-91a6-bab4bc2762e2
[2017-05-04 10:46:56,170: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/e504a1be-d450-4293-91a6-bab4bc2762e2
[2017-05-04 10:46:56,171: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/e504a1be-d450-4293-91a6-bab4bc2762e2
[2017-05-04 10:46:56,171: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/e504a1be-d450-4293-91a6-bab4bc2762e2
[2017-05-04 10:46:56,171: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0246dc51-c662-4d98-9a42-71c3cc0bd316
[2017-05-04 10:46:56,171: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0246dc51-c662-4d98-9a42-71c3cc0bd316
[2017-05-04 10:46:56,174: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0246dc51-c662-4d98-9a42-71c3cc0bd316
[2017-05-04 10:46:56,174: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0246dc51-c662-4d98-9a42-71c3cc0bd316
[2017-05-04 10:46:56,174: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm7.lotus.hpdd.lab.intel.com (old e504a1be-d450-4293-91a6-bab4bc2762e2)
[2017-05-04 10:46:56,174: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm7.lotus.hpdd.lab.intel.com (old e504a1be-d450-4293-91a6-bab4bc2762e2)
[2017-05-04 10:46:56,391: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/839cc874-0126-46e3-8ee9-e2e3f3508ca5
[2017-05-04 10:46:56,391: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/839cc874-0126-46e3-8ee9-e2e3f3508ca5
[2017-05-04 10:46:56,392: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/839cc874-0126-46e3-8ee9-e2e3f3508ca5
[2017-05-04 10:46:56,392: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/839cc874-0126-46e3-8ee9-e2e3f3508ca5
[2017-05-04 10:46:56,392: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 419f4d25-79a5-443b-96e4-2ad32bc77784
[2017-05-04 10:46:56,392: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 419f4d25-79a5-443b-96e4-2ad32bc77784
[2017-05-04 10:46:56,418: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 419f4d25-79a5-443b-96e4-2ad32bc77784
[2017-05-04 10:46:56,418: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 419f4d25-79a5-443b-96e4-2ad32bc77784
[2017-05-04 10:46:56,419: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-05-04 10:46:56.419418+00:00:
[2017-05-04 10:46:56,419: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-05-04 10:46:56.419418+00:00:
[2017-05-04 10:46:56,419: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-04 10:46:56,419: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-04 10:46:56,433: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:46:56,433: INFO/job_scheduler]   Job 42 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-04 10:46:56,434: INFO/job_scheduler]   Job 43 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:56,434: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-05-04 10:46:56.434222+00:00:
[2017-05-04 10:46:56,434: INFO/job_scheduler]   Job 44 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:56,434: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-05-04 10:46:56.434222+00:00:
[2017-05-04 10:46:56,434: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-04 10:46:56,434: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-04 10:46:56,442: INFO/job_scheduler]   Job 45 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:56,442: INFO/job_scheduler]   Job 46 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:56,443: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:56,443: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:46:56,443: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-04 10:46:56,449: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:46:56,450: INFO/job_scheduler]   Job 42 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-04 10:46:56,450: INFO/job_scheduler]   Job 43 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:56,450: INFO/job_scheduler]   Job 44 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:56,451: INFO/job_scheduler]   Job 45 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:56,451: INFO/job_scheduler]   Job 46 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:56,451: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 10:46:56,451: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com restart_agent {}
[2017-05-04 10:46:56,451: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:46:56,460: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-04 10:46:56,470: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c6e0b6da-df17-4d50-94af-98c82cabac25
[2017-05-04 10:46:56,470: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c6e0b6da-df17-4d50-94af-98c82cabac25
[2017-05-04 10:46:56,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/839cc874-0126-46e3-8ee9-e2e3f3508ca5
[2017-05-04 10:46:56,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/839cc874-0126-46e3-8ee9-e2e3f3508ca5
[2017-05-04 10:46:56,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/839cc874-0126-46e3-8ee9-e2e3f3508ca5
[2017-05-04 10:46:56,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/839cc874-0126-46e3-8ee9-e2e3f3508ca5
[2017-05-04 10:46:56,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c6e0b6da-df17-4d50-94af-98c82cabac25
[2017-05-04 10:46:56,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c6e0b6da-df17-4d50-94af-98c82cabac25
[2017-05-04 10:46:56,584: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c6e0b6da-df17-4d50-94af-98c82cabac25
[2017-05-04 10:46:56,584: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c6e0b6da-df17-4d50-94af-98c82cabac25
[2017-05-04 10:46:56,584: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm8.lotus.hpdd.lab.intel.com (old 839cc874-0126-46e3-8ee9-e2e3f3508ca5)
[2017-05-04 10:46:56,584: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm8.lotus.hpdd.lab.intel.com (old 839cc874-0126-46e3-8ee9-e2e3f3508ca5)
[2017-05-04 10:47:25,457: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e87b6272-1037-46fa-a66b-9f8af8ed61e4
[2017-05-04 10:47:25,457: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e87b6272-1037-46fa-a66b-9f8af8ed61e4
[2017-05-04 10:47:25,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/e87b6272-1037-46fa-a66b-9f8af8ed61e4
[2017-05-04 10:47:25,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/e87b6272-1037-46fa-a66b-9f8af8ed61e4
[2017-05-04 10:47:25,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc aac3a269-ad2f-4e1e-90c0-858a39b00a22
[2017-05-04 10:47:25,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc aac3a269-ad2f-4e1e-90c0-858a39b00a22
[2017-05-04 10:47:25,473: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc aac3a269-ad2f-4e1e-90c0-858a39b00a22
[2017-05-04 10:47:25,473: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc aac3a269-ad2f-4e1e-90c0-858a39b00a22
[2017-05-04 10:47:25,473: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-04 10:47:25.473727+00:00:
[2017-05-04 10:47:25,473: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-04 10:47:25.473727+00:00:
[2017-05-04 10:47:25,474: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-04 10:47:25,474: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-04 10:47:25,502: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:47:25,502: INFO/job_scheduler]   Job 9 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-04 10:47:25,503: INFO/job_scheduler]   Job 10 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 10:47:25,503: INFO/job_scheduler]   Job 11 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 10:47:25,503: INFO/job_scheduler]   Job 12 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 10:47:25,503: INFO/job_scheduler]   Job 13 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 10:47:25,503: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 10:47:25,504: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:47:25,504: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-04 10:47:25,506: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-04 10:47:25.506752+00:00:
[2017-05-04 10:47:25,506: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-04 10:47:25.506752+00:00:
[2017-05-04 10:47:25,507: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-04 10:47:25,507: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-04 10:47:25,521: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:47:25,521: INFO/job_scheduler]   Job 9 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-04 10:47:25,521: INFO/job_scheduler]   Job 10 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 10:47:25,521: INFO/job_scheduler]   Job 11 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 10:47:25,521: INFO/job_scheduler]   Job 12 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 10:47:25,521: INFO/job_scheduler]   Job 13 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 10:47:25,522: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 10:47:25,522: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:47:25,522: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-04 10:47:25,524: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com restart_agent {}
[2017-05-04 10:47:25,536: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 012d4aba-c4e7-46c5-8d88-95d2ae671b98
[2017-05-04 10:47:25,536: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 012d4aba-c4e7-46c5-8d88-95d2ae671b98
[2017-05-04 10:47:25,636: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/3f887698-ee4e-40ba-a714-0bf922db9925
[2017-05-04 10:47:25,636: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/3f887698-ee4e-40ba-a714-0bf922db9925
[2017-05-04 10:47:25,636: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 3f887698-ee4e-40ba-a714-0bf922db9925 because session terminated
[2017-05-04 10:47:25,636: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 3f887698-ee4e-40ba-a714-0bf922db9925 because session terminated
[2017-05-04 10:47:25,639: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:47:25,639: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:47:25,655: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e87b6272-1037-46fa-a66b-9f8af8ed61e4
[2017-05-04 10:47:25,655: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e87b6272-1037-46fa-a66b-9f8af8ed61e4
[2017-05-04 10:47:25,656: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/e87b6272-1037-46fa-a66b-9f8af8ed61e4
[2017-05-04 10:47:25,656: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/e87b6272-1037-46fa-a66b-9f8af8ed61e4
[2017-05-04 10:47:25,656: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 012d4aba-c4e7-46c5-8d88-95d2ae671b98
[2017-05-04 10:47:25,656: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 012d4aba-c4e7-46c5-8d88-95d2ae671b98
[2017-05-04 10:47:25,700: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 012d4aba-c4e7-46c5-8d88-95d2ae671b98
[2017-05-04 10:47:25,700: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 012d4aba-c4e7-46c5-8d88-95d2ae671b98
[2017-05-04 10:47:25,701: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old e87b6272-1037-46fa-a66b-9f8af8ed61e4)
[2017-05-04 10:47:25,701: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old e87b6272-1037-46fa-a66b-9f8af8ed61e4)
[2017-05-04 10:47:25,946: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm6.lotus.hpdd.lab.intel.com new 33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:47:25,946: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm6.lotus.hpdd.lab.intel.com new 33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:47:25,984: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-04 10:47:26,000: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3cf57f60-a0aa-48f8-9e1f-e01c115f43ca
[2017-05-04 10:47:26,000: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3cf57f60-a0aa-48f8-9e1f-e01c115f43ca
[2017-05-04 10:47:26,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:47:26,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:47:26,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:47:26,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:47:26,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3cf57f60-a0aa-48f8-9e1f-e01c115f43ca
[2017-05-04 10:47:26,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3cf57f60-a0aa-48f8-9e1f-e01c115f43ca
[2017-05-04 10:47:26,109: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/e504a1be-d450-4293-91a6-bab4bc2762e2
[2017-05-04 10:47:26,109: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/e504a1be-d450-4293-91a6-bab4bc2762e2
[2017-05-04 10:47:26,109: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session e504a1be-d450-4293-91a6-bab4bc2762e2 because session terminated
[2017-05-04 10:47:26,109: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session e504a1be-d450-4293-91a6-bab4bc2762e2 because session terminated
[2017-05-04 10:47:26,112: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:47:26,112: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:47:26,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3cf57f60-a0aa-48f8-9e1f-e01c115f43ca
[2017-05-04 10:47:26,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3cf57f60-a0aa-48f8-9e1f-e01c115f43ca
[2017-05-04 10:47:26,116: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-05-04 10:47:26,132: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4c49f99e-dd48-44a1-8ff8-4649a193d9eb
[2017-05-04 10:47:26,132: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4c49f99e-dd48-44a1-8ff8-4649a193d9eb
[2017-05-04 10:47:26,204: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm7.lotus.hpdd.lab.intel.com new 936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:47:26,204: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm7.lotus.hpdd.lab.intel.com new 936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:47:26,205: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-04 10:47:26,218: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cf4ff2ba-87f1-4a7b-a598-75d31dd752af
[2017-05-04 10:47:26,218: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cf4ff2ba-87f1-4a7b-a598-75d31dd752af
[2017-05-04 10:47:26,609: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm8.lotus.hpdd.lab.intel.com timeout after 30.0s
[2017-05-04 10:47:26,609: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm8.lotus.hpdd.lab.intel.com timeout after 30.0s
[2017-05-04 10:47:26,615: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/839cc874-0126-46e3-8ee9-e2e3f3508ca5
[2017-05-04 10:47:26,615: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/839cc874-0126-46e3-8ee9-e2e3f3508ca5
[2017-05-04 10:47:26,616: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 839cc874-0126-46e3-8ee9-e2e3f3508ca5 because session terminated
[2017-05-04 10:47:26,616: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 839cc874-0126-46e3-8ee9-e2e3f3508ca5 because session terminated
[2017-05-04 10:47:26,618: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:47:26,618: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:47:27,424: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:47:27,424: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:47:27,424: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:47:27,424: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:47:27,424: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cf4ff2ba-87f1-4a7b-a598-75d31dd752af
[2017-05-04 10:47:27,424: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cf4ff2ba-87f1-4a7b-a598-75d31dd752af
[2017-05-04 10:47:27,433: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cf4ff2ba-87f1-4a7b-a598-75d31dd752af
[2017-05-04 10:47:27,433: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cf4ff2ba-87f1-4a7b-a598-75d31dd752af
[2017-05-04 10:47:27,433: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-05-04 10:47:27,452: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5f30423f-6b29-4140-96be-8f235570abf2
[2017-05-04 10:47:27,452: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5f30423f-6b29-4140-96be-8f235570abf2
[2017-05-04 10:47:27,611: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm8.lotus.hpdd.lab.intel.com new 81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:47:27,611: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm8.lotus.hpdd.lab.intel.com new 81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:47:27,612: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-04 10:47:27,626: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0e11979e-9aec-42e6-8ba4-90c6c52693d7
[2017-05-04 10:47:27,626: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0e11979e-9aec-42e6-8ba4-90c6c52693d7
[2017-05-04 10:47:27,723: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:47:27,723: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:47:27,724: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:47:27,724: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:47:27,724: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0e11979e-9aec-42e6-8ba4-90c6c52693d7
[2017-05-04 10:47:27,724: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0e11979e-9aec-42e6-8ba4-90c6c52693d7
[2017-05-04 10:47:27,743: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0e11979e-9aec-42e6-8ba4-90c6c52693d7
[2017-05-04 10:47:27,743: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0e11979e-9aec-42e6-8ba4-90c6c52693d7
[2017-05-04 10:47:27,744: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-05-04 10:47:27,762: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 779526ea-117c-4565-86bd-60a4f8bafad4
[2017-05-04 10:47:27,762: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 779526ea-117c-4565-86bd-60a4f8bafad4
[2017-05-04 10:47:35,505: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:47:35,505: INFO/job_scheduler]   Job 20 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-04 10:47:35,505: INFO/job_scheduler]   Job 21 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 10:47:35,505: INFO/job_scheduler]   Job 22 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 10:47:35,505: INFO/job_scheduler]   Job 23 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 10:47:35,505: INFO/job_scheduler]   Job 24 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 10:47:35,505: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 10:47:35,505: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:47:35,506: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-04 10:47:36,064: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:47:36,065: INFO/job_scheduler]   Job 31 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-04 10:47:36,065: INFO/job_scheduler]   Job 32 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 10:47:36,065: INFO/job_scheduler]   Job 33 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 10:47:36,065: INFO/job_scheduler]   Job 34 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 10:47:36,065: INFO/job_scheduler]   Job 35 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 10:47:36,065: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 10:47:36,065: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:47:36,066: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-04 10:47:36,490: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:47:36,490: INFO/job_scheduler]   Job 42 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-04 10:47:36,491: INFO/job_scheduler]   Job 43 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 10:47:36,491: INFO/job_scheduler]   Job 44 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 10:47:36,491: INFO/job_scheduler]   Job 45 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 10:47:36,491: INFO/job_scheduler]   Job 46 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 10:47:36,491: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 10:47:36,491: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:47:36,491: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-04 10:47:55,434: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e87b6272-1037-46fa-a66b-9f8af8ed61e4
[2017-05-04 10:47:55,434: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e87b6272-1037-46fa-a66b-9f8af8ed61e4
[2017-05-04 10:47:55,434: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session e87b6272-1037-46fa-a66b-9f8af8ed61e4 because session terminated
[2017-05-04 10:47:55,434: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session e87b6272-1037-46fa-a66b-9f8af8ed61e4 because session terminated
[2017-05-04 10:47:55,436: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:47:55,436: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:47:55,731: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new 532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:47:55,731: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new 532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:47:55,732: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-04 10:47:55,745: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f4289d77-6d27-4752-b0b4-e228c46b7512
[2017-05-04 10:47:55,745: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f4289d77-6d27-4752-b0b4-e228c46b7512
[2017-05-04 10:47:55,863: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:47:55,863: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:47:55,864: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:47:55,864: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:47:55,864: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f4289d77-6d27-4752-b0b4-e228c46b7512
[2017-05-04 10:47:55,864: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f4289d77-6d27-4752-b0b4-e228c46b7512
[2017-05-04 10:47:55,909: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f4289d77-6d27-4752-b0b4-e228c46b7512
[2017-05-04 10:47:55,909: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f4289d77-6d27-4752-b0b4-e228c46b7512
[2017-05-04 10:47:55,909: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-05-04 10:47:55,927: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ca15a2eb-b9d5-4322-95c4-187d805e61bf
[2017-05-04 10:47:55,927: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ca15a2eb-b9d5-4322-95c4-187d805e61bf
[2017-05-04 10:48:04,312: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:48:04,313: INFO/job_scheduler]   Job 9 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-04 10:48:04,313: INFO/job_scheduler]   Job 10 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 10:48:04,313: INFO/job_scheduler]   Job 11 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 10:48:04,313: INFO/job_scheduler]   Job 12 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 10:48:04,313: INFO/job_scheduler]   Job 13 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 10:48:04,313: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 10:48:04,313: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:48:04,314: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-04 10:48:12,132: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:12,132: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:12,133: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:12,133: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:12,133: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5f30423f-6b29-4140-96be-8f235570abf2
[2017-05-04 10:48:12,133: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5f30423f-6b29-4140-96be-8f235570abf2
[2017-05-04 10:48:12,177: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5f30423f-6b29-4140-96be-8f235570abf2
[2017-05-04 10:48:12,177: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5f30423f-6b29-4140-96be-8f235570abf2
[2017-05-04 10:48:12,554: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com install_packages {'repos': [u'e2fsprogs', u'iml-agent', u'lustre', u'zfs'], 'packages': [u'chroma-agent-management', u'kernel-devel-3.10.0-514.10.2.el7_lustre', u'lustre-osd-zfs', u'lustre-osd-ldiskfs', u'lustre-modules', u'lustre', u'zfs']}
[2017-05-04 10:48:12,571: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dfb1d201-bafa-42e6-807b-48e62ca419ca
[2017-05-04 10:48:12,571: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dfb1d201-bafa-42e6-807b-48e62ca419ca
[2017-05-04 10:48:13,402: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:13,402: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:13,402: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:13,402: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:13,402: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 779526ea-117c-4565-86bd-60a4f8bafad4
[2017-05-04 10:48:13,402: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 779526ea-117c-4565-86bd-60a4f8bafad4
[2017-05-04 10:48:13,439: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 779526ea-117c-4565-86bd-60a4f8bafad4
[2017-05-04 10:48:13,439: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 779526ea-117c-4565-86bd-60a4f8bafad4
[2017-05-04 10:48:13,872: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 10:48:13,873: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-04 10:48:13,905: INFO/command_plan] add_jobs: created Job 53 (Trigger plugin poll for linux plugins)
[2017-05-04 10:48:14,433: INFO/job_scheduler] run_next: 1 runnable jobs of (41 pending, 4 tasked)
[2017-05-04 10:48:14,436: INFO/job_scheduler] Job 53: RunJobThread.run
[2017-05-04 10:48:14,437: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-04 10:48:14,445: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-04 10:48:14,446: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-04 10:48:14,449: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com install_packages {'repos': [u'e2fsprogs', u'iml-agent', u'lustre', u'zfs'], 'packages': [u'chroma-agent-management', u'kernel-devel-3.10.0-514.10.2.el7_lustre', u'lustre-osd-zfs', u'lustre-osd-ldiskfs', u'lustre-modules', u'lustre', u'zfs']}
[2017-05-04 10:48:14,453: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bf32a6cd-4e91-4ee5-b3d9-6f155cdac356
[2017-05-04 10:48:14,453: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bf32a6cd-4e91-4ee5-b3d9-6f155cdac356
[2017-05-04 10:48:14,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6745f5dc-2569-4c41-a220-76510d55c35e
[2017-05-04 10:48:14,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6745f5dc-2569-4c41-a220-76510d55c35e
[2017-05-04 10:48:14,486: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 73582111-be51-471c-b740-0a142dbe25ed
[2017-05-04 10:48:14,486: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 73582111-be51-471c-b740-0a142dbe25ed
[2017-05-04 10:48:14,499: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f325d744-1542-4341-aec6-6e61308af9d9
[2017-05-04 10:48:14,499: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f325d744-1542-4341-aec6-6e61308af9d9
[2017-05-04 10:48:14,521: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:14,521: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:14,521: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:14,521: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:14,521: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6745f5dc-2569-4c41-a220-76510d55c35e
[2017-05-04 10:48:14,521: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6745f5dc-2569-4c41-a220-76510d55c35e
[2017-05-04 10:48:14,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:48:14,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:48:14,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:48:14,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:48:14,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bf32a6cd-4e91-4ee5-b3d9-6f155cdac356
[2017-05-04 10:48:14,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bf32a6cd-4e91-4ee5-b3d9-6f155cdac356
[2017-05-04 10:48:14,533: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bf32a6cd-4e91-4ee5-b3d9-6f155cdac356
[2017-05-04 10:48:14,533: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bf32a6cd-4e91-4ee5-b3d9-6f155cdac356
[2017-05-04 10:48:14,533: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-04 10:48:14,543: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6745f5dc-2569-4c41-a220-76510d55c35e
[2017-05-04 10:48:14,543: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6745f5dc-2569-4c41-a220-76510d55c35e
[2017-05-04 10:48:14,543: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-04 10:48:14,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:14,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:14,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:14,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:14,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f325d744-1542-4341-aec6-6e61308af9d9
[2017-05-04 10:48:14,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f325d744-1542-4341-aec6-6e61308af9d9
[2017-05-04 10:48:14,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f325d744-1542-4341-aec6-6e61308af9d9
[2017-05-04 10:48:14,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f325d744-1542-4341-aec6-6e61308af9d9
[2017-05-04 10:48:14,563: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-04 10:48:14,563: INFO/job_scheduler] Job 53 finished 1 steps successfully
[2017-05-04 10:48:14,658: INFO/job_scheduler] Job 53: complete_job: Updating cache
[2017-05-04 10:48:14,658: INFO/job_scheduler] Job 53 completing (errored=False, cancelled=False)
[2017-05-04 10:48:14,719: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-04 10:48:21,823: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:21,823: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:21,823: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:21,823: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:21,823: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4c49f99e-dd48-44a1-8ff8-4649a193d9eb
[2017-05-04 10:48:21,823: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4c49f99e-dd48-44a1-8ff8-4649a193d9eb
[2017-05-04 10:48:21,858: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4c49f99e-dd48-44a1-8ff8-4649a193d9eb
[2017-05-04 10:48:21,858: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4c49f99e-dd48-44a1-8ff8-4649a193d9eb
[2017-05-04 10:48:22,423: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com install_packages {'repos': [u'e2fsprogs', u'iml-agent', u'lustre', u'zfs'], 'packages': [u'chroma-agent-management', u'kernel-devel-3.10.0-514.10.2.el7_lustre', u'lustre-osd-zfs', u'lustre-osd-ldiskfs', u'lustre-modules', u'lustre', u'zfs']}
[2017-05-04 10:48:22,441: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 437b3ff7-fe59-4ec3-a237-cd9e0a377563
[2017-05-04 10:48:22,441: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 437b3ff7-fe59-4ec3-a237-cd9e0a377563
[2017-05-04 10:48:32,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:32,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:32,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:32,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:32,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dfb1d201-bafa-42e6-807b-48e62ca419ca
[2017-05-04 10:48:32,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dfb1d201-bafa-42e6-807b-48e62ca419ca
[2017-05-04 10:48:32,240: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dfb1d201-bafa-42e6-807b-48e62ca419ca
[2017-05-04 10:48:32,240: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dfb1d201-bafa-42e6-807b-48e62ca419ca
[2017-05-04 10:48:32,663: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com kernel_status {}
[2017-05-04 10:48:32,680: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 40e2e512-8996-4efb-a7dd-d421892c6465
[2017-05-04 10:48:32,680: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 40e2e512-8996-4efb-a7dd-d421892c6465
[2017-05-04 10:48:34,940: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:34,940: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:34,940: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:34,940: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:34,940: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 73582111-be51-471c-b740-0a142dbe25ed
[2017-05-04 10:48:34,940: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 73582111-be51-471c-b740-0a142dbe25ed
[2017-05-04 10:48:34,957: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 73582111-be51-471c-b740-0a142dbe25ed
[2017-05-04 10:48:34,957: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 73582111-be51-471c-b740-0a142dbe25ed
[2017-05-04 10:48:35,381: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com kernel_status {}
[2017-05-04 10:48:35,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 66c2fd41-c57d-46d8-88f9-3612da6152df
[2017-05-04 10:48:35,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 66c2fd41-c57d-46d8-88f9-3612da6152df
[2017-05-04 10:48:35,767: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:35,767: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:35,767: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:35,767: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:35,768: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 40e2e512-8996-4efb-a7dd-d421892c6465
[2017-05-04 10:48:35,768: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 40e2e512-8996-4efb-a7dd-d421892c6465
[2017-05-04 10:48:35,796: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 40e2e512-8996-4efb-a7dd-d421892c6465
[2017-05-04 10:48:35,796: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 40e2e512-8996-4efb-a7dd-d421892c6465
[2017-05-04 10:48:35,796: INFO/job_scheduler] Job 31 finished 4 steps successfully
[2017-05-04 10:48:35,856: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 139993922588304
[2017-05-04 10:48:35,860: INFO/job] Job 31: StateChangeJob complete, setting state packages_installed on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 10:48:35,860: INFO/job] Job 31: not_deleted=True
[2017-05-04 10:48:35,860: INFO/job_scheduler] Job 31: complete_job: Updating cache
[2017-05-04 10:48:35,860: INFO/job_scheduler] Job 31: locked_item 139993922577744 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-04 10:48:35,863: INFO/job_scheduler] Job 31 completing (errored=False, cancelled=False)
[2017-05-04 10:48:35,878: INFO/job_scheduler] run_next: 4 runnable jobs of (40 pending, 3 tasked)
[2017-05-04 10:48:35,897: INFO/job_scheduler] Job 32: RunJobThread.run
[2017-05-04 10:48:35,898: INFO/job_scheduler] Job 33: RunJobThread.run
[2017-05-04 10:48:35,899: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-05-04 10:48:35,907: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-05-04 10:48:35,907: INFO/job_scheduler] Job 34: RunJobThread.run
[2017-05-04 10:48:35,908: INFO/job_scheduler] Job 35: open DB connection during completion
[2017-05-04 10:48:35,909: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-05-04 10:48:35,914: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d1eeaf14-0b49-4959-935f-8655e96cf95d
[2017-05-04 10:48:35,914: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d1eeaf14-0b49-4959-935f-8655e96cf95d
[2017-05-04 10:48:35,929: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 48bf76df-564e-4fdd-8cad-7c2fa6b432e1
[2017-05-04 10:48:35,929: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 48bf76df-564e-4fdd-8cad-7c2fa6b432e1
[2017-05-04 10:48:35,943: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 85369799-dbf9-41ee-946d-99bf3514f7d1
[2017-05-04 10:48:35,943: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 85369799-dbf9-41ee-946d-99bf3514f7d1
[2017-05-04 10:48:35,977: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 139993922585424
[2017-05-04 10:48:35,985: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-04 10:48:35,988: INFO/job] Job 35: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-04 10:48:35,990: INFO/job] Job 35: not_deleted=True
[2017-05-04 10:48:35,998: INFO/job_scheduler] Job 35: complete_job: Updating cache
[2017-05-04 10:48:35,998: INFO/job_scheduler] Job 35: locked_item 139993922585424 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-04 10:48:36,000: INFO/job_scheduler] Job 35 completing (errored=False, cancelled=False)
[2017-05-04 10:48:36,133: INFO/job_scheduler] run_next: 1 runnable jobs of (36 pending, 6 tasked)
[2017-05-04 10:48:36,138: INFO/job_scheduler] Job 36: RunJobThread.run
[2017-05-04 10:48:36,138: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com load_lnet {}
[2017-05-04 10:48:36,152: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9c5712fd-8680-48c9-8fe6-afce74c4c6d9
[2017-05-04 10:48:36,152: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9c5712fd-8680-48c9-8fe6-afce74c4c6d9
[2017-05-04 10:48:37,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:37,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:37,256: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:37,256: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:37,256: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9c5712fd-8680-48c9-8fe6-afce74c4c6d9
[2017-05-04 10:48:37,256: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9c5712fd-8680-48c9-8fe6-afce74c4c6d9
[2017-05-04 10:48:37,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9c5712fd-8680-48c9-8fe6-afce74c4c6d9
[2017-05-04 10:48:37,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9c5712fd-8680-48c9-8fe6-afce74c4c6d9
[2017-05-04 10:48:37,269: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-04 10:48:37,281: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 077fa525-dc29-439f-9ef4-a90717abfb9f
[2017-05-04 10:48:37,281: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 077fa525-dc29-439f-9ef4-a90717abfb9f
[2017-05-04 10:48:37,378: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:37,378: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:37,379: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:37,379: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:37,379: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 077fa525-dc29-439f-9ef4-a90717abfb9f
[2017-05-04 10:48:37,379: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 077fa525-dc29-439f-9ef4-a90717abfb9f
[2017-05-04 10:48:37,396: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 077fa525-dc29-439f-9ef4-a90717abfb9f
[2017-05-04 10:48:37,396: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 077fa525-dc29-439f-9ef4-a90717abfb9f
[2017-05-04 10:48:37,496: INFO/job_scheduler] Job 36 finished 2 steps successfully
[2017-05-04 10:48:37,498: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:37,498: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:37,499: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:37,499: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:37,499: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 66c2fd41-c57d-46d8-88f9-3612da6152df
[2017-05-04 10:48:37,499: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 66c2fd41-c57d-46d8-88f9-3612da6152df
[2017-05-04 10:48:37,516: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 66c2fd41-c57d-46d8-88f9-3612da6152df
[2017-05-04 10:48:37,516: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 66c2fd41-c57d-46d8-88f9-3612da6152df
[2017-05-04 10:48:37,517: INFO/job_scheduler] Job 42 finished 4 steps successfully
[2017-05-04 10:48:37,543: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 139993922585424
[2017-05-04 10:48:37,549: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-04 10:48:37,564: INFO/job] Job 36: StateChangeJob complete, setting state lnet_down on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-04 10:48:37,564: INFO/job] Job 36: not_deleted=True
[2017-05-04 10:48:37,564: INFO/job_scheduler] Job 36: complete_job: Updating cache
[2017-05-04 10:48:37,564: INFO/job_scheduler] Job 36: locked_item 139993922585424 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-04 10:48:37,567: INFO/job_scheduler] Job 36 completing (errored=False, cancelled=False)
[2017-05-04 10:48:37,599: INFO/job_scheduler] run_next: 1 runnable jobs of (35 pending, 6 tasked)
[2017-05-04 10:48:37,602: INFO/job_scheduler] Job 39: RunJobThread.run
[2017-05-04 10:48:37,603: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_lnet {}
[2017-05-04 10:48:37,621: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8e4298c0-f527-4b7e-a959-65a57b80f520
[2017-05-04 10:48:37,621: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8e4298c0-f527-4b7e-a959-65a57b80f520
[2017-05-04 10:48:37,660: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 139993922600848
[2017-05-04 10:48:37,664: INFO/job] Job 42: StateChangeJob complete, setting state packages_installed on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 10:48:37,664: INFO/job] Job 42: not_deleted=True
[2017-05-04 10:48:37,664: INFO/job_scheduler] Job 42: complete_job: Updating cache
[2017-05-04 10:48:37,664: INFO/job_scheduler] Job 42: locked_item 139993318325008 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-04 10:48:37,668: INFO/job_scheduler] Job 42 completing (errored=False, cancelled=False)
[2017-05-04 10:48:37,691: INFO/job_scheduler] run_next: 4 runnable jobs of (34 pending, 6 tasked)
[2017-05-04 10:48:37,713: INFO/job_scheduler] Job 43: RunJobThread.run
[2017-05-04 10:48:37,714: INFO/job_scheduler] Job 44: open DB connection during completion
[2017-05-04 10:48:37,716: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-05-04 10:48:37,724: INFO/job_scheduler] Job 45: RunJobThread.run
[2017-05-04 10:48:37,726: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-05-04 10:48:37,727: INFO/job_scheduler] Job 46: RunJobThread.run
[2017-05-04 10:48:37,728: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-05-04 10:48:37,734: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 37a3086b-29bf-4e31-95f9-72665847a4b7
[2017-05-04 10:48:37,734: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 37a3086b-29bf-4e31-95f9-72665847a4b7
[2017-05-04 10:48:37,745: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 139993922579664
[2017-05-04 10:48:37,752: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-04 10:48:37,754: INFO/job] Job 44: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-04 10:48:37,754: INFO/job] Job 44: not_deleted=True
[2017-05-04 10:48:37,754: INFO/job_scheduler] Job 44: complete_job: Updating cache
[2017-05-04 10:48:37,754: INFO/job_scheduler] Job 44: locked_item 139993922579664 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-04 10:48:37,758: INFO/job_scheduler] Job 44 completing (errored=False, cancelled=False)
[2017-05-04 10:48:37,785: INFO/job_scheduler] run_next: 1 runnable jobs of (30 pending, 9 tasked)
[2017-05-04 10:48:37,790: INFO/job_scheduler] Job 48: RunJobThread.run
[2017-05-04 10:48:37,790: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com load_lnet {}
[2017-05-04 10:48:37,792: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 20cd27dc-9c1e-49b7-a337-8300175b7ddd
[2017-05-04 10:48:37,792: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 20cd27dc-9c1e-49b7-a337-8300175b7ddd
[2017-05-04 10:48:37,804: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 440a6b94-69ce-4b5a-b2fa-71dc16cda3b8
[2017-05-04 10:48:37,804: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 440a6b94-69ce-4b5a-b2fa-71dc16cda3b8
[2017-05-04 10:48:37,819: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 28578dd4-b38a-462a-9b8e-6c280134d834
[2017-05-04 10:48:37,819: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 28578dd4-b38a-462a-9b8e-6c280134d834
[2017-05-04 10:48:38,735: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:38,735: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:38,735: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:38,735: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:38,735: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8e4298c0-f527-4b7e-a959-65a57b80f520
[2017-05-04 10:48:38,735: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8e4298c0-f527-4b7e-a959-65a57b80f520
[2017-05-04 10:48:38,736: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8e4298c0-f527-4b7e-a959-65a57b80f520
[2017-05-04 10:48:38,736: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8e4298c0-f527-4b7e-a959-65a57b80f520
[2017-05-04 10:48:38,737: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-04 10:48:38,750: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ecf5f052-debc-46c4-b30c-8498854d69f6
[2017-05-04 10:48:38,750: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ecf5f052-debc-46c4-b30c-8498854d69f6
[2017-05-04 10:48:38,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:38,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:38,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:38,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:38,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ecf5f052-debc-46c4-b30c-8498854d69f6
[2017-05-04 10:48:38,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ecf5f052-debc-46c4-b30c-8498854d69f6
[2017-05-04 10:48:38,865: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ecf5f052-debc-46c4-b30c-8498854d69f6
[2017-05-04 10:48:38,865: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ecf5f052-debc-46c4-b30c-8498854d69f6
[2017-05-04 10:48:38,869: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:38,869: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:38,869: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:38,869: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:38,869: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 28578dd4-b38a-462a-9b8e-6c280134d834
[2017-05-04 10:48:38,869: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 28578dd4-b38a-462a-9b8e-6c280134d834
[2017-05-04 10:48:38,886: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 28578dd4-b38a-462a-9b8e-6c280134d834
[2017-05-04 10:48:38,886: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 28578dd4-b38a-462a-9b8e-6c280134d834
[2017-05-04 10:48:38,887: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-04 10:48:38,903: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 841701dd-0db3-401f-a012-3e24e2f881ed
[2017-05-04 10:48:38,903: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 841701dd-0db3-401f-a012-3e24e2f881ed
[2017-05-04 10:48:38,991: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:38,991: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:38,991: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:38,991: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:38,991: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 841701dd-0db3-401f-a012-3e24e2f881ed
[2017-05-04 10:48:38,991: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 841701dd-0db3-401f-a012-3e24e2f881ed
[2017-05-04 10:48:38,995: INFO/job_scheduler] Job 39 finished 2 steps successfully
[2017-05-04 10:48:39,019: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 841701dd-0db3-401f-a012-3e24e2f881ed
[2017-05-04 10:48:39,019: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 841701dd-0db3-401f-a012-3e24e2f881ed
[2017-05-04 10:48:39,099: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 139993922585424
[2017-05-04 10:48:39,103: INFO/job] Job 39: StateChangeJob complete, setting state lnet_up on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-04 10:48:39,103: INFO/job] Job 39: not_deleted=True
[2017-05-04 10:48:39,103: INFO/job_scheduler] Job 39: complete_job: Updating cache
[2017-05-04 10:48:39,103: INFO/job_scheduler] Job 39: locked_item 139993922585424 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-04 10:48:39,106: INFO/job_scheduler] Job 39 completing (errored=False, cancelled=False)
[2017-05-04 10:48:39,122: INFO/job_scheduler] run_next: 0 runnable jobs of (29 pending, 9 tasked)
[2017-05-04 10:48:39,330: INFO/job_scheduler] Job 48 finished 2 steps successfully
[2017-05-04 10:48:39,357: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 139993922579664
[2017-05-04 10:48:39,363: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-04 10:48:39,372: INFO/job] Job 48: StateChangeJob complete, setting state lnet_down on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-04 10:48:39,375: INFO/job] Job 48: not_deleted=True
[2017-05-04 10:48:39,375: INFO/job_scheduler] Job 48: complete_job: Updating cache
[2017-05-04 10:48:39,376: INFO/job_scheduler] Job 48: locked_item 139993922579664 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-04 10:48:39,387: INFO/job_scheduler] Job 48 completing (errored=False, cancelled=False)
[2017-05-04 10:48:39,421: INFO/job_scheduler] run_next: 1 runnable jobs of (29 pending, 8 tasked)
[2017-05-04 10:48:39,428: INFO/job_scheduler] Job 49: RunJobThread.run
[2017-05-04 10:48:39,429: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_lnet {}
[2017-05-04 10:48:39,443: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a887e945-25f6-4ba1-a7c8-ab1970466eb0
[2017-05-04 10:48:39,443: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a887e945-25f6-4ba1-a7c8-ab1970466eb0
[2017-05-04 10:48:39,492: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:39,492: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:39,492: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:39,492: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:39,493: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a887e945-25f6-4ba1-a7c8-ab1970466eb0
[2017-05-04 10:48:39,493: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a887e945-25f6-4ba1-a7c8-ab1970466eb0
[2017-05-04 10:48:39,507: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a887e945-25f6-4ba1-a7c8-ab1970466eb0
[2017-05-04 10:48:39,507: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a887e945-25f6-4ba1-a7c8-ab1970466eb0
[2017-05-04 10:48:39,508: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-04 10:48:39,519: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 04b2c19c-fc90-4baf-b026-e973a3109b46
[2017-05-04 10:48:39,519: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 04b2c19c-fc90-4baf-b026-e973a3109b46
[2017-05-04 10:48:39,617: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:39,617: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:39,617: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:39,617: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:39,618: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 04b2c19c-fc90-4baf-b026-e973a3109b46
[2017-05-04 10:48:39,618: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 04b2c19c-fc90-4baf-b026-e973a3109b46
[2017-05-04 10:48:39,634: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 04b2c19c-fc90-4baf-b026-e973a3109b46
[2017-05-04 10:48:39,634: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 04b2c19c-fc90-4baf-b026-e973a3109b46
[2017-05-04 10:48:39,673: INFO/job_scheduler] Job 49 finished 2 steps successfully
[2017-05-04 10:48:39,858: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 139993922579664
[2017-05-04 10:48:39,894: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm8.lotus.hpdd.lab.intel.com at severity 30
[2017-05-04 10:48:39,904: INFO/job] Job 49: StateChangeJob complete, setting state lnet_up on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-04 10:48:39,917: INFO/job] Job 49: not_deleted=True
[2017-05-04 10:48:39,917: INFO/job_scheduler] Job 49: complete_job: Updating cache
[2017-05-04 10:48:39,917: INFO/job_scheduler] Job 49: locked_item 139993922579664 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-04 10:48:39,920: INFO/job_scheduler] Job 49 completing (errored=False, cancelled=False)
[2017-05-04 10:48:39,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:39,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:39,942: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:39,942: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:39,944: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 437b3ff7-fe59-4ec3-a237-cd9e0a377563
[2017-05-04 10:48:39,944: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 437b3ff7-fe59-4ec3-a237-cd9e0a377563
[2017-05-04 10:48:39,953: INFO/job_scheduler] run_next: 0 runnable jobs of (28 pending, 8 tasked)
[2017-05-04 10:48:39,970: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 437b3ff7-fe59-4ec3-a237-cd9e0a377563
[2017-05-04 10:48:39,970: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 437b3ff7-fe59-4ec3-a237-cd9e0a377563
[2017-05-04 10:48:40,406: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com kernel_status {}
[2017-05-04 10:48:40,420: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 301169d1-c31f-45e3-a2c2-84b040a5d735
[2017-05-04 10:48:40,420: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 301169d1-c31f-45e3-a2c2-84b040a5d735
[2017-05-04 10:48:41,639: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:48:41,639: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:48:41,639: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:48:41,639: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:48:41,639: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ca15a2eb-b9d5-4322-95c4-187d805e61bf
[2017-05-04 10:48:41,639: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ca15a2eb-b9d5-4322-95c4-187d805e61bf
[2017-05-04 10:48:41,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ca15a2eb-b9d5-4322-95c4-187d805e61bf
[2017-05-04 10:48:41,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ca15a2eb-b9d5-4322-95c4-187d805e61bf
[2017-05-04 10:48:42,014: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:42,014: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:42,014: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:42,014: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:42,014: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 48bf76df-564e-4fdd-8cad-7c2fa6b432e1
[2017-05-04 10:48:42,014: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 48bf76df-564e-4fdd-8cad-7c2fa6b432e1
[2017-05-04 10:48:42,044: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com install_packages {'repos': [u'e2fsprogs', u'iml-agent', u'lustre', u'zfs'], 'packages': [u'chroma-agent-management', u'kernel-devel-3.10.0-514.10.2.el7_lustre', u'lustre-osd-zfs', u'lustre-osd-ldiskfs', u'lustre-modules', u'lustre', u'zfs']}
[2017-05-04 10:48:42,052: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 48bf76df-564e-4fdd-8cad-7c2fa6b432e1
[2017-05-04 10:48:42,052: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 48bf76df-564e-4fdd-8cad-7c2fa6b432e1
[2017-05-04 10:48:42,053: INFO/job_scheduler] Job 32 finished 1 steps successfully
[2017-05-04 10:48:42,062: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7557b77f-b5bf-48ad-93a3-311fbd4011d3
[2017-05-04 10:48:42,062: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7557b77f-b5bf-48ad-93a3-311fbd4011d3
[2017-05-04 10:48:42,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:42,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:42,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:42,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:42,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 301169d1-c31f-45e3-a2c2-84b040a5d735
[2017-05-04 10:48:42,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 301169d1-c31f-45e3-a2c2-84b040a5d735
[2017-05-04 10:48:42,586: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 301169d1-c31f-45e3-a2c2-84b040a5d735
[2017-05-04 10:48:42,586: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 301169d1-c31f-45e3-a2c2-84b040a5d735
[2017-05-04 10:48:42,587: INFO/job_scheduler] Job 20 finished 4 steps successfully
[2017-05-04 10:48:42,748: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 139993922585616
[2017-05-04 10:48:42,752: INFO/job] Job 32: StateChangeJob complete, setting state configured on lotus-32vm7.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-04 10:48:42,752: INFO/job] Job 32: not_deleted=True
[2017-05-04 10:48:42,752: INFO/job_scheduler] Job 32: complete_job: Updating cache
[2017-05-04 10:48:42,752: INFO/job_scheduler] Job 32: locked_item 139993922585616 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-04 10:48:42,755: INFO/job_scheduler] Job 32 completing (errored=False, cancelled=False)
[2017-05-04 10:48:42,772: INFO/job_scheduler] run_next: 0 runnable jobs of (28 pending, 7 tasked)
[2017-05-04 10:48:42,925: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 139993922597840
[2017-05-04 10:48:42,930: INFO/job] Job 20: StateChangeJob complete, setting state packages_installed on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 10:48:42,930: INFO/job] Job 20: not_deleted=True
[2017-05-04 10:48:42,930: INFO/job_scheduler] Job 20: complete_job: Updating cache
[2017-05-04 10:48:42,930: INFO/job_scheduler] Job 20: locked_item 139993922578704 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-04 10:48:42,933: INFO/job_scheduler] Job 20 completing (errored=False, cancelled=False)
[2017-05-04 10:48:42,948: INFO/job_scheduler] run_next: 4 runnable jobs of (28 pending, 6 tasked)
[2017-05-04 10:48:42,966: INFO/job_scheduler] Job 21: RunJobThread.run
[2017-05-04 10:48:42,967: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-05-04 10:48:42,975: INFO/job_scheduler] Job 22: RunJobThread.run
[2017-05-04 10:48:42,976: INFO/job_scheduler] Job 23: RunJobThread.run
[2017-05-04 10:48:42,977: INFO/job_scheduler] Job 24: open DB connection during completion
[2017-05-04 10:48:42,979: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-05-04 10:48:42,982: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-05-04 10:48:42,984: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3566daa9-9ce8-4f78-ba98-af777d5f0c0f
[2017-05-04 10:48:42,984: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3566daa9-9ce8-4f78-ba98-af777d5f0c0f
[2017-05-04 10:48:42,997: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f675fcb3-49c0-4429-94ed-716dff903c32
[2017-05-04 10:48:42,997: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f675fcb3-49c0-4429-94ed-716dff903c32
[2017-05-04 10:48:43,011: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 807d6d6f-dc7d-4fea-a497-3db39749dc52
[2017-05-04 10:48:43,011: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 807d6d6f-dc7d-4fea-a497-3db39749dc52
[2017-05-04 10:48:43,034: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 139993922414416
[2017-05-04 10:48:43,041: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-04 10:48:43,043: INFO/job] Job 24: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-04 10:48:43,045: INFO/job] Job 24: not_deleted=True
[2017-05-04 10:48:43,045: INFO/job_scheduler] Job 24: complete_job: Updating cache
[2017-05-04 10:48:43,045: INFO/job_scheduler] Job 24: locked_item 139993922414416 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-04 10:48:43,051: INFO/job_scheduler] Job 24 completing (errored=False, cancelled=False)
[2017-05-04 10:48:43,064: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:43,064: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:43,067: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:43,067: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:48:43,071: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d1eeaf14-0b49-4959-935f-8655e96cf95d
[2017-05-04 10:48:43,071: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d1eeaf14-0b49-4959-935f-8655e96cf95d
[2017-05-04 10:48:43,082: INFO/job_scheduler] run_next: 1 runnable jobs of (24 pending, 9 tasked)
[2017-05-04 10:48:43,087: INFO/job_scheduler] Job 25: RunJobThread.run
[2017-05-04 10:48:43,088: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com load_lnet {}
[2017-05-04 10:48:43,095: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d1eeaf14-0b49-4959-935f-8655e96cf95d
[2017-05-04 10:48:43,095: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d1eeaf14-0b49-4959-935f-8655e96cf95d
[2017-05-04 10:48:43,096: INFO/job_scheduler] Job 33 finished 1 steps successfully
[2017-05-04 10:48:43,100: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1b1cfbcf-8f3b-4bcc-b286-c73b0bce4ece
[2017-05-04 10:48:43,100: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1b1cfbcf-8f3b-4bcc-b286-c73b0bce4ece
[2017-05-04 10:48:43,159: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 139993922585808
[2017-05-04 10:48:43,162: INFO/job] Job 33: StateChangeJob complete, setting state configured on lotus-32vm7.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-04 10:48:43,163: INFO/job] Job 33: not_deleted=True
[2017-05-04 10:48:43,163: INFO/job_scheduler] Job 33: complete_job: Updating cache
[2017-05-04 10:48:43,163: INFO/job_scheduler] Job 33: locked_item 139993922585808 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-04 10:48:43,166: INFO/job_scheduler] Job 33 completing (errored=False, cancelled=False)
[2017-05-04 10:48:43,183: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:43,183: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:43,184: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:43,184: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:43,184: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1b1cfbcf-8f3b-4bcc-b286-c73b0bce4ece
[2017-05-04 10:48:43,184: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1b1cfbcf-8f3b-4bcc-b286-c73b0bce4ece
[2017-05-04 10:48:43,191: INFO/job_scheduler] run_next: 0 runnable jobs of (23 pending, 9 tasked)
[2017-05-04 10:48:43,214: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1b1cfbcf-8f3b-4bcc-b286-c73b0bce4ece
[2017-05-04 10:48:43,214: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1b1cfbcf-8f3b-4bcc-b286-c73b0bce4ece
[2017-05-04 10:48:43,215: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-04 10:48:43,231: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 740a19d8-1722-412e-acd9-028fbac5ba28
[2017-05-04 10:48:43,231: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 740a19d8-1722-412e-acd9-028fbac5ba28
[2017-05-04 10:48:43,354: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:43,354: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:43,354: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:43,354: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:43,355: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 740a19d8-1722-412e-acd9-028fbac5ba28
[2017-05-04 10:48:43,355: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 740a19d8-1722-412e-acd9-028fbac5ba28
[2017-05-04 10:48:43,395: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 740a19d8-1722-412e-acd9-028fbac5ba28
[2017-05-04 10:48:43,395: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 740a19d8-1722-412e-acd9-028fbac5ba28
[2017-05-04 10:48:43,765: INFO/job_scheduler] Job 25 finished 2 steps successfully
[2017-05-04 10:48:43,801: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 139993922414416
[2017-05-04 10:48:43,806: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-04 10:48:43,817: INFO/job] Job 25: StateChangeJob complete, setting state lnet_down on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-04 10:48:43,820: INFO/job] Job 25: not_deleted=True
[2017-05-04 10:48:43,820: INFO/job_scheduler] Job 25: complete_job: Updating cache
[2017-05-04 10:48:43,821: INFO/job_scheduler] Job 25: locked_item 139993922414416 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-04 10:48:43,826: INFO/job_scheduler] Job 25 completing (errored=False, cancelled=False)
[2017-05-04 10:48:43,848: INFO/job_scheduler] run_next: 1 runnable jobs of (23 pending, 8 tasked)
[2017-05-04 10:48:43,852: INFO/job_scheduler] Job 28: RunJobThread.run
[2017-05-04 10:48:43,852: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_lnet {}
[2017-05-04 10:48:43,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 085f6439-a791-493b-b31d-9d8f79f0931a
[2017-05-04 10:48:43,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 085f6439-a791-493b-b31d-9d8f79f0931a
[2017-05-04 10:48:43,882: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:43,882: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:43,882: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:43,882: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:43,882: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 37a3086b-29bf-4e31-95f9-72665847a4b7
[2017-05-04 10:48:43,882: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 37a3086b-29bf-4e31-95f9-72665847a4b7
[2017-05-04 10:48:43,910: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 37a3086b-29bf-4e31-95f9-72665847a4b7
[2017-05-04 10:48:43,910: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 37a3086b-29bf-4e31-95f9-72665847a4b7
[2017-05-04 10:48:43,911: INFO/job_scheduler] Job 43 finished 1 steps successfully
[2017-05-04 10:48:43,926: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:43,926: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:43,926: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:43,926: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:43,926: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 085f6439-a791-493b-b31d-9d8f79f0931a
[2017-05-04 10:48:43,926: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 085f6439-a791-493b-b31d-9d8f79f0931a
[2017-05-04 10:48:43,941: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 085f6439-a791-493b-b31d-9d8f79f0931a
[2017-05-04 10:48:43,941: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 085f6439-a791-493b-b31d-9d8f79f0931a
[2017-05-04 10:48:43,941: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-04 10:48:43,961: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f0111d79-8764-4518-babe-4388ed5ececf
[2017-05-04 10:48:43,961: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f0111d79-8764-4518-babe-4388ed5ececf
[2017-05-04 10:48:44,001: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 139993922579792
[2017-05-04 10:48:44,005: INFO/job] Job 43: StateChangeJob complete, setting state configured on lotus-32vm8.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-04 10:48:44,005: INFO/job] Job 43: not_deleted=True
[2017-05-04 10:48:44,005: INFO/job_scheduler] Job 43: complete_job: Updating cache
[2017-05-04 10:48:44,006: INFO/job_scheduler] Job 43: locked_item 139993922579792 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-04 10:48:44,008: INFO/job_scheduler] Job 43 completing (errored=False, cancelled=False)
[2017-05-04 10:48:44,033: INFO/job_scheduler] run_next: 0 runnable jobs of (22 pending, 8 tasked)
[2017-05-04 10:48:44,926: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:44,926: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:44,926: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:44,926: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:44,927: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 440a6b94-69ce-4b5a-b2fa-71dc16cda3b8
[2017-05-04 10:48:44,927: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 440a6b94-69ce-4b5a-b2fa-71dc16cda3b8
[2017-05-04 10:48:44,970: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 440a6b94-69ce-4b5a-b2fa-71dc16cda3b8
[2017-05-04 10:48:44,970: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 440a6b94-69ce-4b5a-b2fa-71dc16cda3b8
[2017-05-04 10:48:44,970: INFO/job_scheduler] Job 46 finished 1 steps successfully
[2017-05-04 10:48:45,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:45,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:45,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:45,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:45,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f0111d79-8764-4518-babe-4388ed5ececf
[2017-05-04 10:48:45,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f0111d79-8764-4518-babe-4388ed5ececf
[2017-05-04 10:48:45,059: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 139993922578896
[2017-05-04 10:48:45,064: INFO/job] Job 46: StateChangeJob complete, setting state configured on lotus-32vm8.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-04 10:48:45,064: INFO/job] Job 46: not_deleted=True
[2017-05-04 10:48:45,064: INFO/job_scheduler] Job 46: complete_job: Updating cache
[2017-05-04 10:48:45,064: INFO/job_scheduler] Job 46: locked_item 139993922578896 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-04 10:48:45,067: INFO/job_scheduler] Job 46 completing (errored=False, cancelled=False)
[2017-05-04 10:48:45,075: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f0111d79-8764-4518-babe-4388ed5ececf
[2017-05-04 10:48:45,075: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f0111d79-8764-4518-babe-4388ed5ececf
[2017-05-04 10:48:45,092: INFO/job_scheduler] run_next: 0 runnable jobs of (22 pending, 7 tasked)
[2017-05-04 10:48:45,115: INFO/job_scheduler] Job 28 finished 2 steps successfully
[2017-05-04 10:48:45,126: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 139993922414416
[2017-05-04 10:48:45,146: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm6.lotus.hpdd.lab.intel.com at severity 30
[2017-05-04 10:48:45,153: INFO/job] Job 28: StateChangeJob complete, setting state lnet_up on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-04 10:48:45,153: INFO/job] Job 28: not_deleted=True
[2017-05-04 10:48:45,153: INFO/job_scheduler] Job 28: complete_job: Updating cache
[2017-05-04 10:48:45,153: INFO/job_scheduler] Job 28: locked_item 139993922414416 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-04 10:48:45,156: INFO/job_scheduler] Job 28 completing (errored=False, cancelled=False)
[2017-05-04 10:48:45,183: INFO/job_scheduler] run_next: 0 runnable jobs of (22 pending, 6 tasked)
[2017-05-04 10:48:49,126: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:49,126: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:49,126: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:49,126: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:49,126: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 807d6d6f-dc7d-4fea-a497-3db39749dc52
[2017-05-04 10:48:49,126: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 807d6d6f-dc7d-4fea-a497-3db39749dc52
[2017-05-04 10:48:49,128: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 807d6d6f-dc7d-4fea-a497-3db39749dc52
[2017-05-04 10:48:49,128: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 807d6d6f-dc7d-4fea-a497-3db39749dc52
[2017-05-04 10:48:49,129: INFO/job_scheduler] Job 22 finished 1 steps successfully
[2017-05-04 10:48:49,326: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 139993922597008
[2017-05-04 10:48:49,330: INFO/job] Job 22: StateChangeJob complete, setting state configured on lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-04 10:48:49,330: INFO/job] Job 22: not_deleted=True
[2017-05-04 10:48:49,330: INFO/job_scheduler] Job 22: complete_job: Updating cache
[2017-05-04 10:48:49,330: INFO/job_scheduler] Job 22: locked_item 139993922597008 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-04 10:48:49,333: INFO/job_scheduler] Job 22 completing (errored=False, cancelled=False)
[2017-05-04 10:48:49,371: INFO/job_scheduler] run_next: 0 runnable jobs of (22 pending, 5 tasked)
[2017-05-04 10:48:50,083: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:50,083: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:50,083: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:50,083: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:48:50,083: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3566daa9-9ce8-4f78-ba98-af777d5f0c0f
[2017-05-04 10:48:50,083: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3566daa9-9ce8-4f78-ba98-af777d5f0c0f
[2017-05-04 10:48:50,101: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3566daa9-9ce8-4f78-ba98-af777d5f0c0f
[2017-05-04 10:48:50,101: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3566daa9-9ce8-4f78-ba98-af777d5f0c0f
[2017-05-04 10:48:50,102: INFO/job_scheduler] Job 21 finished 1 steps successfully
[2017-05-04 10:48:50,259: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 139993922597136
[2017-05-04 10:48:50,263: INFO/job] Job 21: StateChangeJob complete, setting state configured on lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-04 10:48:50,263: INFO/job] Job 21: not_deleted=True
[2017-05-04 10:48:50,263: INFO/job_scheduler] Job 21: complete_job: Updating cache
[2017-05-04 10:48:50,263: INFO/job_scheduler] Job 21: locked_item 139993922597136 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-04 10:48:50,266: INFO/job_scheduler] Job 21 completing (errored=False, cancelled=False)
[2017-05-04 10:48:50,298: INFO/job_scheduler] run_next: 0 runnable jobs of (22 pending, 4 tasked)
[2017-05-04 10:48:54,199: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:54,199: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:54,200: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:54,200: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:54,200: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 20cd27dc-9c1e-49b7-a337-8300175b7ddd
[2017-05-04 10:48:54,200: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 20cd27dc-9c1e-49b7-a337-8300175b7ddd
[2017-05-04 10:48:54,210: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 20cd27dc-9c1e-49b7-a337-8300175b7ddd
[2017-05-04 10:48:54,210: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 20cd27dc-9c1e-49b7-a337-8300175b7ddd
[2017-05-04 10:48:54,211: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.227', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-05-04 10:48:54,227: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cc5483bc-d69a-4c5b-b7ca-fc9e4ef652cd
[2017-05-04 10:48:54,227: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cc5483bc-d69a-4c5b-b7ca-fc9e4ef652cd
[2017-05-04 10:48:54,316: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:54,316: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:54,316: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:54,316: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:48:54,316: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cc5483bc-d69a-4c5b-b7ca-fc9e4ef652cd
[2017-05-04 10:48:54,316: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cc5483bc-d69a-4c5b-b7ca-fc9e4ef652cd
[2017-05-04 10:48:54,341: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cc5483bc-d69a-4c5b-b7ca-fc9e4ef652cd
[2017-05-04 10:48:54,341: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cc5483bc-d69a-4c5b-b7ca-fc9e4ef652cd
[2017-05-04 10:48:54,360: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 53219, 'pcs_password': 'IcxXcHgS3vJnrhVBEE5V'}
[2017-05-04 10:48:54,376: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc eb2bb05e-702a-43ec-a2ab-d7b802b079ca
[2017-05-04 10:48:54,376: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc eb2bb05e-702a-43ec-a2ab-d7b802b079ca
[2017-05-04 10:48:59,830: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:48:59,830: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:48:59,831: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:48:59,831: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:48:59,831: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7557b77f-b5bf-48ad-93a3-311fbd4011d3
[2017-05-04 10:48:59,831: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7557b77f-b5bf-48ad-93a3-311fbd4011d3
[2017-05-04 10:48:59,832: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7557b77f-b5bf-48ad-93a3-311fbd4011d3
[2017-05-04 10:48:59,832: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7557b77f-b5bf-48ad-93a3-311fbd4011d3
[2017-05-04 10:49:00,249: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com kernel_status {}
[2017-05-04 10:49:00,263: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bf9aa80c-d5d6-4adb-aac7-c80c2d06b82c
[2017-05-04 10:49:00,263: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bf9aa80c-d5d6-4adb-aac7-c80c2d06b82c
[2017-05-04 10:49:02,405: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:02,405: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:02,405: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:02,405: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:02,406: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bf9aa80c-d5d6-4adb-aac7-c80c2d06b82c
[2017-05-04 10:49:02,406: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bf9aa80c-d5d6-4adb-aac7-c80c2d06b82c
[2017-05-04 10:49:02,428: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bf9aa80c-d5d6-4adb-aac7-c80c2d06b82c
[2017-05-04 10:49:02,428: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bf9aa80c-d5d6-4adb-aac7-c80c2d06b82c
[2017-05-04 10:49:02,428: INFO/job_scheduler] Job 9 finished 4 steps successfully
[2017-05-04 10:49:02,525: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 139993318381264
[2017-05-04 10:49:02,529: INFO/job] Job 9: StateChangeJob complete, setting state packages_installed on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 10:49:02,529: INFO/job] Job 9: not_deleted=True
[2017-05-04 10:49:02,529: INFO/job_scheduler] Job 9: complete_job: Updating cache
[2017-05-04 10:49:02,529: INFO/job_scheduler] Job 9: locked_item 139993922468496 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-04 10:49:02,532: INFO/job_scheduler] Job 9 completing (errored=False, cancelled=False)
[2017-05-04 10:49:02,548: INFO/job_scheduler] run_next: 4 runnable jobs of (22 pending, 3 tasked)
[2017-05-04 10:49:02,567: INFO/job_scheduler] Job 10: RunJobThread.run
[2017-05-04 10:49:02,569: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-05-04 10:49:02,568: INFO/job_scheduler] Job 11: RunJobThread.run
[2017-05-04 10:49:02,568: INFO/job_scheduler] Job 12: open DB connection during completion
[2017-05-04 10:49:02,580: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-05-04 10:49:02,582: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8109b4ac-e49d-4f3c-94b5-3d879e050ecb
[2017-05-04 10:49:02,582: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8109b4ac-e49d-4f3c-94b5-3d879e050ecb
[2017-05-04 10:49:02,592: INFO/job_scheduler] Job 13: RunJobThread.run
[2017-05-04 10:49:02,593: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-05-04 10:49:02,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c8dce5ca-0757-42df-aa73-184c3738a6e8
[2017-05-04 10:49:02,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c8dce5ca-0757-42df-aa73-184c3738a6e8
[2017-05-04 10:49:02,615: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 35862c2f-a4dc-431c-af63-ea5660084f98
[2017-05-04 10:49:02,615: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 35862c2f-a4dc-431c-af63-ea5660084f98
[2017-05-04 10:49:02,617: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 139993318502032
[2017-05-04 10:49:02,625: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-04 10:49:02,628: INFO/job] Job 12: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-04 10:49:02,633: INFO/job] Job 12: not_deleted=True
[2017-05-04 10:49:02,641: INFO/job_scheduler] Job 12: complete_job: Updating cache
[2017-05-04 10:49:02,641: INFO/job_scheduler] Job 12: locked_item 139993318502032 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-04 10:49:02,654: INFO/job_scheduler] Job 12 completing (errored=False, cancelled=False)
[2017-05-04 10:49:02,692: INFO/job_scheduler] run_next: 1 runnable jobs of (18 pending, 6 tasked)
[2017-05-04 10:49:02,698: INFO/job_scheduler] Job 14: RunJobThread.run
[2017-05-04 10:49:02,699: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com load_lnet {}
[2017-05-04 10:49:02,712: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 30eb9e7b-fcb1-4d46-99d3-7dd67500517d
[2017-05-04 10:49:02,712: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 30eb9e7b-fcb1-4d46-99d3-7dd67500517d
[2017-05-04 10:49:03,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:03,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:03,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:03,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:03,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 30eb9e7b-fcb1-4d46-99d3-7dd67500517d
[2017-05-04 10:49:03,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 30eb9e7b-fcb1-4d46-99d3-7dd67500517d
[2017-05-04 10:49:03,826: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 30eb9e7b-fcb1-4d46-99d3-7dd67500517d
[2017-05-04 10:49:03,826: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 30eb9e7b-fcb1-4d46-99d3-7dd67500517d
[2017-05-04 10:49:03,827: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-04 10:49:03,839: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a5a53417-1fa0-4ba6-b747-6a4a77a125ba
[2017-05-04 10:49:03,839: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a5a53417-1fa0-4ba6-b747-6a4a77a125ba
[2017-05-04 10:49:03,948: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:03,948: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:03,948: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:03,948: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:03,948: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a5a53417-1fa0-4ba6-b747-6a4a77a125ba
[2017-05-04 10:49:03,948: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a5a53417-1fa0-4ba6-b747-6a4a77a125ba
[2017-05-04 10:49:03,953: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a5a53417-1fa0-4ba6-b747-6a4a77a125ba
[2017-05-04 10:49:03,953: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a5a53417-1fa0-4ba6-b747-6a4a77a125ba
[2017-05-04 10:49:04,272: INFO/job_scheduler] Job 14 finished 2 steps successfully
[2017-05-04 10:49:04,291: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 139993318502032
[2017-05-04 10:49:04,297: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-04 10:49:04,301: INFO/job] Job 14: StateChangeJob complete, setting state lnet_down on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-04 10:49:04,302: INFO/job] Job 14: not_deleted=True
[2017-05-04 10:49:04,302: INFO/job_scheduler] Job 14: complete_job: Updating cache
[2017-05-04 10:49:04,303: INFO/job_scheduler] Job 14: locked_item 139993318502032 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-04 10:49:04,308: INFO/job_scheduler] Job 14 completing (errored=False, cancelled=False)
[2017-05-04 10:49:04,355: INFO/job_scheduler] run_next: 1 runnable jobs of (17 pending, 6 tasked)
[2017-05-04 10:49:04,360: INFO/job_scheduler] Job 17: RunJobThread.run
[2017-05-04 10:49:04,362: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_lnet {}
[2017-05-04 10:49:04,376: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc aefbaa71-51fe-4426-89d9-19b30f548cb1
[2017-05-04 10:49:04,376: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc aefbaa71-51fe-4426-89d9-19b30f548cb1
[2017-05-04 10:49:06,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:49:06,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:49:06,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:49:06,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:49:06,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc eb2bb05e-702a-43ec-a2ab-d7b802b079ca
[2017-05-04 10:49:06,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc eb2bb05e-702a-43ec-a2ab-d7b802b079ca
[2017-05-04 10:49:06,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc eb2bb05e-702a-43ec-a2ab-d7b802b079ca
[2017-05-04 10:49:06,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc eb2bb05e-702a-43ec-a2ab-d7b802b079ca
[2017-05-04 10:49:06,546: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': True, 'mcast_port': 53219, 'new_node_fqdn': u'lotus-32vm8.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'IcxXcHgS3vJnrhVBEE5V'}
[2017-05-04 10:49:06,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fee3bdef-a759-4a5f-9b68-1ecbba685176
[2017-05-04 10:49:06,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fee3bdef-a759-4a5f-9b68-1ecbba685176
[2017-05-04 10:49:06,567: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:06,567: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:06,567: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:06,567: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:06,567: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc aefbaa71-51fe-4426-89d9-19b30f548cb1
[2017-05-04 10:49:06,567: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc aefbaa71-51fe-4426-89d9-19b30f548cb1
[2017-05-04 10:49:06,591: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc aefbaa71-51fe-4426-89d9-19b30f548cb1
[2017-05-04 10:49:06,591: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc aefbaa71-51fe-4426-89d9-19b30f548cb1
[2017-05-04 10:49:06,592: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-04 10:49:06,609: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 10b816b3-5046-4fb5-ade6-3ff79a5bb0ff
[2017-05-04 10:49:06,609: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 10b816b3-5046-4fb5-ade6-3ff79a5bb0ff
[2017-05-04 10:49:06,705: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:06,705: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:06,705: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:06,705: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:06,705: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 10b816b3-5046-4fb5-ade6-3ff79a5bb0ff
[2017-05-04 10:49:06,705: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 10b816b3-5046-4fb5-ade6-3ff79a5bb0ff
[2017-05-04 10:49:06,723: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 10b816b3-5046-4fb5-ade6-3ff79a5bb0ff
[2017-05-04 10:49:06,723: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 10b816b3-5046-4fb5-ade6-3ff79a5bb0ff
[2017-05-04 10:49:07,102: INFO/job_scheduler] Job 17 finished 2 steps successfully
[2017-05-04 10:49:07,160: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 139993318502032
[2017-05-04 10:49:07,164: INFO/job] Job 17: StateChangeJob complete, setting state lnet_up on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-04 10:49:07,164: INFO/job] Job 17: not_deleted=True
[2017-05-04 10:49:07,164: INFO/job_scheduler] Job 17: complete_job: Updating cache
[2017-05-04 10:49:07,165: INFO/job_scheduler] Job 17: locked_item 139993318502032 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-04 10:49:07,167: INFO/job_scheduler] Job 17 completing (errored=False, cancelled=False)
[2017-05-04 10:49:07,192: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 6 tasked)
[2017-05-04 10:49:08,774: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:08,774: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:08,774: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:08,774: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:08,774: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c8dce5ca-0757-42df-aa73-184c3738a6e8
[2017-05-04 10:49:08,774: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c8dce5ca-0757-42df-aa73-184c3738a6e8
[2017-05-04 10:49:08,775: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c8dce5ca-0757-42df-aa73-184c3738a6e8
[2017-05-04 10:49:08,775: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c8dce5ca-0757-42df-aa73-184c3738a6e8
[2017-05-04 10:49:08,775: INFO/job_scheduler] Job 11 finished 1 steps successfully
[2017-05-04 10:49:08,857: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 139993318501584
[2017-05-04 10:49:08,862: INFO/job] Job 11: StateChangeJob complete, setting state configured on lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-04 10:49:08,863: INFO/job] Job 11: not_deleted=True
[2017-05-04 10:49:08,863: INFO/job_scheduler] Job 11: complete_job: Updating cache
[2017-05-04 10:49:08,863: INFO/job_scheduler] Job 11: locked_item 139993318501584 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-04 10:49:08,866: INFO/job_scheduler] Job 11 completing (errored=False, cancelled=False)
[2017-05-04 10:49:08,888: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 5 tasked)
[2017-05-04 10:49:09,771: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:09,771: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:09,771: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:09,771: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:09,772: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 35862c2f-a4dc-431c-af63-ea5660084f98
[2017-05-04 10:49:09,772: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 35862c2f-a4dc-431c-af63-ea5660084f98
[2017-05-04 10:49:09,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 35862c2f-a4dc-431c-af63-ea5660084f98
[2017-05-04 10:49:09,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 35862c2f-a4dc-431c-af63-ea5660084f98
[2017-05-04 10:49:09,781: INFO/job_scheduler] Job 13 finished 1 steps successfully
[2017-05-04 10:49:09,898: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 139993318450960
[2017-05-04 10:49:09,902: INFO/job] Job 13: StateChangeJob complete, setting state configured on lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-04 10:49:09,902: INFO/job] Job 13: not_deleted=True
[2017-05-04 10:49:09,903: INFO/job_scheduler] Job 13: complete_job: Updating cache
[2017-05-04 10:49:09,903: INFO/job_scheduler] Job 13: locked_item 139993318450960 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-04 10:49:09,905: INFO/job_scheduler] Job 13 completing (errored=False, cancelled=False)
[2017-05-04 10:49:09,930: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 4 tasked)
[2017-05-04 10:49:12,680: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:49:12,680: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:49:12,681: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:49:12,681: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:49:12,681: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fee3bdef-a759-4a5f-9b68-1ecbba685176
[2017-05-04 10:49:12,681: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fee3bdef-a759-4a5f-9b68-1ecbba685176
[2017-05-04 10:49:12,681: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fee3bdef-a759-4a5f-9b68-1ecbba685176
[2017-05-04 10:49:12,681: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fee3bdef-a759-4a5f-9b68-1ecbba685176
[2017-05-04 10:49:12,682: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-04 10:49:12.682334+00:00:
[2017-05-04 10:49:12,682: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-04 10:49:12.682334+00:00:
[2017-05-04 10:49:12,682: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->53219
[2017-05-04 10:49:12,682: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->53219
[2017-05-04 10:49:12,686: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-04 10:49:12,686: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-04 10:49:12,701: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:49:12,702: INFO/job_scheduler]   Job 45 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-05-04 10:49:12,703: INFO/job_scheduler]   Job 47 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-05-04 10:49:12,704: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:49:12,704: INFO/job_scheduler]   Job 51 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:49:12,704: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 4 tasked)
[2017-05-04 10:49:12,703: INFO/job_scheduler] Job 45 finished 1 steps successfully
[2017-05-04 10:49:12,731: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 139993922599440
[2017-05-04 10:49:12,739: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-04 10:49:12,748: INFO/job] Job 45: StateChangeJob complete, setting state stopped on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:49:12,749: INFO/job] Job 45: not_deleted=True
[2017-05-04 10:49:12,758: INFO/job_scheduler] Job 45: complete_job: Updating cache
[2017-05-04 10:49:12,759: INFO/job_scheduler] Job 45: locked_item 139993922599440 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-04 10:49:12,774: INFO/job_scheduler] Job 45 completing (errored=False, cancelled=False)
[2017-05-04 10:49:12,798: INFO/job_scheduler] run_next: 1 runnable jobs of (16 pending, 3 tasked)
[2017-05-04 10:49:12,806: INFO/job_scheduler] Job 47: RunJobThread.run
[2017-05-04 10:49:12,807: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-05-04 10:49:12,823: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 65022b3b-0b48-4b3a-8061-4b20daabf49d
[2017-05-04 10:49:12,823: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 65022b3b-0b48-4b3a-8061-4b20daabf49d
[2017-05-04 10:49:14,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:49:14,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:49:14,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:49:14,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:49:14,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 85369799-dbf9-41ee-946d-99bf3514f7d1
[2017-05-04 10:49:14,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 85369799-dbf9-41ee-946d-99bf3514f7d1
[2017-05-04 10:49:14,170: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 85369799-dbf9-41ee-946d-99bf3514f7d1
[2017-05-04 10:49:14,170: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 85369799-dbf9-41ee-946d-99bf3514f7d1
[2017-05-04 10:49:14,171: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.226', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-05-04 10:49:14,183: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d68a571f-c78d-4e17-b924-34f71a9fa297
[2017-05-04 10:49:14,183: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d68a571f-c78d-4e17-b924-34f71a9fa297
[2017-05-04 10:49:14,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:49:14,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:49:14,244: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:49:14,244: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:49:14,244: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d68a571f-c78d-4e17-b924-34f71a9fa297
[2017-05-04 10:49:14,244: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d68a571f-c78d-4e17-b924-34f71a9fa297
[2017-05-04 10:49:14,247: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d68a571f-c78d-4e17-b924-34f71a9fa297
[2017-05-04 10:49:14,247: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d68a571f-c78d-4e17-b924-34f71a9fa297
[2017-05-04 10:49:14,262: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 53219, 'pcs_password': 'C5QPKz6Wxnoy4V1JaR67'}
[2017-05-04 10:49:14,278: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2f764fb3-3a88-475c-ae38-e20dabb6b33b
[2017-05-04 10:49:14,278: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2f764fb3-3a88-475c-ae38-e20dabb6b33b
[2017-05-04 10:49:16,992: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:16,992: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:16,992: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:16,992: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:16,992: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8109b4ac-e49d-4f3c-94b5-3d879e050ecb
[2017-05-04 10:49:16,992: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8109b4ac-e49d-4f3c-94b5-3d879e050ecb
[2017-05-04 10:49:16,999: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8109b4ac-e49d-4f3c-94b5-3d879e050ecb
[2017-05-04 10:49:16,999: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8109b4ac-e49d-4f3c-94b5-3d879e050ecb
[2017-05-04 10:49:17,000: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.224', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-05-04 10:49:17,013: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0086026d-8f8d-4ead-bcbe-878aea0a6082
[2017-05-04 10:49:17,013: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0086026d-8f8d-4ead-bcbe-878aea0a6082
[2017-05-04 10:49:17,062: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:17,062: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:17,063: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:17,063: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:17,063: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0086026d-8f8d-4ead-bcbe-878aea0a6082
[2017-05-04 10:49:17,063: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0086026d-8f8d-4ead-bcbe-878aea0a6082
[2017-05-04 10:49:17,079: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0086026d-8f8d-4ead-bcbe-878aea0a6082
[2017-05-04 10:49:17,079: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0086026d-8f8d-4ead-bcbe-878aea0a6082
[2017-05-04 10:49:17,093: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 47491, 'pcs_password': 'D7hdvHRHBiokUeuO8uKQ'}
[2017-05-04 10:49:17,107: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 82dceb63-544c-4621-a855-80e7f83d64cf
[2017-05-04 10:49:17,107: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 82dceb63-544c-4621-a855-80e7f83d64cf
[2017-05-04 10:49:20,930: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:49:20,930: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:49:20,930: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:49:20,930: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:49:20,931: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 65022b3b-0b48-4b3a-8061-4b20daabf49d
[2017-05-04 10:49:20,931: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 65022b3b-0b48-4b3a-8061-4b20daabf49d
[2017-05-04 10:49:20,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 65022b3b-0b48-4b3a-8061-4b20daabf49d
[2017-05-04 10:49:20,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 65022b3b-0b48-4b3a-8061-4b20daabf49d
[2017-05-04 10:49:20,939: INFO/job_scheduler] Job 47 finished 1 steps successfully
[2017-05-04 10:49:21,000: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 139993922599440
[2017-05-04 10:49:21,031: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm8.lotus.hpdd.lab.intel.com at severity 30
[2017-05-04 10:49:21,044: INFO/job] Job 47: StateChangeJob complete, setting state started on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:49:21,044: INFO/job] Job 47: not_deleted=True
[2017-05-04 10:49:21,044: INFO/job_scheduler] Job 47: complete_job: Updating cache
[2017-05-04 10:49:21,044: INFO/job_scheduler] Job 47: locked_item 139993922599440 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-04 10:49:21,048: INFO/job_scheduler] Job 47 completing (errored=False, cancelled=False)
[2017-05-04 10:49:21,073: INFO/job_scheduler] run_next: 1 runnable jobs of (15 pending, 3 tasked)
[2017-05-04 10:49:21,104: INFO/job_scheduler] Job 50: RunJobThread.run
[2017-05-04 10:49:21,104: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-04 10:49:21,118: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3d4a16b9-9c30-463d-af81-bb276c449e68
[2017-05-04 10:49:21,118: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3d4a16b9-9c30-463d-af81-bb276c449e68
[2017-05-04 10:49:21,135: INFO/job_scheduler] run_next: 0 runnable jobs of (14 pending, 4 tasked)
[2017-05-04 10:49:21,162: INFO/job_scheduler] run_next: 0 runnable jobs of (14 pending, 4 tasked)
[2017-05-04 10:49:26,428: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:49:26,428: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:49:26,428: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:49:26,428: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:49:26,428: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2f764fb3-3a88-475c-ae38-e20dabb6b33b
[2017-05-04 10:49:26,428: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2f764fb3-3a88-475c-ae38-e20dabb6b33b
[2017-05-04 10:49:26,443: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2f764fb3-3a88-475c-ae38-e20dabb6b33b
[2017-05-04 10:49:26,443: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2f764fb3-3a88-475c-ae38-e20dabb6b33b
[2017-05-04 10:49:26,444: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': False, 'mcast_port': 53219, 'new_node_fqdn': u'lotus-32vm7.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'C5QPKz6Wxnoy4V1JaR67'}
[2017-05-04 10:49:26,461: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 712bf469-c278-4778-bbb1-645de2ec1b59
[2017-05-04 10:49:26,461: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 712bf469-c278-4778-bbb1-645de2ec1b59
[2017-05-04 10:49:27,267: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:49:27,267: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:49:27,267: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:49:27,267: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:49:27,268: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f675fcb3-49c0-4429-94ed-716dff903c32
[2017-05-04 10:49:27,268: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f675fcb3-49c0-4429-94ed-716dff903c32
[2017-05-04 10:49:27,274: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f675fcb3-49c0-4429-94ed-716dff903c32
[2017-05-04 10:49:27,274: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f675fcb3-49c0-4429-94ed-716dff903c32
[2017-05-04 10:49:27,275: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.225', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-05-04 10:49:27,288: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b916c0c7-f19c-454e-bcc4-ac61b61f6847
[2017-05-04 10:49:27,288: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b916c0c7-f19c-454e-bcc4-ac61b61f6847
[2017-05-04 10:49:27,386: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:49:27,386: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:49:27,387: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:49:27,387: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:49:27,387: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b916c0c7-f19c-454e-bcc4-ac61b61f6847
[2017-05-04 10:49:27,387: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b916c0c7-f19c-454e-bcc4-ac61b61f6847
[2017-05-04 10:49:27,407: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b916c0c7-f19c-454e-bcc4-ac61b61f6847
[2017-05-04 10:49:27,407: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b916c0c7-f19c-454e-bcc4-ac61b61f6847
[2017-05-04 10:49:28,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:49:28,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:49:28,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:49:28,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:49:28,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3d4a16b9-9c30-463d-af81-bb276c449e68
[2017-05-04 10:49:28,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3d4a16b9-9c30-463d-af81-bb276c449e68
[2017-05-04 10:49:28,233: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3d4a16b9-9c30-463d-af81-bb276c449e68
[2017-05-04 10:49:28,233: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3d4a16b9-9c30-463d-af81-bb276c449e68
[2017-05-04 10:49:28,234: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-05-04 10:49:28,248: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7a9a761e-42d9-4216-8d99-2fb75bf4b751
[2017-05-04 10:49:28,248: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7a9a761e-42d9-4216-8d99-2fb75bf4b751
[2017-05-04 10:49:29,211: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:29,211: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:29,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:29,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:29,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 82dceb63-544c-4621-a855-80e7f83d64cf
[2017-05-04 10:49:29,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 82dceb63-544c-4621-a855-80e7f83d64cf
[2017-05-04 10:49:29,227: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 82dceb63-544c-4621-a855-80e7f83d64cf
[2017-05-04 10:49:29,227: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 82dceb63-544c-4621-a855-80e7f83d64cf
[2017-05-04 10:49:29,228: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': True, 'mcast_port': 47491, 'new_node_fqdn': u'lotus-32vm5.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'D7hdvHRHBiokUeuO8uKQ'}
[2017-05-04 10:49:29,241: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3b1f67a4-4ab4-47b8-be0f-fa83f231190f
[2017-05-04 10:49:29,241: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3b1f67a4-4ab4-47b8-be0f-fa83f231190f
[2017-05-04 10:49:31,159: INFO/job_scheduler] run_next: 0 runnable jobs of (14 pending, 4 tasked)
[2017-05-04 10:49:31,173: INFO/job_scheduler] run_next: 0 runnable jobs of (14 pending, 4 tasked)
[2017-05-04 10:49:32,569: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:49:32,569: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:49:32,569: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:49:32,569: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:49:32,570: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 712bf469-c278-4778-bbb1-645de2ec1b59
[2017-05-04 10:49:32,570: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 712bf469-c278-4778-bbb1-645de2ec1b59
[2017-05-04 10:49:32,579: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 712bf469-c278-4778-bbb1-645de2ec1b59
[2017-05-04 10:49:32,579: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 712bf469-c278-4778-bbb1-645de2ec1b59
[2017-05-04 10:49:32,579: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-04 10:49:32.579710+00:00:
[2017-05-04 10:49:32,579: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-04 10:49:32.579710+00:00:
[2017-05-04 10:49:32,580: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->53219
[2017-05-04 10:49:32,580: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->53219
[2017-05-04 10:49:32,581: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-04 10:49:32,581: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-04 10:49:32,594: INFO/job_scheduler] Job 34 finished 1 steps successfully
[2017-05-04 10:49:32,595: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:49:32,596: INFO/job_scheduler]   Job 34 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-05-04 10:49:32,596: INFO/job_scheduler]   Job 37 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-05-04 10:49:32,596: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:49:32,596: INFO/job_scheduler]   Job 40 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:49:32,596: INFO/job_scheduler] run_next: 0 runnable jobs of (14 pending, 4 tasked)
[2017-05-04 10:49:32,641: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 139993922587984
[2017-05-04 10:49:32,648: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-04 10:49:32,663: INFO/job] Job 34: StateChangeJob complete, setting state stopped on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:49:32,663: INFO/job] Job 34: not_deleted=True
[2017-05-04 10:49:32,663: INFO/job_scheduler] Job 34: complete_job: Updating cache
[2017-05-04 10:49:32,663: INFO/job_scheduler] Job 34: locked_item 139993922587984 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-04 10:49:32,667: INFO/job_scheduler] Job 34 completing (errored=False, cancelled=False)
[2017-05-04 10:49:32,697: INFO/job_scheduler] run_next: 1 runnable jobs of (14 pending, 3 tasked)
[2017-05-04 10:49:32,703: INFO/job_scheduler] Job 37: RunJobThread.run
[2017-05-04 10:49:32,703: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-05-04 10:49:32,716: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a9f029d4-01f4-4ee1-be6f-acea6f8c3051
[2017-05-04 10:49:32,716: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a9f029d4-01f4-4ee1-be6f-acea6f8c3051
[2017-05-04 10:49:35,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:35,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:35,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:35,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:35,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3b1f67a4-4ab4-47b8-be0f-fa83f231190f
[2017-05-04 10:49:35,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3b1f67a4-4ab4-47b8-be0f-fa83f231190f
[2017-05-04 10:49:35,356: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3b1f67a4-4ab4-47b8-be0f-fa83f231190f
[2017-05-04 10:49:35,356: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3b1f67a4-4ab4-47b8-be0f-fa83f231190f
[2017-05-04 10:49:35,357: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-04 10:49:35.357417+00:00:
[2017-05-04 10:49:35,357: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-04 10:49:35.357417+00:00:
[2017-05-04 10:49:35,359: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->47491
[2017-05-04 10:49:35,359: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->47491
[2017-05-04 10:49:35,360: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-04 10:49:35,360: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-04 10:49:35,380: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:49:35,382: INFO/job_scheduler]   Job 10 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-05-04 10:49:35,382: INFO/job_scheduler]   Job 15 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-05-04 10:49:35,385: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:49:35,385: INFO/job_scheduler]   Job 18 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:49:35,385: INFO/job_scheduler] run_next: 0 runnable jobs of (13 pending, 4 tasked)
[2017-05-04 10:49:35,387: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 47491, 'pcs_password': 'nwODZK3W299dCxu7iUPB'}
[2017-05-04 10:49:35,384: INFO/job_scheduler] Job 10 finished 1 steps successfully
[2017-05-04 10:49:35,401: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ed24b1e1-d232-4a5e-ad33-7ece2037df54
[2017-05-04 10:49:35,401: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ed24b1e1-d232-4a5e-ad33-7ece2037df54
[2017-05-04 10:49:35,472: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 139993318469392
[2017-05-04 10:49:35,479: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-04 10:49:35,498: INFO/job] Job 10: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:49:35,499: INFO/job] Job 10: not_deleted=True
[2017-05-04 10:49:35,505: INFO/job_scheduler] Job 10: complete_job: Updating cache
[2017-05-04 10:49:35,510: INFO/job_scheduler] Job 10: locked_item 139993318469392 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-04 10:49:35,520: INFO/job_scheduler] Job 10 completing (errored=False, cancelled=False)
[2017-05-04 10:49:35,554: INFO/job_scheduler] run_next: 1 runnable jobs of (13 pending, 3 tasked)
[2017-05-04 10:49:35,557: INFO/job_scheduler] Job 15: RunJobThread.run
[2017-05-04 10:49:35,558: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-05-04 10:49:35,573: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f7baccc7-1bd9-477a-92fd-609c184195b1
[2017-05-04 10:49:35,573: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f7baccc7-1bd9-477a-92fd-609c184195b1
[2017-05-04 10:49:39,928: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-04 10:49:39,941: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-04 10:49:40,621: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-04 10:49:40,636: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-04 10:49:40,826: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:49:40,826: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:49:40,827: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:49:40,827: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:49:40,827: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a9f029d4-01f4-4ee1-be6f-acea6f8c3051
[2017-05-04 10:49:40,827: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a9f029d4-01f4-4ee1-be6f-acea6f8c3051
[2017-05-04 10:49:40,831: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a9f029d4-01f4-4ee1-be6f-acea6f8c3051
[2017-05-04 10:49:40,831: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a9f029d4-01f4-4ee1-be6f-acea6f8c3051
[2017-05-04 10:49:40,831: INFO/job_scheduler] Job 37 finished 1 steps successfully
[2017-05-04 10:49:40,888: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 139993922587984
[2017-05-04 10:49:40,918: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 30
[2017-05-04 10:49:40,929: INFO/job] Job 37: StateChangeJob complete, setting state started on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:49:40,929: INFO/job] Job 37: not_deleted=True
[2017-05-04 10:49:40,929: INFO/job_scheduler] Job 37: complete_job: Updating cache
[2017-05-04 10:49:40,929: INFO/job_scheduler] Job 37: locked_item 139993922587984 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-04 10:49:40,938: INFO/job_scheduler] Job 37 completing (errored=False, cancelled=False)
[2017-05-04 10:49:40,973: INFO/job_scheduler] run_next: 1 runnable jobs of (12 pending, 3 tasked)
[2017-05-04 10:49:40,987: INFO/job_scheduler] Job 38: RunJobThread.run
[2017-05-04 10:49:40,988: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-04 10:49:41,002: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 99766e37-fedc-4114-99da-6be6be127d49
[2017-05-04 10:49:41,002: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 99766e37-fedc-4114-99da-6be6be127d49
[2017-05-04 10:49:41,168: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 4 tasked)
[2017-05-04 10:49:41,184: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 4 tasked)
[2017-05-04 10:49:43,671: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:43,671: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:43,671: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:43,671: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:43,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f7baccc7-1bd9-477a-92fd-609c184195b1
[2017-05-04 10:49:43,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f7baccc7-1bd9-477a-92fd-609c184195b1
[2017-05-04 10:49:43,688: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f7baccc7-1bd9-477a-92fd-609c184195b1
[2017-05-04 10:49:43,688: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f7baccc7-1bd9-477a-92fd-609c184195b1
[2017-05-04 10:49:43,688: INFO/job_scheduler] Job 15 finished 1 steps successfully
[2017-05-04 10:49:43,799: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 139993318469392
[2017-05-04 10:49:43,839: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 30
[2017-05-04 10:49:43,866: INFO/job] Job 15: StateChangeJob complete, setting state started on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:49:43,867: INFO/job] Job 15: not_deleted=True
[2017-05-04 10:49:43,868: INFO/job_scheduler] Job 15: complete_job: Updating cache
[2017-05-04 10:49:43,868: INFO/job_scheduler] Job 15: locked_item 139993318469392 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-04 10:49:43,882: INFO/job_scheduler] Job 15 completing (errored=False, cancelled=False)
[2017-05-04 10:49:43,918: INFO/job_scheduler] run_next: 1 runnable jobs of (11 pending, 3 tasked)
[2017-05-04 10:49:43,931: INFO/job_scheduler] Job 16: RunJobThread.run
[2017-05-04 10:49:43,931: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-04 10:49:43,945: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d3045af4-1b39-4da5-981c-b3044bcd4cdd
[2017-05-04 10:49:43,945: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d3045af4-1b39-4da5-981c-b3044bcd4cdd
[2017-05-04 10:49:47,542: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:49:47,542: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:49:47,542: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:49:47,542: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:49:47,542: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ed24b1e1-d232-4a5e-ad33-7ece2037df54
[2017-05-04 10:49:47,542: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ed24b1e1-d232-4a5e-ad33-7ece2037df54
[2017-05-04 10:49:47,568: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ed24b1e1-d232-4a5e-ad33-7ece2037df54
[2017-05-04 10:49:47,568: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ed24b1e1-d232-4a5e-ad33-7ece2037df54
[2017-05-04 10:49:47,569: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': False, 'mcast_port': 47491, 'new_node_fqdn': u'lotus-32vm6.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'nwODZK3W299dCxu7iUPB'}
[2017-05-04 10:49:47,584: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0ff5143b-5d75-4c17-95c0-8fad867d0d0b
[2017-05-04 10:49:47,584: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0ff5143b-5d75-4c17-95c0-8fad867d0d0b
[2017-05-04 10:49:48,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:49:48,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:49:48,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:49:48,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:49:48,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 99766e37-fedc-4114-99da-6be6be127d49
[2017-05-04 10:49:48,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 99766e37-fedc-4114-99da-6be6be127d49
[2017-05-04 10:49:48,118: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 99766e37-fedc-4114-99da-6be6be127d49
[2017-05-04 10:49:48,118: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 99766e37-fedc-4114-99da-6be6be127d49
[2017-05-04 10:49:48,119: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-05-04 10:49:48,137: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cffa7008-26c4-4d36-9243-ca538fe08874
[2017-05-04 10:49:48,137: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cffa7008-26c4-4d36-9243-ca538fe08874
[2017-05-04 10:49:49,953: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 4 tasked)
[2017-05-04 10:49:49,967: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 4 tasked)
[2017-05-04 10:49:50,649: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 4 tasked)
[2017-05-04 10:49:50,667: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 4 tasked)
[2017-05-04 10:49:50,703: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:49:50,704: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:49:50,704: INFO/job_scheduler]   Job 40 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:49:50,704: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 4 tasked)
[2017-05-04 10:49:50,714: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:49:50,714: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 10:49:50,714: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:49:50,715: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 4 tasked)
[2017-05-04 10:49:50,733: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:49:50,733: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 10:49:50,733: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:49:50,733: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 4 tasked)
[2017-05-04 10:49:51,059: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:51,059: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:51,059: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:51,059: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:51,059: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d3045af4-1b39-4da5-981c-b3044bcd4cdd
[2017-05-04 10:49:51,059: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d3045af4-1b39-4da5-981c-b3044bcd4cdd
[2017-05-04 10:49:51,060: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d3045af4-1b39-4da5-981c-b3044bcd4cdd
[2017-05-04 10:49:51,060: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d3045af4-1b39-4da5-981c-b3044bcd4cdd
[2017-05-04 10:49:51,061: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-05-04 10:49:51,074: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cbc119b8-5386-439b-b57f-7047c868997c
[2017-05-04 10:49:51,074: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cbc119b8-5386-439b-b57f-7047c868997c
[2017-05-04 10:49:53,671: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:53,671: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:53,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:53,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:49:53,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0ff5143b-5d75-4c17-95c0-8fad867d0d0b
[2017-05-04 10:49:53,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0ff5143b-5d75-4c17-95c0-8fad867d0d0b
[2017-05-04 10:49:53,699: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0ff5143b-5d75-4c17-95c0-8fad867d0d0b
[2017-05-04 10:49:53,699: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0ff5143b-5d75-4c17-95c0-8fad867d0d0b
[2017-05-04 10:49:53,699: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-04 10:49:53.699886+00:00:
[2017-05-04 10:49:53,699: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-04 10:49:53.699886+00:00:
[2017-05-04 10:49:53,700: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->47491
[2017-05-04 10:49:53,700: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->47491
[2017-05-04 10:49:53,702: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-04 10:49:53,702: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-04 10:49:53,715: INFO/job_scheduler] Job 23 finished 1 steps successfully
[2017-05-04 10:49:53,716: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:49:53,717: INFO/job_scheduler]   Job 23 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-05-04 10:49:53,717: INFO/job_scheduler]   Job 26 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-05-04 10:49:53,717: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:49:53,717: INFO/job_scheduler]   Job 29 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:49:53,718: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 4 tasked)
[2017-05-04 10:49:53,794: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 139993922599632
[2017-05-04 10:49:53,801: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-04 10:49:53,810: INFO/job] Job 23: StateChangeJob complete, setting state stopped on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:49:53,812: INFO/job] Job 23: not_deleted=True
[2017-05-04 10:49:53,813: INFO/job_scheduler] Job 23: complete_job: Updating cache
[2017-05-04 10:49:53,815: INFO/job_scheduler] Job 23: locked_item 139993922599632 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-04 10:49:53,830: INFO/job_scheduler] Job 23 completing (errored=False, cancelled=False)
[2017-05-04 10:49:53,878: INFO/job_scheduler] run_next: 1 runnable jobs of (10 pending, 3 tasked)
[2017-05-04 10:49:53,882: INFO/job_scheduler] Job 26: RunJobThread.run
[2017-05-04 10:49:53,883: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-05-04 10:49:53,897: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 76635f4b-abde-4e5d-beec-3196132a98a6
[2017-05-04 10:49:53,897: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 76635f4b-abde-4e5d-beec-3196132a98a6
[2017-05-04 10:50:00,017: INFO/job_scheduler] run_next: 0 runnable jobs of (9 pending, 4 tasked)
[2017-05-04 10:50:00,034: INFO/job_scheduler] run_next: 0 runnable jobs of (9 pending, 4 tasked)
[2017-05-04 10:50:00,288: INFO/job_scheduler] run_next: 0 runnable jobs of (9 pending, 4 tasked)
[2017-05-04 10:50:00,302: INFO/job_scheduler] run_next: 0 runnable jobs of (9 pending, 4 tasked)
[2017-05-04 10:50:00,696: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:00,697: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:00,697: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:00,697: INFO/job_scheduler] run_next: 0 runnable jobs of (9 pending, 4 tasked)
[2017-05-04 10:50:00,716: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:00,716: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:00,716: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:00,716: INFO/job_scheduler] run_next: 0 runnable jobs of (9 pending, 4 tasked)
[2017-05-04 10:50:01,207: INFO/job_scheduler] run_next: 0 runnable jobs of (9 pending, 4 tasked)
[2017-05-04 10:50:01,223: INFO/job_scheduler] run_next: 0 runnable jobs of (9 pending, 4 tasked)
[2017-05-04 10:50:01,251: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:01,251: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:01,251: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:01,252: INFO/job_scheduler] run_next: 0 runnable jobs of (9 pending, 4 tasked)
[2017-05-04 10:50:01,271: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:01,272: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:01,272: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:01,272: INFO/job_scheduler] run_next: 0 runnable jobs of (9 pending, 4 tasked)
[2017-05-04 10:50:01,969: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:50:01,969: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:50:01,969: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:50:01,969: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:50:01,969: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 76635f4b-abde-4e5d-beec-3196132a98a6
[2017-05-04 10:50:01,969: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 76635f4b-abde-4e5d-beec-3196132a98a6
[2017-05-04 10:50:02,014: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 76635f4b-abde-4e5d-beec-3196132a98a6
[2017-05-04 10:50:02,014: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 76635f4b-abde-4e5d-beec-3196132a98a6
[2017-05-04 10:50:02,014: INFO/job_scheduler] Job 26 finished 1 steps successfully
[2017-05-04 10:50:02,243: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 139993922599632
[2017-05-04 10:50:02,277: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm6.lotus.hpdd.lab.intel.com at severity 30
[2017-05-04 10:50:02,284: INFO/job] Job 26: StateChangeJob complete, setting state started on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:02,284: INFO/job] Job 26: not_deleted=True
[2017-05-04 10:50:02,284: INFO/job_scheduler] Job 26: complete_job: Updating cache
[2017-05-04 10:50:02,284: INFO/job_scheduler] Job 26: locked_item 139993922599632 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-04 10:50:02,288: INFO/job_scheduler] Job 26 completing (errored=False, cancelled=False)
[2017-05-04 10:50:02,340: INFO/job_scheduler] run_next: 1 runnable jobs of (9 pending, 3 tasked)
[2017-05-04 10:50:02,373: INFO/job_scheduler] Job 27: RunJobThread.run
[2017-05-04 10:50:02,373: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-04 10:50:02,389: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b84cd0ac-9d19-49dd-8cfc-64ac827cee2b
[2017-05-04 10:50:02,389: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b84cd0ac-9d19-49dd-8cfc-64ac827cee2b
[2017-05-04 10:50:09,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:50:09,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:50:09,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:50:09,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:50:09,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b84cd0ac-9d19-49dd-8cfc-64ac827cee2b
[2017-05-04 10:50:09,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b84cd0ac-9d19-49dd-8cfc-64ac827cee2b
[2017-05-04 10:50:09,505: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b84cd0ac-9d19-49dd-8cfc-64ac827cee2b
[2017-05-04 10:50:09,505: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b84cd0ac-9d19-49dd-8cfc-64ac827cee2b
[2017-05-04 10:50:09,506: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-05-04 10:50:09,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3ba5ed09-974f-45b8-bfba-e7c4ccb90e47
[2017-05-04 10:50:09,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3ba5ed09-974f-45b8-bfba-e7c4ccb90e47
[2017-05-04 10:50:10,088: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:50:10,088: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:10,088: INFO/job_scheduler]   Job 18 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:10,088: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:10,103: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:10,103: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:10,103: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:10,103: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:10,126: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:50:10,126: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:10,126: INFO/job_scheduler]   Job 29 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:10,127: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:10,138: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:10,138: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:10,138: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:10,139: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:10,319: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:10,335: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:10,361: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:50:10,361: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:10,361: INFO/job_scheduler]   Job 18 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:10,361: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:10,376: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:10,376: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:10,376: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:10,376: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:10,395: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:50:10,395: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:10,395: INFO/job_scheduler]   Job 29 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:10,396: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:10,410: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:10,410: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:10,410: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:10,410: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:10,706: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:10,707: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:10,707: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:10,707: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:10,727: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:10,727: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:10,727: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:10,728: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:11,281: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:11,281: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:11,281: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:11,282: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:11,299: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:11,299: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:11,299: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:11,299: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:20,065: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:20,081: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:20,109: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:20,110: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:20,110: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:20,110: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:20,131: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:20,131: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:20,132: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:20,132: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:20,742: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:50:20,743: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:20,743: INFO/job_scheduler]   Job 40 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:20,743: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:20,758: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:20,758: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:20,758: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:20,758: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:20,792: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:50:20,792: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:20,793: INFO/job_scheduler]   Job 51 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:20,793: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:20,805: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:20,805: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:20,805: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:20,805: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:21,305: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:50:21,305: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:21,305: INFO/job_scheduler]   Job 40 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:21,305: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:21,321: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:21,321: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:21,321: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:21,321: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:21,340: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:50:21,340: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:21,340: INFO/job_scheduler]   Job 51 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:21,341: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:21,355: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:21,356: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:21,356: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:21,356: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:30,093: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:30,094: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:30,094: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:30,094: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:30,112: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:30,112: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:30,112: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:30,112: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:30,755: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:50:30,755: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:30,755: INFO/job_scheduler]   Job 40 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:30,755: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:30,770: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:30,771: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:30,771: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:30,771: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:30,789: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:50:30,789: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:30,789: INFO/job_scheduler]   Job 51 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:30,789: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:30,803: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:30,803: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:30,803: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:30,803: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:31,335: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:50:31,336: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:31,336: INFO/job_scheduler]   Job 40 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:31,336: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:31,349: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:31,349: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:31,349: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:31,350: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:31,366: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:50:31,366: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:31,366: INFO/job_scheduler]   Job 51 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:31,366: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:31,382: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:31,382: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:31,383: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:31,383: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:40,178: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:50:40,178: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:40,179: INFO/job_scheduler]   Job 18 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:40,179: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:40,193: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:40,194: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:40,194: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:40,194: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:40,228: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:50:40,228: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:40,228: INFO/job_scheduler]   Job 29 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:40,229: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:40,244: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:40,245: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:40,245: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:40,245: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:40,804: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:50:40,804: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:40,804: INFO/job_scheduler]   Job 40 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:40,805: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:40,819: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:40,819: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:40,819: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:40,820: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:40,838: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:50:40,839: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:40,839: INFO/job_scheduler]   Job 51 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:40,839: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:40,852: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:40,853: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:40,853: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:40,853: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:41,379: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:50:41,379: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:41,379: INFO/job_scheduler]   Job 40 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:41,379: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:41,392: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:41,392: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:41,392: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:41,392: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:41,409: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:50:41,409: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:41,409: INFO/job_scheduler]   Job 51 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:41,409: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:41,422: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:41,422: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:41,423: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:41,423: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:48,497: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:50:48,497: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:50:48,498: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:50:48,498: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:50:48,498: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cffa7008-26c4-4d36-9243-ca538fe08874
[2017-05-04 10:50:48,498: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cffa7008-26c4-4d36-9243-ca538fe08874
[2017-05-04 10:50:48,513: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cffa7008-26c4-4d36-9243-ca538fe08874
[2017-05-04 10:50:48,513: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cffa7008-26c4-4d36-9243-ca538fe08874
[2017-05-04 10:50:48,515: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-04 10:50:48,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ea708db5-4b7b-4512-9119-35b66225627b
[2017-05-04 10:50:48,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ea708db5-4b7b-4512-9119-35b66225627b
[2017-05-04 10:50:49,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:50:49,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:50:49,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:50:49,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:50:49,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7a9a761e-42d9-4216-8d99-2fb75bf4b751
[2017-05-04 10:50:49,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7a9a761e-42d9-4216-8d99-2fb75bf4b751
[2017-05-04 10:50:49,731: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7a9a761e-42d9-4216-8d99-2fb75bf4b751
[2017-05-04 10:50:49,731: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7a9a761e-42d9-4216-8d99-2fb75bf4b751
[2017-05-04 10:50:49,733: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-04 10:50:49,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 31344933-ded5-421b-a2c8-d51bdcdcf505
[2017-05-04 10:50:49,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 31344933-ded5-421b-a2c8-d51bdcdcf505
[2017-05-04 10:50:50,173: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:50:50,173: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:50,174: INFO/job_scheduler]   Job 18 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:50,174: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:50,186: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:50,186: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:50,187: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:50,187: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:50,204: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:50:50,204: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:50,204: INFO/job_scheduler]   Job 29 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:50,205: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:50,219: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:50,219: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:50,219: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:50,219: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:50,809: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:50,825: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:50,850: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:50:50,850: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:50,851: INFO/job_scheduler]   Job 40 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:50,851: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:50,866: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:50,866: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:50,866: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:50,866: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:50,884: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:50:50,884: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:50,884: INFO/job_scheduler]   Job 51 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:50,884: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:50,897: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:50,897: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:50,897: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:50,897: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:51,374: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:50:51,374: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:50:51,435: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:50:51,435: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:50:51,435: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cbc119b8-5386-439b-b57f-7047c868997c
[2017-05-04 10:50:51,435: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cbc119b8-5386-439b-b57f-7047c868997c
[2017-05-04 10:50:51,448: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:51,450: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cbc119b8-5386-439b-b57f-7047c868997c
[2017-05-04 10:50:51,450: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cbc119b8-5386-439b-b57f-7047c868997c
[2017-05-04 10:50:51,452: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-04 10:50:51,466: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:51,468: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3518ae57-053d-4446-8c86-66393c98480d
[2017-05-04 10:50:51,468: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3518ae57-053d-4446-8c86-66393c98480d
[2017-05-04 10:50:51,492: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:50:51,492: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:51,492: INFO/job_scheduler]   Job 40 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:51,493: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:51,517: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:51,517: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:51,517: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:51,517: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:51,537: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:50:51,537: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:51,537: INFO/job_scheduler]   Job 51 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:50:51,538: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:51,561: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:50:51,561: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 10:50:51,561: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:50:51,562: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 10:50:55,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:50:55,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:50:55,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:50:55,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:50:55,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ea708db5-4b7b-4512-9119-35b66225627b
[2017-05-04 10:50:55,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ea708db5-4b7b-4512-9119-35b66225627b
[2017-05-04 10:50:55,596: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ea708db5-4b7b-4512-9119-35b66225627b
[2017-05-04 10:50:55,596: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ea708db5-4b7b-4512-9119-35b66225627b
[2017-05-04 10:50:55,596: INFO/job_scheduler] Job 38 finished 3 steps successfully
[2017-05-04 10:50:55,655: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 139993922585872
[2017-05-04 10:50:55,663: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-04 10:50:55,670: INFO/job] Job 38: StateChangeJob complete, setting state stopped on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-04 10:50:55,670: INFO/job] Job 38: not_deleted=True
[2017-05-04 10:50:55,670: INFO/job_scheduler] Job 38: complete_job: Updating cache
[2017-05-04 10:50:55,670: INFO/job_scheduler] Job 38: locked_item 139993922585872 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-04 10:50:55,684: INFO/job_scheduler] Job 38 completing (errored=False, cancelled=False)
[2017-05-04 10:50:55,716: INFO/job_scheduler] run_next: 1 runnable jobs of (8 pending, 3 tasked)
[2017-05-04 10:50:55,733: INFO/job_scheduler] Job 40: RunJobThread.run
[2017-05-04 10:50:55,733: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-04 10:50:55,749: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d440b753-045b-481d-bd9c-9421c677e7f7
[2017-05-04 10:50:55,749: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d440b753-045b-481d-bd9c-9421c677e7f7
[2017-05-04 10:50:56,833: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:50:56,833: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:50:56,833: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:50:56,833: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:50:56,833: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 31344933-ded5-421b-a2c8-d51bdcdcf505
[2017-05-04 10:50:56,833: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 31344933-ded5-421b-a2c8-d51bdcdcf505
[2017-05-04 10:50:56,861: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 31344933-ded5-421b-a2c8-d51bdcdcf505
[2017-05-04 10:50:56,861: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 31344933-ded5-421b-a2c8-d51bdcdcf505
[2017-05-04 10:50:56,862: INFO/job_scheduler] Job 50 finished 3 steps successfully
[2017-05-04 10:50:56,904: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 139993922579920
[2017-05-04 10:50:56,915: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-04 10:50:56,920: INFO/job] Job 50: StateChangeJob complete, setting state stopped on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-04 10:50:56,920: INFO/job] Job 50: not_deleted=True
[2017-05-04 10:50:56,923: INFO/job_scheduler] Job 50: complete_job: Updating cache
[2017-05-04 10:50:56,923: INFO/job_scheduler] Job 50: locked_item 139993922579920 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-04 10:50:56,936: INFO/job_scheduler] Job 50 completing (errored=False, cancelled=False)
[2017-05-04 10:50:56,979: INFO/job_scheduler] run_next: 1 runnable jobs of (7 pending, 3 tasked)
[2017-05-04 10:50:56,993: INFO/job_scheduler] Job 51: RunJobThread.run
[2017-05-04 10:50:56,994: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-04 10:50:57,010: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ebd127c6-9efc-4d61-b8ff-8b24cb21587a
[2017-05-04 10:50:57,010: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ebd127c6-9efc-4d61-b8ff-8b24cb21587a
[2017-05-04 10:50:58,543: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:50:58,543: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:50:58,543: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:50:58,543: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:50:58,543: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3518ae57-053d-4446-8c86-66393c98480d
[2017-05-04 10:50:58,543: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3518ae57-053d-4446-8c86-66393c98480d
[2017-05-04 10:50:58,583: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3518ae57-053d-4446-8c86-66393c98480d
[2017-05-04 10:50:58,583: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3518ae57-053d-4446-8c86-66393c98480d
[2017-05-04 10:50:58,583: INFO/job_scheduler] Job 16 finished 3 steps successfully
[2017-05-04 10:50:58,655: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 139993318500240
[2017-05-04 10:50:58,662: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-04 10:50:58,676: INFO/job] Job 16: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-04 10:50:58,676: INFO/job] Job 16: not_deleted=True
[2017-05-04 10:50:58,677: INFO/job_scheduler] Job 16: complete_job: Updating cache
[2017-05-04 10:50:58,677: INFO/job_scheduler] Job 16: locked_item 139993318500240 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-04 10:50:58,679: INFO/job_scheduler] Job 16 completing (errored=False, cancelled=False)
[2017-05-04 10:50:58,709: INFO/job_scheduler] run_next: 1 runnable jobs of (6 pending, 3 tasked)
[2017-05-04 10:50:58,720: INFO/job_scheduler] Job 18: RunJobThread.run
[2017-05-04 10:50:58,720: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-04 10:50:58,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 049098b3-a5f5-4a23-b073-40523d72bece
[2017-05-04 10:50:58,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 049098b3-a5f5-4a23-b073-40523d72bece
[2017-05-04 10:51:00,466: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:51:00,466: INFO/job_scheduler]   Job 18 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:51:00,467: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-04 10:51:00,482: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:51:00,482: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:51:00,483: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-04 10:51:00,502: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:51:00,503: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:51:00,503: INFO/job_scheduler]   Job 29 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:51:00,503: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-04 10:51:00,517: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:51:00,517: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 10:51:00,517: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:51:00,517: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-04 10:51:00,828: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-04 10:51:00,842: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-04 10:51:00,869: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:51:00,869: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:51:00,869: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-04 10:51:00,889: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:51:00,890: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:51:00,890: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-04 10:51:01,401: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-04 10:51:01,415: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-04 10:51:01,441: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:51:01,483: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:51:01,483: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-04 10:51:01,495: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:51:01,496: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:51:01,496: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-04 10:51:02,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:51:02,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:51:02,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:51:02,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:51:02,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d440b753-045b-481d-bd9c-9421c677e7f7
[2017-05-04 10:51:02,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d440b753-045b-481d-bd9c-9421c677e7f7
[2017-05-04 10:51:02,864: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d440b753-045b-481d-bd9c-9421c677e7f7
[2017-05-04 10:51:02,864: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d440b753-045b-481d-bd9c-9421c677e7f7
[2017-05-04 10:51:02,864: INFO/job_scheduler] Job 40 finished 1 steps successfully
[2017-05-04 10:51:02,955: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 139993922585872
[2017-05-04 10:51:02,984: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 30
[2017-05-04 10:51:02,994: INFO/job] Job 40: StateChangeJob complete, setting state started on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-04 10:51:02,994: INFO/job] Job 40: not_deleted=True
[2017-05-04 10:51:02,994: INFO/job_scheduler] Job 40: complete_job: Updating cache
[2017-05-04 10:51:02,994: INFO/job_scheduler] Job 40: locked_item 139993922585872 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-04 10:51:02,997: INFO/job_scheduler] Job 40 completing (errored=False, cancelled=False)
[2017-05-04 10:51:03,022: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-3
[2017-05-04 10:51:03,023: INFO/job_scheduler] _notify: Updating .mcast_port of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 53219
[2017-05-04 10:51:03,027: INFO/job_scheduler] _notify: Set mcast_port=53219 on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-3) and saved
[2017-05-04 10:51:03,028: INFO/job_scheduler] _notify: Updating .network_interfaces of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-05-04 10:51:03,036: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-3) and saved
[2017-05-04 10:51:03,043: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-04 10:51:03,047: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-3) and saved
[2017-05-04 10:51:03,063: INFO/job_scheduler] run_next: 1 runnable jobs of (5 pending, 3 tasked)
[2017-05-04 10:51:03,084: INFO/job_scheduler] Job 41: RunJobThread.run
[2017-05-04 10:51:03,087: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-05-04 10:51:03,105: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cf94b9ea-3a25-46d6-a0bb-8e6cb6054dd1
[2017-05-04 10:51:03,105: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cf94b9ea-3a25-46d6-a0bb-8e6cb6054dd1
[2017-05-04 10:51:03,864: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:51:03,864: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:51:03,865: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:51:03,865: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:51:03,865: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3ba5ed09-974f-45b8-bfba-e7c4ccb90e47
[2017-05-04 10:51:03,865: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3ba5ed09-974f-45b8-bfba-e7c4ccb90e47
[2017-05-04 10:51:03,900: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3ba5ed09-974f-45b8-bfba-e7c4ccb90e47
[2017-05-04 10:51:03,900: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3ba5ed09-974f-45b8-bfba-e7c4ccb90e47
[2017-05-04 10:51:03,902: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-04 10:51:03,915: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b29fba3f-6563-460c-96f8-aa05dfd8409c
[2017-05-04 10:51:03,915: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b29fba3f-6563-460c-96f8-aa05dfd8409c
[2017-05-04 10:51:04,083: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:51:04,083: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:51:04,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:51:04,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:51:04,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ebd127c6-9efc-4d61-b8ff-8b24cb21587a
[2017-05-04 10:51:04,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ebd127c6-9efc-4d61-b8ff-8b24cb21587a
[2017-05-04 10:51:04,126: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ebd127c6-9efc-4d61-b8ff-8b24cb21587a
[2017-05-04 10:51:04,126: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ebd127c6-9efc-4d61-b8ff-8b24cb21587a
[2017-05-04 10:51:04,126: INFO/job_scheduler] Job 51 finished 1 steps successfully
[2017-05-04 10:51:04,366: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 139993922579920
[2017-05-04 10:51:04,407: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm8.lotus.hpdd.lab.intel.com at severity 30
[2017-05-04 10:51:04,425: INFO/job] Job 51: StateChangeJob complete, setting state started on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-04 10:51:04,427: INFO/job] Job 51: not_deleted=True
[2017-05-04 10:51:04,427: INFO/job_scheduler] Job 51: complete_job: Updating cache
[2017-05-04 10:51:04,428: INFO/job_scheduler] Job 51: locked_item 139993922579920 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-04 10:51:04,442: INFO/job_scheduler] Job 51 completing (errored=False, cancelled=False)
[2017-05-04 10:51:04,468: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-4
[2017-05-04 10:51:04,468: INFO/job_scheduler] _notify: Updating .mcast_port of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 53219
[2017-05-04 10:51:04,473: INFO/job_scheduler] _notify: Set mcast_port=53219 on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-4) and saved
[2017-05-04 10:51:04,474: INFO/job_scheduler] _notify: Updating .network_interfaces of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-05-04 10:51:04,482: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-4) and saved
[2017-05-04 10:51:04,487: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-04 10:51:04,492: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-4) and saved
[2017-05-04 10:51:04,497: INFO/job_scheduler] run_next: 1 runnable jobs of (4 pending, 3 tasked)
[2017-05-04 10:51:04,518: INFO/job_scheduler] Job 52: RunJobThread.run
[2017-05-04 10:51:04,520: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-05-04 10:51:04,537: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d7fd5e8f-6630-4c76-8ea8-8307a69ad84f
[2017-05-04 10:51:04,537: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d7fd5e8f-6630-4c76-8ea8-8307a69ad84f
[2017-05-04 10:51:05,860: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:51:05,860: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:51:05,860: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:51:05,860: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:51:05,861: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 049098b3-a5f5-4a23-b073-40523d72bece
[2017-05-04 10:51:05,861: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 049098b3-a5f5-4a23-b073-40523d72bece
[2017-05-04 10:51:05,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 049098b3-a5f5-4a23-b073-40523d72bece
[2017-05-04 10:51:05,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 049098b3-a5f5-4a23-b073-40523d72bece
[2017-05-04 10:51:05,898: INFO/job_scheduler] Job 18 finished 1 steps successfully
[2017-05-04 10:51:05,933: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 139993318500240
[2017-05-04 10:51:05,961: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 30
[2017-05-04 10:51:05,989: INFO/job] Job 18: StateChangeJob complete, setting state started on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-04 10:51:05,994: INFO/job] Job 18: not_deleted=True
[2017-05-04 10:51:05,997: INFO/job_scheduler] Job 18: complete_job: Updating cache
[2017-05-04 10:51:05,999: INFO/job_scheduler] Job 18: locked_item 139993318500240 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-04 10:51:06,014: INFO/job_scheduler] Job 18 completing (errored=False, cancelled=False)
[2017-05-04 10:51:06,043: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-1
[2017-05-04 10:51:06,044: INFO/job_scheduler] _notify: Updating .mcast_port of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 47491
[2017-05-04 10:51:06,048: INFO/job_scheduler] _notify: Set mcast_port=47491 on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-05-04 10:51:06,049: INFO/job_scheduler] _notify: Updating .network_interfaces of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-05-04 10:51:06,058: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-05-04 10:51:06,064: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-04 10:51:06,068: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-05-04 10:51:06,080: INFO/job_scheduler] run_next: 1 runnable jobs of (3 pending, 3 tasked)
[2017-05-04 10:51:06,101: INFO/job_scheduler] Job 19: RunJobThread.run
[2017-05-04 10:51:06,104: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-05-04 10:51:06,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 117ecbf8-089f-46e5-968b-e44761aba704
[2017-05-04 10:51:06,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 117ecbf8-089f-46e5-968b-e44761aba704
[2017-05-04 10:51:10,227: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:51:10,227: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:51:10,227: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-05-04 10:51:10,243: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:51:10,243: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:51:10,244: INFO/job_scheduler]   Job 29 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:51:10,244: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-05-04 10:51:10,259: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:51:10,259: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 10:51:10,259: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:51:10,259: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-05-04 10:51:10,535: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-05-04 10:51:10,550: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-05-04 10:51:10,868: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from True to False
[2017-05-04 10:51:10,875: INFO/job_scheduler] _notify: Set corosync_reported_up=False on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-3) and saved
[2017-05-04 10:51:10,882: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-05-04 10:51:10,893: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:51:10,893: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:51:10,893: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-05-04 10:51:10,902: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from True to False
[2017-05-04 10:51:10,906: INFO/job_scheduler] _notify: Set corosync_reported_up=False on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-4) and saved
[2017-05-04 10:51:10,912: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-05-04 10:51:10,918: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:51:10,918: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:51:10,919: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-05-04 10:51:11,445: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:51:11,445: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:51:11,445: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-05-04 10:51:11,462: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:51:11,462: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:51:11,462: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-05-04 10:51:12,988: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:51:12,988: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:51:12,988: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:51:12,988: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:51:12,989: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b29fba3f-6563-460c-96f8-aa05dfd8409c
[2017-05-04 10:51:12,989: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b29fba3f-6563-460c-96f8-aa05dfd8409c
[2017-05-04 10:51:13,032: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b29fba3f-6563-460c-96f8-aa05dfd8409c
[2017-05-04 10:51:13,032: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b29fba3f-6563-460c-96f8-aa05dfd8409c
[2017-05-04 10:51:13,032: INFO/job_scheduler] Job 27 finished 3 steps successfully
[2017-05-04 10:51:13,084: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 139993922416016
[2017-05-04 10:51:13,091: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-04 10:51:13,097: INFO/job] Job 27: StateChangeJob complete, setting state stopped on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-04 10:51:13,097: INFO/job] Job 27: not_deleted=True
[2017-05-04 10:51:13,097: INFO/job_scheduler] Job 27: complete_job: Updating cache
[2017-05-04 10:51:13,097: INFO/job_scheduler] Job 27: locked_item 139993922416016 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-04 10:51:13,106: INFO/job_scheduler] Job 27 completing (errored=False, cancelled=False)
[2017-05-04 10:51:13,141: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 3 tasked)
[2017-05-04 10:51:13,152: INFO/job_scheduler] Job 29: RunJobThread.run
[2017-05-04 10:51:13,153: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-04 10:51:13,169: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 40cdd8b6-ec89-4363-811b-82db48611453
[2017-05-04 10:51:13,169: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 40cdd8b6-ec89-4363-811b-82db48611453
[2017-05-04 10:51:20,259: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:51:20,260: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:51:20,260: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 4 tasked)
[2017-05-04 10:51:20,277: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 10:51:20,277: INFO/job_scheduler]   Job 29 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 10:51:20,277: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 4 tasked)
[2017-05-04 10:51:20,288: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:51:20,288: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:51:20,288: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:51:20,288: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:51:20,288: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 40cdd8b6-ec89-4363-811b-82db48611453
[2017-05-04 10:51:20,288: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 40cdd8b6-ec89-4363-811b-82db48611453
[2017-05-04 10:51:20,293: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:51:20,294: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:51:20,294: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 4 tasked)
[2017-05-04 10:51:20,336: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 40cdd8b6-ec89-4363-811b-82db48611453
[2017-05-04 10:51:20,336: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 40cdd8b6-ec89-4363-811b-82db48611453
[2017-05-04 10:51:20,336: INFO/job_scheduler] Job 29 finished 1 steps successfully
[2017-05-04 10:51:20,369: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 139993922416016
[2017-05-04 10:51:20,408: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm6.lotus.hpdd.lab.intel.com at severity 30
[2017-05-04 10:51:20,417: INFO/job] Job 29: StateChangeJob complete, setting state started on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-04 10:51:20,417: INFO/job] Job 29: not_deleted=True
[2017-05-04 10:51:20,417: INFO/job_scheduler] Job 29: complete_job: Updating cache
[2017-05-04 10:51:20,417: INFO/job_scheduler] Job 29: locked_item 139993922416016 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-04 10:51:20,420: INFO/job_scheduler] Job 29 completing (errored=False, cancelled=False)
[2017-05-04 10:51:20,444: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-2
[2017-05-04 10:51:20,444: INFO/job_scheduler] _notify: Updating .mcast_port of item 2 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 47491
[2017-05-04 10:51:20,456: INFO/job_scheduler] _notify: Set mcast_port=47491 on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-2) and saved
[2017-05-04 10:51:20,460: INFO/job_scheduler] _notify: Updating .network_interfaces of item 2 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-05-04 10:51:20,485: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-2) and saved
[2017-05-04 10:51:20,491: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 2 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-04 10:51:20,497: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-2) and saved
[2017-05-04 10:51:20,505: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-05-04 10:51:20,527: INFO/job_scheduler] Job 30: RunJobThread.run
[2017-05-04 10:51:20,529: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-05-04 10:51:20,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 38a299aa-b8d9-4633-9df1-7c19b65087c3
[2017-05-04 10:51:20,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 38a299aa-b8d9-4633-9df1-7c19b65087c3
[2017-05-04 10:51:20,589: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-04 10:51:20,601: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-04 10:51:20,919: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:51:20,919: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:51:20,919: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-04 10:51:20,938: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:51:20,938: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:51:20,938: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-04 10:51:21,510: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-04 10:51:21,515: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-3) and saved
[2017-05-04 10:51:21,522: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-04 10:51:21,528: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:51:21,528: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:51:21,528: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-04 10:51:21,557: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-04 10:51:21,561: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-4) and saved
[2017-05-04 10:51:21,568: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-04 10:51:21,578: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:51:21,578: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:51:21,579: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-04 10:51:29,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:51:29,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:51:29,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:51:29,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:51:29,529: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cf94b9ea-3a25-46d6-a0bb-8e6cb6054dd1
[2017-05-04 10:51:29,529: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cf94b9ea-3a25-46d6-a0bb-8e6cb6054dd1
[2017-05-04 10:51:29,532: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cf94b9ea-3a25-46d6-a0bb-8e6cb6054dd1
[2017-05-04 10:51:29,532: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cf94b9ea-3a25-46d6-a0bb-8e6cb6054dd1
[2017-05-04 10:51:29,535: INFO/job_scheduler] Job 41 finished 1 steps successfully
[2017-05-04 10:51:29,884: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:51:29,884: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:51:29,884: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:51:29,884: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:51:29,884: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d7fd5e8f-6630-4c76-8ea8-8307a69ad84f
[2017-05-04 10:51:29,884: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d7fd5e8f-6630-4c76-8ea8-8307a69ad84f
[2017-05-04 10:51:29,916: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d7fd5e8f-6630-4c76-8ea8-8307a69ad84f
[2017-05-04 10:51:29,916: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d7fd5e8f-6630-4c76-8ea8-8307a69ad84f
[2017-05-04 10:51:29,918: INFO/job_scheduler] Job 52 finished 1 steps successfully
[2017-05-04 10:51:29,970: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 139993922577744
[2017-05-04 10:51:29,976: INFO/job] Job 41: StateChangeJob complete, setting state managed on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 10:51:29,976: INFO/job] Job 41: not_deleted=True
[2017-05-04 10:51:29,976: INFO/job_scheduler] Job 41: complete_job: Updating cache
[2017-05-04 10:51:29,976: INFO/job_scheduler] Job 41: locked_item 139993922577744 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-04 10:51:29,979: INFO/job_scheduler] Job 41 completing (errored=False, cancelled=False)
[2017-05-04 10:51:30,101: INFO/job_scheduler] Replaying 4 buffered notifications for ManagedHost-3
[2017-05-04 10:51:30,107: INFO/job_scheduler] _notify: Updating .immutable_state of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com) from True to False
[2017-05-04 10:51:30,112: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-3) and saved
[2017-05-04 10:51:30,126: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7f52c4065b10> to [4]
[2017-05-04 10:51:30,138: INFO/job_scheduler] _notify: Set ha_cluster_peers=[4] on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-3) and saved
[2017-05-04 10:51:30,197: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-04 10:51:30,290: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:51:30,290: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:51:30,290: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-04 10:51:30,305: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 10:51:30,306: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 10:51:30,306: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-04 10:51:30,463: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 139993318325008
[2017-05-04 10:51:30,468: INFO/job] Job 52: StateChangeJob complete, setting state managed on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 10:51:30,468: INFO/job] Job 52: not_deleted=True
[2017-05-04 10:51:30,469: INFO/job_scheduler] Job 52: complete_job: Updating cache
[2017-05-04 10:51:30,469: INFO/job_scheduler] Job 52: locked_item 139993318325008 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-04 10:51:30,471: INFO/job_scheduler] Job 52 completing (errored=False, cancelled=False)
[2017-05-04 10:51:30,491: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:51:30,491: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:51:30,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:51:30,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:51:30,500: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 117ecbf8-089f-46e5-968b-e44761aba704
[2017-05-04 10:51:30,500: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 117ecbf8-089f-46e5-968b-e44761aba704
[2017-05-04 10:51:30,530: INFO/job_scheduler] Replaying 4 buffered notifications for ManagedHost-4
[2017-05-04 10:51:30,537: INFO/job_scheduler] _notify: Updating .immutable_state of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com) from True to False
[2017-05-04 10:51:30,542: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-4) and saved
[2017-05-04 10:51:30,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 117ecbf8-089f-46e5-968b-e44761aba704
[2017-05-04 10:51:30,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 117ecbf8-089f-46e5-968b-e44761aba704
[2017-05-04 10:51:30,552: INFO/job_scheduler] Job 19 finished 1 steps successfully
[2017-05-04 10:51:30,560: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7f52bc023c90> to [3]
[2017-05-04 10:51:30,570: INFO/job_scheduler] _notify: Set ha_cluster_peers=[3] on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-4) and saved
[2017-05-04 10:51:30,627: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-05-04 10:51:30,990: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 139993922468496
[2017-05-04 10:51:30,995: INFO/job] Job 19: StateChangeJob complete, setting state managed on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 10:51:30,995: INFO/job] Job 19: not_deleted=True
[2017-05-04 10:51:30,995: INFO/job_scheduler] Job 19: complete_job: Updating cache
[2017-05-04 10:51:30,995: INFO/job_scheduler] Job 19: locked_item 139993922468496 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-04 10:51:30,998: INFO/job_scheduler] Job 19 completing (errored=False, cancelled=False)
[2017-05-04 10:51:31,047: INFO/job_scheduler] Replaying 4 buffered notifications for ManagedHost-1
[2017-05-04 10:51:31,064: INFO/job_scheduler] _notify: Updating .immutable_state of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com) from True to False
[2017-05-04 10:51:31,077: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-1) and saved
[2017-05-04 10:51:31,092: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7f52c4064210> to [2]
[2017-05-04 10:51:31,101: INFO/job_scheduler] _notify: Set ha_cluster_peers=[2] on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-1) and saved
[2017-05-04 10:51:31,160: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-04 10:51:45,875: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:51:45,875: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:51:45,875: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:51:45,875: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:51:45,875: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 38a299aa-b8d9-4633-9df1-7c19b65087c3
[2017-05-04 10:51:45,875: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 38a299aa-b8d9-4633-9df1-7c19b65087c3
[2017-05-04 10:51:45,918: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 38a299aa-b8d9-4633-9df1-7c19b65087c3
[2017-05-04 10:51:45,918: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 38a299aa-b8d9-4633-9df1-7c19b65087c3
[2017-05-04 10:51:45,920: INFO/job_scheduler] Job 30 finished 1 steps successfully
[2017-05-04 10:51:46,290: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 139993922578704
[2017-05-04 10:51:46,296: INFO/job] Job 30: StateChangeJob complete, setting state managed on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 10:51:46,296: INFO/job] Job 30: not_deleted=True
[2017-05-04 10:51:46,296: INFO/job_scheduler] Job 30: complete_job: Updating cache
[2017-05-04 10:51:46,296: INFO/job_scheduler] Job 30: locked_item 139993922578704 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-04 10:51:46,299: INFO/job_scheduler] Job 30 completing (errored=False, cancelled=False)
[2017-05-04 10:51:46,348: INFO/job_scheduler] Replaying 4 buffered notifications for ManagedHost-2
[2017-05-04 10:51:46,365: INFO/job_scheduler] _notify: Updating .immutable_state of item 2 (lotus-32vm6.lotus.hpdd.lab.intel.com) from True to False
[2017-05-04 10:51:46,376: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-2) and saved
[2017-05-04 10:51:46,396: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 2 (lotus-32vm6.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7f52c4113390> to [1]
[2017-05-04 10:51:46,407: INFO/job_scheduler] _notify: Set ha_cluster_peers=[1] on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-2) and saved
[2017-05-04 10:51:46,468: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 10:52:04,520: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 10:52:04,533: INFO/command_plan] set_state: <class 'chroma_core.models.filesystem.ManagedFilesystem'>-1 transitioning from unavailable to available
[2017-05-04 10:52:04,794: INFO/command_plan] Created command 14 (Creating filesystem testfs) with 17 jobs
[2017-05-04 10:52:04,843: INFO/job_scheduler] run_next: 4 runnable jobs of (17 pending, 0 tasked)
[2017-05-04 10:52:05,122: INFO/job_scheduler] Job 54: RunJobThread.run
[2017-05-04 10:52:05,124: INFO/job_scheduler] Job 55: RunJobThread.run
[2017-05-04 10:52:05,125: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5', 'pacemaker_ha_operation': False, 'device_type': 'zfs'}
[2017-05-04 10:52:05,126: INFO/job_scheduler] Job 56: RunJobThread.run
[2017-05-04 10:52:05,136: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5', 'pacemaker_ha_operation': False, 'device_type': 'zfs'}
[2017-05-04 10:52:05,137: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5', 'pacemaker_ha_operation': False, 'device_type': 'zfs'}
[2017-05-04 10:52:05,145: INFO/job_scheduler] Job 57: RunJobThread.run
[2017-05-04 10:52:05,147: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'pacemaker_ha_operation': False, 'device_type': 'linux'}
[2017-05-04 10:52:05,155: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b396836d-09ca-49f7-977f-f77148acdad6
[2017-05-04 10:52:05,155: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b396836d-09ca-49f7-977f-f77148acdad6
[2017-05-04 10:52:05,155: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1', 'pacemaker_ha_operation': False, 'device_type': 'zfs'}
[2017-05-04 10:52:05,163: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1', 'pacemaker_ha_operation': False, 'device_type': 'zfs'}
[2017-05-04 10:52:05,163: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1', 'pacemaker_ha_operation': False, 'device_type': 'zfs'}
[2017-05-04 10:52:05,169: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3', 'pacemaker_ha_operation': False, 'device_type': 'zfs'}
[2017-05-04 10:52:05,170: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'pacemaker_ha_operation': False, 'device_type': 'linux'}
[2017-05-04 10:52:05,173: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3', 'pacemaker_ha_operation': False, 'device_type': 'zfs'}
[2017-05-04 10:52:05,174: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3', 'pacemaker_ha_operation': False, 'device_type': 'zfs'}
[2017-05-04 10:52:05,177: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c0b65395-55af-4e89-8dcc-c71b6f78e48d
[2017-05-04 10:52:05,177: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c0b65395-55af-4e89-8dcc-c71b6f78e48d
[2017-05-04 10:52:05,186: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'pacemaker_ha_operation': False, 'device_type': 'linux'}
[2017-05-04 10:52:05,199: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b03b20e1-5a2e-44c2-859e-eab6526cac86
[2017-05-04 10:52:05,199: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b03b20e1-5a2e-44c2-859e-eab6526cac86
[2017-05-04 10:52:05,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b96aed0c-0786-473d-9085-7f7ee9fbe136
[2017-05-04 10:52:05,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b96aed0c-0786-473d-9085-7f7ee9fbe136
[2017-05-04 10:52:05,239: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e88d428c-1f67-46a1-9a6d-9f144d39c233
[2017-05-04 10:52:05,239: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e88d428c-1f67-46a1-9a6d-9f144d39c233
[2017-05-04 10:52:05,253: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8abf5c0f-7bbe-49bb-9c7c-89930e1ca23f
[2017-05-04 10:52:05,253: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8abf5c0f-7bbe-49bb-9c7c-89930e1ca23f
[2017-05-04 10:52:05,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 874aeb3e-e76a-430f-b30a-addc39db43b0
[2017-05-04 10:52:05,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 874aeb3e-e76a-430f-b30a-addc39db43b0
[2017-05-04 10:52:05,283: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8aaf8c6b-37ff-4660-8706-212a8740d566
[2017-05-04 10:52:05,283: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8aaf8c6b-37ff-4660-8706-212a8740d566
[2017-05-04 10:52:05,297: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e8f2a59c-4dde-4d45-9157-e12b1e2244da
[2017-05-04 10:52:05,297: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e8f2a59c-4dde-4d45-9157-e12b1e2244da
[2017-05-04 10:52:05,309: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3f6f1784-5eff-4ea7-a3d5-aac5053bf441
[2017-05-04 10:52:05,309: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3f6f1784-5eff-4ea7-a3d5-aac5053bf441
[2017-05-04 10:52:05,325: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2ed980b9-33a5-4100-bfac-dea2c197a216
[2017-05-04 10:52:05,325: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2ed980b9-33a5-4100-bfac-dea2c197a216
[2017-05-04 10:52:05,342: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9ba74497-2e36-4e22-913d-19a94983679b
[2017-05-04 10:52:05,342: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9ba74497-2e36-4e22-913d-19a94983679b
[2017-05-04 10:52:05,343: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:05,343: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:05,343: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:05,343: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:05,343: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b03b20e1-5a2e-44c2-859e-eab6526cac86
[2017-05-04 10:52:05,343: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b03b20e1-5a2e-44c2-859e-eab6526cac86
[2017-05-04 10:52:05,344: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:05,344: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:05,344: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:05,344: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:05,344: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8aaf8c6b-37ff-4660-8706-212a8740d566
[2017-05-04 10:52:05,344: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8aaf8c6b-37ff-4660-8706-212a8740d566
[2017-05-04 10:52:05,345: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:05,345: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:05,345: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:05,345: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:05,345: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8abf5c0f-7bbe-49bb-9c7c-89930e1ca23f
[2017-05-04 10:52:05,345: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8abf5c0f-7bbe-49bb-9c7c-89930e1ca23f
[2017-05-04 10:52:05,349: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8abf5c0f-7bbe-49bb-9c7c-89930e1ca23f
[2017-05-04 10:52:05,349: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8abf5c0f-7bbe-49bb-9c7c-89930e1ca23f
[2017-05-04 10:52:05,365: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8aaf8c6b-37ff-4660-8706-212a8740d566
[2017-05-04 10:52:05,365: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8aaf8c6b-37ff-4660-8706-212a8740d566
[2017-05-04 10:52:05,367: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b03b20e1-5a2e-44c2-859e-eab6526cac86
[2017-05-04 10:52:05,367: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b03b20e1-5a2e-44c2-859e-eab6526cac86
[2017-05-04 10:52:05,368: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com import_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'pacemaker_ha_operation': False, 'device_type': 'linux', 'validate_importable': False}
[2017-05-04 10:52:05,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 312e5ac2-084a-45cb-92f2-21008cfba0ee
[2017-05-04 10:52:05,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 312e5ac2-084a-45cb-92f2-21008cfba0ee
[2017-05-04 10:52:05,401: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:05,401: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:05,401: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:05,401: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:05,401: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9ba74497-2e36-4e22-913d-19a94983679b
[2017-05-04 10:52:05,401: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9ba74497-2e36-4e22-913d-19a94983679b
[2017-05-04 10:52:05,409: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9ba74497-2e36-4e22-913d-19a94983679b
[2017-05-04 10:52:05,409: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9ba74497-2e36-4e22-913d-19a94983679b
[2017-05-04 10:52:05,420: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:05,420: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:05,422: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:05,422: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:05,423: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 312e5ac2-084a-45cb-92f2-21008cfba0ee
[2017-05-04 10:52:05,423: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 312e5ac2-084a-45cb-92f2-21008cfba0ee
[2017-05-04 10:52:05,445: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 312e5ac2-084a-45cb-92f2-21008cfba0ee
[2017-05-04 10:52:05,445: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 312e5ac2-084a-45cb-92f2-21008cfba0ee
[2017-05-04 10:52:05,445: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com check_block_device {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux'}
[2017-05-04 10:52:05,468: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4f3cd716-10ef-4534-ad64-15b176695e96
[2017-05-04 10:52:05,468: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4f3cd716-10ef-4534-ad64-15b176695e96
[2017-05-04 10:52:06,252: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:06,252: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:06,252: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:06,252: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:06,253: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b396836d-09ca-49f7-977f-f77148acdad6
[2017-05-04 10:52:06,253: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b396836d-09ca-49f7-977f-f77148acdad6
[2017-05-04 10:52:06,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:06,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:06,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:06,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:06,256: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c0b65395-55af-4e89-8dcc-c71b6f78e48d
[2017-05-04 10:52:06,256: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c0b65395-55af-4e89-8dcc-c71b6f78e48d
[2017-05-04 10:52:06,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b396836d-09ca-49f7-977f-f77148acdad6
[2017-05-04 10:52:06,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b396836d-09ca-49f7-977f-f77148acdad6
[2017-05-04 10:52:06,293: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c0b65395-55af-4e89-8dcc-c71b6f78e48d
[2017-05-04 10:52:06,293: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c0b65395-55af-4e89-8dcc-c71b6f78e48d
[2017-05-04 10:52:06,324: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:06,324: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:06,324: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:06,324: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:06,324: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b96aed0c-0786-473d-9085-7f7ee9fbe136
[2017-05-04 10:52:06,324: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b96aed0c-0786-473d-9085-7f7ee9fbe136
[2017-05-04 10:52:06,327: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:06,327: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:06,327: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b96aed0c-0786-473d-9085-7f7ee9fbe136
[2017-05-04 10:52:06,327: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:06,327: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:06,327: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e88d428c-1f67-46a1-9a6d-9f144d39c233
[2017-05-04 10:52:06,327: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e88d428c-1f67-46a1-9a6d-9f144d39c233
[2017-05-04 10:52:06,327: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b96aed0c-0786-473d-9085-7f7ee9fbe136
[2017-05-04 10:52:06,355: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e88d428c-1f67-46a1-9a6d-9f144d39c233
[2017-05-04 10:52:06,355: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e88d428c-1f67-46a1-9a6d-9f144d39c233
[2017-05-04 10:52:06,356: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com import_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1', 'pacemaker_ha_operation': False, 'device_type': 'zfs', 'validate_importable': False}
[2017-05-04 10:52:06,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc abdab506-4025-4b5a-8af2-646a5261f2b6
[2017-05-04 10:52:06,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc abdab506-4025-4b5a-8af2-646a5261f2b6
[2017-05-04 10:52:06,398: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:06,398: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:06,398: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:06,398: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:06,398: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e8f2a59c-4dde-4d45-9157-e12b1e2244da
[2017-05-04 10:52:06,398: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e8f2a59c-4dde-4d45-9157-e12b1e2244da
[2017-05-04 10:52:06,402: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:06,402: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:06,402: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:06,402: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:06,402: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3f6f1784-5eff-4ea7-a3d5-aac5053bf441
[2017-05-04 10:52:06,402: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3f6f1784-5eff-4ea7-a3d5-aac5053bf441
[2017-05-04 10:52:06,419: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e8f2a59c-4dde-4d45-9157-e12b1e2244da
[2017-05-04 10:52:06,419: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e8f2a59c-4dde-4d45-9157-e12b1e2244da
[2017-05-04 10:52:06,428: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3f6f1784-5eff-4ea7-a3d5-aac5053bf441
[2017-05-04 10:52:06,428: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3f6f1784-5eff-4ea7-a3d5-aac5053bf441
[2017-05-04 10:52:06,428: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com import_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5', 'pacemaker_ha_operation': False, 'device_type': 'zfs', 'validate_importable': False}
[2017-05-04 10:52:06,440: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e0f199a6-8639-40a1-9796-5974f17480ab
[2017-05-04 10:52:06,440: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e0f199a6-8639-40a1-9796-5974f17480ab
[2017-05-04 10:52:06,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:06,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:06,527: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:06,527: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:06,527: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4f3cd716-10ef-4534-ad64-15b176695e96
[2017-05-04 10:52:06,527: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4f3cd716-10ef-4534-ad64-15b176695e96
[2017-05-04 10:52:06,533: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4f3cd716-10ef-4534-ad64-15b176695e96
[2017-05-04 10:52:06,533: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4f3cd716-10ef-4534-ad64-15b176695e96
[2017-05-04 10:52:06,534: INFO/job] testfs-MDT0000 passed pre-format check, allowing subsequent reformats
[2017-05-04 10:52:06,555: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com format_target {'index': 0, 'mkfsoptions': '-J size=2048', 'backfstype': 'ldiskfs', 'target_name': u'testfs-MDT0000', 'mgsnode': ((u'10.14.81.224@tcp0',), (u'10.14.81.225@tcp0',)), 'target_types': 'mdt', 'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux', 'failnode': [u'10.14.81.224@tcp0'], 'fsname': u'testfs'}
[2017-05-04 10:52:06,587: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c5029cc7-9fb6-4e08-8a55-188e635975f4
[2017-05-04 10:52:06,587: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c5029cc7-9fb6-4e08-8a55-188e635975f4
[2017-05-04 10:52:07,347: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:07,347: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:07,348: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:07,348: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:07,348: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 874aeb3e-e76a-430f-b30a-addc39db43b0
[2017-05-04 10:52:07,348: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 874aeb3e-e76a-430f-b30a-addc39db43b0
[2017-05-04 10:52:07,385: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 874aeb3e-e76a-430f-b30a-addc39db43b0
[2017-05-04 10:52:07,385: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 874aeb3e-e76a-430f-b30a-addc39db43b0
[2017-05-04 10:52:07,416: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:07,416: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:07,416: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:07,416: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:07,416: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2ed980b9-33a5-4100-bfac-dea2c197a216
[2017-05-04 10:52:07,416: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2ed980b9-33a5-4100-bfac-dea2c197a216
[2017-05-04 10:52:07,440: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2ed980b9-33a5-4100-bfac-dea2c197a216
[2017-05-04 10:52:07,440: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2ed980b9-33a5-4100-bfac-dea2c197a216
[2017-05-04 10:52:07,440: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com import_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3', 'pacemaker_ha_operation': False, 'device_type': 'zfs', 'validate_importable': False}
[2017-05-04 10:52:07,454: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2d92107b-3fd0-433a-a8fa-98cc7eb9e263
[2017-05-04 10:52:07,454: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2d92107b-3fd0-433a-a8fa-98cc7eb9e263
[2017-05-04 10:52:10,474: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:10,474: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:10,474: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:10,474: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:10,474: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc abdab506-4025-4b5a-8af2-646a5261f2b6
[2017-05-04 10:52:10,474: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc abdab506-4025-4b5a-8af2-646a5261f2b6
[2017-05-04 10:52:10,497: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc abdab506-4025-4b5a-8af2-646a5261f2b6
[2017-05-04 10:52:10,497: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc abdab506-4025-4b5a-8af2-646a5261f2b6
[2017-05-04 10:52:10,499: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com check_block_device {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1', 'device_type': 'zfs'}
[2017-05-04 10:52:10,511: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1af88551-8c66-4f19-a826-4b574ce4fd48
[2017-05-04 10:52:10,511: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1af88551-8c66-4f19-a826-4b574ce4fd48
[2017-05-04 10:52:10,593: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:10,593: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:10,593: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:10,593: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:10,593: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1af88551-8c66-4f19-a826-4b574ce4fd48
[2017-05-04 10:52:10,593: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1af88551-8c66-4f19-a826-4b574ce4fd48
[2017-05-04 10:52:10,629: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1af88551-8c66-4f19-a826-4b574ce4fd48
[2017-05-04 10:52:10,629: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1af88551-8c66-4f19-a826-4b574ce4fd48
[2017-05-04 10:52:10,631: INFO/job] testfs-OST0001 passed pre-format check, allowing subsequent reformats
[2017-05-04 10:52:10,656: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com format_target {'index': 1, 'backfstype': 'zfs', 'target_name': u'testfs-OST0001', 'mgsnode': ((u'10.14.81.224@tcp0',), (u'10.14.81.225@tcp0',)), 'target_types': 'ost', 'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1', 'device_type': 'zfs', 'failnode': [u'10.14.81.226@tcp0'], 'fsname': u'testfs'}
[2017-05-04 10:52:10,671: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ddf373d0-b81e-494d-b58e-82bdb503c7c4
[2017-05-04 10:52:10,671: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ddf373d0-b81e-494d-b58e-82bdb503c7c4
[2017-05-04 10:52:12,594: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:12,594: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:12,594: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:12,594: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:12,594: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e0f199a6-8639-40a1-9796-5974f17480ab
[2017-05-04 10:52:12,594: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e0f199a6-8639-40a1-9796-5974f17480ab
[2017-05-04 10:52:12,597: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:12,597: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:12,597: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:12,597: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:12,597: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2d92107b-3fd0-433a-a8fa-98cc7eb9e263
[2017-05-04 10:52:12,597: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2d92107b-3fd0-433a-a8fa-98cc7eb9e263
[2017-05-04 10:52:12,613: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e0f199a6-8639-40a1-9796-5974f17480ab
[2017-05-04 10:52:12,613: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e0f199a6-8639-40a1-9796-5974f17480ab
[2017-05-04 10:52:12,614: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com check_block_device {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5', 'device_type': 'zfs'}
[2017-05-04 10:52:12,622: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2d92107b-3fd0-433a-a8fa-98cc7eb9e263
[2017-05-04 10:52:12,622: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2d92107b-3fd0-433a-a8fa-98cc7eb9e263
[2017-05-04 10:52:12,625: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com check_block_device {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3', 'device_type': 'zfs'}
[2017-05-04 10:52:12,629: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 62d878d0-f50e-4e28-bb8f-dfded3dee20e
[2017-05-04 10:52:12,629: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 62d878d0-f50e-4e28-bb8f-dfded3dee20e
[2017-05-04 10:52:12,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c4e1a16f-698d-4ef8-a89b-ff9af9c6eb5f
[2017-05-04 10:52:12,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c4e1a16f-698d-4ef8-a89b-ff9af9c6eb5f
[2017-05-04 10:52:12,667: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:12,667: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:12,667: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:12,667: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:12,667: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c5029cc7-9fb6-4e08-8a55-188e635975f4
[2017-05-04 10:52:12,667: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c5029cc7-9fb6-4e08-8a55-188e635975f4
[2017-05-04 10:52:12,705: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c5029cc7-9fb6-4e08-8a55-188e635975f4
[2017-05-04 10:52:12,705: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c5029cc7-9fb6-4e08-8a55-188e635975f4
[2017-05-04 10:52:12,736: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:12,736: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:12,736: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:12,736: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:12,736: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c4e1a16f-698d-4ef8-a89b-ff9af9c6eb5f
[2017-05-04 10:52:12,736: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c4e1a16f-698d-4ef8-a89b-ff9af9c6eb5f
[2017-05-04 10:52:12,775: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c4e1a16f-698d-4ef8-a89b-ff9af9c6eb5f
[2017-05-04 10:52:12,775: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c4e1a16f-698d-4ef8-a89b-ff9af9c6eb5f
[2017-05-04 10:52:12,776: INFO/job] MGS passed pre-format check, allowing subsequent reformats
[2017-05-04 10:52:12,914: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com format_target {'backfstype': 'zfs', 'target_name': u'MGS', 'target_types': 'mgs', 'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3', 'device_type': 'zfs', 'failnode': [u'10.14.81.225@tcp0']}
[2017-05-04 10:52:12,926: INFO/job] Updating mtm volume_nodes for target testfs-MDT0000
[2017-05-04 10:52:12,935: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2d8ca70d-1bde-444c-b843-d0ca3a7bf818
[2017-05-04 10:52:12,935: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2d8ca70d-1bde-444c-b843-d0ca3a7bf818
[2017-05-04 10:52:13,327: INFO/job_scheduler] Job 55 finished 5 steps successfully
[2017-05-04 10:52:13,954: INFO/job] StatefulObject.set_state testfs-MDT0000 unformatted->formatted (intentional=True) 139993922403088
[2017-05-04 10:52:13,962: INFO/job] Job 55: StateChangeJob complete, setting state formatted on testfs-MDT0000
[2017-05-04 10:52:13,962: INFO/job] Job 55: not_deleted=True
[2017-05-04 10:52:13,969: INFO/job_scheduler] Job 55: complete_job: Updating cache
[2017-05-04 10:52:13,970: INFO/job_scheduler] Job 55: locked_item 139993922403088 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-04 10:52:13,972: INFO/job_scheduler] Job 55: locked_item 139993453533136 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-04 10:52:13,974: INFO/job_scheduler] Job 55: locked_item 139993453533328 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-04 10:52:13,975: INFO/job_scheduler] Job 55 completing (errored=False, cancelled=False)
[2017-05-04 10:52:14,009: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:14,009: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:14,009: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:14,009: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:14,009: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 62d878d0-f50e-4e28-bb8f-dfded3dee20e
[2017-05-04 10:52:14,009: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 62d878d0-f50e-4e28-bb8f-dfded3dee20e
[2017-05-04 10:52:14,048: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 62d878d0-f50e-4e28-bb8f-dfded3dee20e
[2017-05-04 10:52:14,048: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 62d878d0-f50e-4e28-bb8f-dfded3dee20e
[2017-05-04 10:52:14,050: INFO/job] testfs-OST0000 passed pre-format check, allowing subsequent reformats
[2017-05-04 10:52:14,153: INFO/job_scheduler] run_next: 0 runnable jobs of (13 pending, 3 tasked)
[2017-05-04 10:52:14,180: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com format_target {'index': 0, 'backfstype': 'zfs', 'target_name': u'testfs-OST0000', 'mgsnode': ((u'10.14.81.224@tcp0',), (u'10.14.81.225@tcp0',)), 'target_types': 'ost', 'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5', 'device_type': 'zfs', 'failnode': [u'10.14.81.227@tcp0'], 'fsname': u'testfs'}
[2017-05-04 10:52:14,193: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9ab2ebda-b9d7-4d35-9cb3-4168b1c30b05
[2017-05-04 10:52:14,193: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9ab2ebda-b9d7-4d35-9cb3-4168b1c30b05
[2017-05-04 10:52:19,424: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:19,424: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:19,424: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:19,424: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:19,425: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2d8ca70d-1bde-444c-b843-d0ca3a7bf818
[2017-05-04 10:52:19,425: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2d8ca70d-1bde-444c-b843-d0ca3a7bf818
[2017-05-04 10:52:19,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2d8ca70d-1bde-444c-b843-d0ca3a7bf818
[2017-05-04 10:52:19,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2d8ca70d-1bde-444c-b843-d0ca3a7bf818
[2017-05-04 10:52:19,504: INFO/job] Updating mtm volume_nodes for target MGS
[2017-05-04 10:52:21,797: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:21,797: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:21,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:21,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:21,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ddf373d0-b81e-494d-b58e-82bdb503c7c4
[2017-05-04 10:52:21,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ddf373d0-b81e-494d-b58e-82bdb503c7c4
[2017-05-04 10:52:21,838: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ddf373d0-b81e-494d-b58e-82bdb503c7c4
[2017-05-04 10:52:21,838: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ddf373d0-b81e-494d-b58e-82bdb503c7c4
[2017-05-04 10:52:21,872: INFO/job] Updating mtm volume_nodes for target testfs-OST0001
[2017-05-04 10:52:22,578: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:22,578: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:22,578: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:22,578: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:22,578: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9ab2ebda-b9d7-4d35-9cb3-4168b1c30b05
[2017-05-04 10:52:22,578: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9ab2ebda-b9d7-4d35-9cb3-4168b1c30b05
[2017-05-04 10:52:22,609: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9ab2ebda-b9d7-4d35-9cb3-4168b1c30b05
[2017-05-04 10:52:22,609: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9ab2ebda-b9d7-4d35-9cb3-4168b1c30b05
[2017-05-04 10:52:22,634: INFO/job] Updating mtm volume_nodes for target testfs-OST0000
[2017-05-04 10:52:56,798: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 10:52:56,798: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-04 10:52:56,819: INFO/command_plan] add_jobs: created Job 71 (Trigger plugin poll for linux plugins)
[2017-05-04 10:52:56,867: INFO/job_scheduler] run_next: 1 runnable jobs of (14 pending, 3 tasked)
[2017-05-04 10:52:56,870: INFO/job_scheduler] Job 71: RunJobThread.run
[2017-05-04 10:52:56,871: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-04 10:52:56,885: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d10d68dd-b712-4033-a562-9c00de509056
[2017-05-04 10:52:56,885: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d10d68dd-b712-4033-a562-9c00de509056
[2017-05-04 10:52:56,885: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-04 10:52:56,900: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 60cb32c7-d5f4-47f3-b953-9cb293986d9c
[2017-05-04 10:52:56,900: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 60cb32c7-d5f4-47f3-b953-9cb293986d9c
[2017-05-04 10:52:56,902: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-04 10:52:56,926: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4cb82f13-fb84-40b8-b64c-9bdb14714b2a
[2017-05-04 10:52:56,926: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4cb82f13-fb84-40b8-b64c-9bdb14714b2a
[2017-05-04 10:52:56,964: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:56,964: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:56,964: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:56,964: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:56,964: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d10d68dd-b712-4033-a562-9c00de509056
[2017-05-04 10:52:56,964: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d10d68dd-b712-4033-a562-9c00de509056
[2017-05-04 10:52:56,965: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d10d68dd-b712-4033-a562-9c00de509056
[2017-05-04 10:52:56,965: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d10d68dd-b712-4033-a562-9c00de509056
[2017-05-04 10:52:56,965: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-04 10:52:56,973: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:56,973: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:56,973: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:56,973: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:56,973: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 60cb32c7-d5f4-47f3-b953-9cb293986d9c
[2017-05-04 10:52:56,973: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 60cb32c7-d5f4-47f3-b953-9cb293986d9c
[2017-05-04 10:52:56,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:56,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:56,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:56,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:56,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4cb82f13-fb84-40b8-b64c-9bdb14714b2a
[2017-05-04 10:52:56,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4cb82f13-fb84-40b8-b64c-9bdb14714b2a
[2017-05-04 10:52:56,978: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 60cb32c7-d5f4-47f3-b953-9cb293986d9c
[2017-05-04 10:52:56,978: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 60cb32c7-d5f4-47f3-b953-9cb293986d9c
[2017-05-04 10:52:56,978: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-04 10:52:56,991: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4cb82f13-fb84-40b8-b64c-9bdb14714b2a
[2017-05-04 10:52:56,991: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4cb82f13-fb84-40b8-b64c-9bdb14714b2a
[2017-05-04 10:52:56,991: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-04 10:52:56,991: INFO/job_scheduler] Job 71 finished 1 steps successfully
[2017-05-04 10:52:57,110: INFO/job_scheduler] Job 71: complete_job: Updating cache
[2017-05-04 10:52:57,110: INFO/job_scheduler] Job 71 completing (errored=False, cancelled=False)
[2017-05-04 10:52:57,142: INFO/job_scheduler] run_next: 0 runnable jobs of (13 pending, 3 tasked)
[2017-05-04 10:52:58,014: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 10:52:58,016: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-04 10:52:58,050: INFO/command_plan] add_jobs: created Job 72 (Trigger plugin poll for linux plugins)
[2017-05-04 10:52:58,070: INFO/job_scheduler] run_next: 1 runnable jobs of (14 pending, 3 tasked)
[2017-05-04 10:52:58,084: INFO/job_scheduler] Job 72: RunJobThread.run
[2017-05-04 10:52:58,097: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-04 10:52:58,122: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-04 10:52:58,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3118cafc-e2a0-47e2-a580-ee31c0f0587a
[2017-05-04 10:52:58,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3118cafc-e2a0-47e2-a580-ee31c0f0587a
[2017-05-04 10:52:58,148: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-04 10:52:58,153: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5523db73-d7f3-4b7a-b81f-dbd95ade3d3d
[2017-05-04 10:52:58,153: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5523db73-d7f3-4b7a-b81f-dbd95ade3d3d
[2017-05-04 10:52:58,182: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 65291b1a-424e-405a-888f-c16fd3eda181
[2017-05-04 10:52:58,182: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 65291b1a-424e-405a-888f-c16fd3eda181
[2017-05-04 10:52:58,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:58,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:58,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:58,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:58,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3118cafc-e2a0-47e2-a580-ee31c0f0587a
[2017-05-04 10:52:58,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3118cafc-e2a0-47e2-a580-ee31c0f0587a
[2017-05-04 10:52:58,209: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:58,209: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:58,210: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:58,210: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:58,210: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5523db73-d7f3-4b7a-b81f-dbd95ade3d3d
[2017-05-04 10:52:58,210: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5523db73-d7f3-4b7a-b81f-dbd95ade3d3d
[2017-05-04 10:52:58,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3118cafc-e2a0-47e2-a580-ee31c0f0587a
[2017-05-04 10:52:58,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3118cafc-e2a0-47e2-a580-ee31c0f0587a
[2017-05-04 10:52:58,212: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-04 10:52:58,236: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5523db73-d7f3-4b7a-b81f-dbd95ade3d3d
[2017-05-04 10:52:58,236: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5523db73-d7f3-4b7a-b81f-dbd95ade3d3d
[2017-05-04 10:52:58,236: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-04 10:52:58,283: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:58,283: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:58,283: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:58,283: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:52:58,283: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 65291b1a-424e-405a-888f-c16fd3eda181
[2017-05-04 10:52:58,283: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 65291b1a-424e-405a-888f-c16fd3eda181
[2017-05-04 10:52:58,301: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 65291b1a-424e-405a-888f-c16fd3eda181
[2017-05-04 10:52:58,301: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 65291b1a-424e-405a-888f-c16fd3eda181
[2017-05-04 10:52:58,301: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-04 10:52:58,302: INFO/job_scheduler] Job 72 finished 1 steps successfully
[2017-05-04 10:52:58,360: INFO/job_scheduler] Job 72: complete_job: Updating cache
[2017-05-04 10:52:58,360: INFO/job_scheduler] Job 72 completing (errored=False, cancelled=False)
[2017-05-04 10:52:58,405: INFO/job_scheduler] run_next: 0 runnable jobs of (13 pending, 3 tasked)
[2017-05-04 10:52:59,245: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 10:52:59,245: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-04 10:52:59,375: INFO/command_plan] add_jobs: created Job 73 (Trigger plugin poll for linux plugins)
[2017-05-04 10:52:59,406: INFO/job_scheduler] run_next: 1 runnable jobs of (14 pending, 3 tasked)
[2017-05-04 10:52:59,422: INFO/job_scheduler] Job 73: RunJobThread.run
[2017-05-04 10:52:59,425: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-04 10:52:59,433: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-04 10:52:59,433: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-04 10:52:59,438: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc deff0d74-9550-4569-b889-6ff4a83c3ea9
[2017-05-04 10:52:59,438: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc deff0d74-9550-4569-b889-6ff4a83c3ea9
[2017-05-04 10:52:59,453: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1876a0c1-669e-4cc7-9f50-d63f388788bd
[2017-05-04 10:52:59,453: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1876a0c1-669e-4cc7-9f50-d63f388788bd
[2017-05-04 10:52:59,473: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e9c952e8-a567-4d51-9e46-83b6c09a88a8
[2017-05-04 10:52:59,473: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e9c952e8-a567-4d51-9e46-83b6c09a88a8
[2017-05-04 10:52:59,537: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:59,537: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:59,537: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:59,537: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:52:59,537: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc deff0d74-9550-4569-b889-6ff4a83c3ea9
[2017-05-04 10:52:59,537: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc deff0d74-9550-4569-b889-6ff4a83c3ea9
[2017-05-04 10:52:59,540: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:59,540: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:59,540: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:59,540: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:52:59,540: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1876a0c1-669e-4cc7-9f50-d63f388788bd
[2017-05-04 10:52:59,540: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1876a0c1-669e-4cc7-9f50-d63f388788bd
[2017-05-04 10:52:59,544: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:59,544: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:59,544: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:59,544: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:52:59,544: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e9c952e8-a567-4d51-9e46-83b6c09a88a8
[2017-05-04 10:52:59,544: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e9c952e8-a567-4d51-9e46-83b6c09a88a8
[2017-05-04 10:52:59,552: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e9c952e8-a567-4d51-9e46-83b6c09a88a8
[2017-05-04 10:52:59,552: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e9c952e8-a567-4d51-9e46-83b6c09a88a8
[2017-05-04 10:52:59,552: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-04 10:52:59,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc deff0d74-9550-4569-b889-6ff4a83c3ea9
[2017-05-04 10:52:59,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc deff0d74-9550-4569-b889-6ff4a83c3ea9
[2017-05-04 10:52:59,566: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-04 10:52:59,582: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1876a0c1-669e-4cc7-9f50-d63f388788bd
[2017-05-04 10:52:59,582: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1876a0c1-669e-4cc7-9f50-d63f388788bd
[2017-05-04 10:52:59,582: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-04 10:52:59,582: INFO/job_scheduler] Job 73 finished 1 steps successfully
[2017-05-04 10:52:59,645: INFO/job_scheduler] Job 73: complete_job: Updating cache
[2017-05-04 10:52:59,645: INFO/job_scheduler] Job 73 completing (errored=False, cancelled=False)
[2017-05-04 10:52:59,737: INFO/job_scheduler] run_next: 0 runnable jobs of (13 pending, 3 tasked)
[2017-05-04 10:52:59,837: INFO/job_scheduler] Job 54 finished 5 steps successfully
[2017-05-04 10:52:59,862: INFO/job] StatefulObject.set_state testfs-OST0000 unformatted->formatted (intentional=True) 139993452763920
[2017-05-04 10:52:59,868: INFO/job] Job 54: StateChangeJob complete, setting state formatted on testfs-OST0000
[2017-05-04 10:52:59,868: INFO/job] Job 54: not_deleted=True
[2017-05-04 10:52:59,874: INFO/job_scheduler] Job 54: complete_job: Updating cache
[2017-05-04 10:52:59,874: INFO/job_scheduler] Job 54: locked_item 139993452763920 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-04 10:52:59,877: INFO/job_scheduler] Job 54: locked_item 139993453252432 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-04 10:52:59,879: INFO/job_scheduler] Job 54: locked_item 139993453531216 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-04 10:52:59,880: INFO/job_scheduler] Job 54 completing (errored=False, cancelled=False)
[2017-05-04 10:52:59,912: INFO/job_scheduler] run_next: 0 runnable jobs of (13 pending, 2 tasked)
[2017-05-04 10:53:00,062: INFO/job_scheduler] Job 56 finished 5 steps successfully
[2017-05-04 10:53:00,095: INFO/job] StatefulObject.set_state testfs-OST0001 unformatted->formatted (intentional=True) 139993922361296
[2017-05-04 10:53:00,101: INFO/job] Job 56: StateChangeJob complete, setting state formatted on testfs-OST0001
[2017-05-04 10:53:00,101: INFO/job] Job 56: not_deleted=True
[2017-05-04 10:53:00,107: INFO/job_scheduler] Job 56: complete_job: Updating cache
[2017-05-04 10:53:00,108: INFO/job_scheduler] Job 56: locked_item 139993922361296 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-04 10:53:00,110: INFO/job_scheduler] Job 56: locked_item 139993453534928 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-04 10:53:00,112: INFO/job_scheduler] Job 56: locked_item 139993453535120 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-04 10:53:00,113: INFO/job_scheduler] Job 56 completing (errored=False, cancelled=False)
[2017-05-04 10:53:00,145: INFO/job_scheduler] run_next: 0 runnable jobs of (13 pending, 1 tasked)
[2017-05-04 10:53:49,125: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 10:53:49,126: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-04 10:53:49,152: INFO/command_plan] add_jobs: created Job 74 (Trigger plugin poll for linux plugins)
[2017-05-04 10:53:49,171: INFO/job_scheduler] run_next: 1 runnable jobs of (14 pending, 1 tasked)
[2017-05-04 10:53:49,174: INFO/job_scheduler] Job 74: RunJobThread.run
[2017-05-04 10:53:49,175: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-04 10:53:49,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 13e8873b-528b-4a76-a537-dea4f9d48b36
[2017-05-04 10:53:49,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 13e8873b-528b-4a76-a537-dea4f9d48b36
[2017-05-04 10:53:49,190: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-04 10:53:49,204: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dd1ef9b7-da1a-4159-bed8-b0103f1f487b
[2017-05-04 10:53:49,204: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dd1ef9b7-da1a-4159-bed8-b0103f1f487b
[2017-05-04 10:53:49,205: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-04 10:53:49,219: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9406fda3-baea-4532-adcf-4658024a1fda
[2017-05-04 10:53:49,219: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9406fda3-baea-4532-adcf-4658024a1fda
[2017-05-04 10:53:49,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:53:49,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:53:49,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:53:49,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:53:49,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 13e8873b-528b-4a76-a537-dea4f9d48b36
[2017-05-04 10:53:49,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 13e8873b-528b-4a76-a537-dea4f9d48b36
[2017-05-04 10:53:49,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 13e8873b-528b-4a76-a537-dea4f9d48b36
[2017-05-04 10:53:49,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 13e8873b-528b-4a76-a537-dea4f9d48b36
[2017-05-04 10:53:49,272: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-04 10:53:49,296: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:53:49,296: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:53:49,296: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:53:49,296: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:53:49,296: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9406fda3-baea-4532-adcf-4658024a1fda
[2017-05-04 10:53:49,296: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9406fda3-baea-4532-adcf-4658024a1fda
[2017-05-04 10:53:49,299: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:53:49,299: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:53:49,299: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:53:49,299: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:53:49,299: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dd1ef9b7-da1a-4159-bed8-b0103f1f487b
[2017-05-04 10:53:49,299: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dd1ef9b7-da1a-4159-bed8-b0103f1f487b
[2017-05-04 10:53:49,328: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dd1ef9b7-da1a-4159-bed8-b0103f1f487b
[2017-05-04 10:53:49,328: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dd1ef9b7-da1a-4159-bed8-b0103f1f487b
[2017-05-04 10:53:49,328: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-04 10:53:49,339: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9406fda3-baea-4532-adcf-4658024a1fda
[2017-05-04 10:53:49,339: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9406fda3-baea-4532-adcf-4658024a1fda
[2017-05-04 10:53:49,339: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-04 10:53:49,339: INFO/job_scheduler] Job 74 finished 1 steps successfully
[2017-05-04 10:53:49,386: INFO/job_scheduler] Job 74: complete_job: Updating cache
[2017-05-04 10:53:49,386: INFO/job_scheduler] Job 74 completing (errored=False, cancelled=False)
[2017-05-04 10:53:49,439: INFO/job_scheduler] run_next: 0 runnable jobs of (13 pending, 1 tasked)
[2017-05-04 10:53:49,870: INFO/job_scheduler] Job 57 finished 5 steps successfully
[2017-05-04 10:53:49,911: INFO/job] StatefulObject.set_state MGS unformatted->formatted (intentional=True) 139993453117840
[2017-05-04 10:53:49,917: INFO/job] Job 57: StateChangeJob complete, setting state formatted on MGS
[2017-05-04 10:53:49,917: INFO/job] Job 57: not_deleted=True
[2017-05-04 10:53:49,924: INFO/job_scheduler] Job 57: complete_job: Updating cache
[2017-05-04 10:53:49,924: INFO/job_scheduler] Job 57: locked_item 139993453117840 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-04 10:53:49,927: INFO/job_scheduler] Job 57: locked_item 139993453561296 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-04 10:53:49,933: INFO/job_scheduler] Job 57: locked_item 139993453561488 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-04 10:53:49,934: INFO/job_scheduler] Job 57 completing (errored=False, cancelled=False)
[2017-05-04 10:53:49,972: INFO/job_scheduler] run_next: 1 runnable jobs of (13 pending, 0 tasked)
[2017-05-04 10:53:50,033: INFO/job_scheduler] Job 58: RunJobThread.run
[2017-05-04 10:53:50,035: INFO/job_scheduler] Job 58 finished 1 steps successfully
[2017-05-04 10:53:50,061: INFO/job] StatefulObject.set_state MGS formatted->registered (intentional=True) 139993453117840
[2017-05-04 10:53:50,066: INFO/job] Job 58: StateChangeJob complete, setting state registered on MGS
[2017-05-04 10:53:50,066: INFO/job] Job 58: not_deleted=True
[2017-05-04 10:53:50,066: INFO/job_scheduler] Job 58: complete_job: Updating cache
[2017-05-04 10:53:50,066: INFO/job_scheduler] Job 58: locked_item 139993453117840 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-04 10:53:50,069: INFO/job_scheduler] Job 58 completing (errored=False, cancelled=False)
[2017-05-04 10:53:50,086: INFO/job_scheduler] run_next: 1 runnable jobs of (12 pending, 0 tasked)
[2017-05-04 10:53:50,133: INFO/job_scheduler] Job 59: RunJobThread.run
[2017-05-04 10:53:50,133: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-05-04 10:53:50,148: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1be8a664-d1a3-4cf8-9d31-d8ef7eaa4600
[2017-05-04 10:53:50,148: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1be8a664-d1a3-4cf8-9d31-d8ef7eaa4600
[2017-05-04 10:53:51,229: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:53:51,229: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:53:51,229: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:53:51,229: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:53:51,229: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1be8a664-d1a3-4cf8-9d31-d8ef7eaa4600
[2017-05-04 10:53:51,229: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1be8a664-d1a3-4cf8-9d31-d8ef7eaa4600
[2017-05-04 10:53:51,262: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1be8a664-d1a3-4cf8-9d31-d8ef7eaa4600
[2017-05-04 10:53:51,262: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1be8a664-d1a3-4cf8-9d31-d8ef7eaa4600
[2017-05-04 10:53:51,263: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-05-04 10:53:51,280: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 86874d71-1c95-485d-86c9-6060b7c44b60
[2017-05-04 10:53:51,280: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 86874d71-1c95-485d-86c9-6060b7c44b60
[2017-05-04 10:53:52,397: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:53:52,397: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:53:52,397: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:53:52,397: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:53:52,397: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 86874d71-1c95-485d-86c9-6060b7c44b60
[2017-05-04 10:53:52,397: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 86874d71-1c95-485d-86c9-6060b7c44b60
[2017-05-04 10:53:52,444: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 86874d71-1c95-485d-86c9-6060b7c44b60
[2017-05-04 10:53:52,444: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 86874d71-1c95-485d-86c9-6060b7c44b60
[2017-05-04 10:53:52,446: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_target_store {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3/MGS', 'device_type': 'zfs', 'mount_point': u'/mnt/MGS', 'uuid': u'10805300282825369826', 'backfstype': 'zfs'}
[2017-05-04 10:53:52,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f5fece44-e863-40e1-802a-3433e641604f
[2017-05-04 10:53:52,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f5fece44-e863-40e1-802a-3433e641604f
[2017-05-04 10:53:52,567: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:53:52,567: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:53:52,567: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:53:52,567: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:53:52,568: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f5fece44-e863-40e1-802a-3433e641604f
[2017-05-04 10:53:52,568: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f5fece44-e863-40e1-802a-3433e641604f
[2017-05-04 10:53:52,572: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f5fece44-e863-40e1-802a-3433e641604f
[2017-05-04 10:53:52,572: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f5fece44-e863-40e1-802a-3433e641604f
[2017-05-04 10:53:52,573: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_target_store {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3/MGS', 'device_type': 'zfs', 'mount_point': u'/mnt/MGS', 'uuid': u'10805300282825369826', 'backfstype': 'zfs'}
[2017-05-04 10:53:52,586: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ebf21a09-e24e-4934-8a8c-a2c7204a244e
[2017-05-04 10:53:52,586: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ebf21a09-e24e-4934-8a8c-a2c7204a244e
[2017-05-04 10:53:52,639: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:53:52,639: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:53:52,639: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:53:52,639: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:53:52,639: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ebf21a09-e24e-4934-8a8c-a2c7204a244e
[2017-05-04 10:53:52,639: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ebf21a09-e24e-4934-8a8c-a2c7204a244e
[2017-05-04 10:53:52,652: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ebf21a09-e24e-4934-8a8c-a2c7204a244e
[2017-05-04 10:53:52,652: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ebf21a09-e24e-4934-8a8c-a2c7204a244e
[2017-05-04 10:53:52,653: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3/MGS', 'mount_point': u'/mnt/MGS', 'uuid': u'10805300282825369826', 'ha_label': u'MGS_9879fb', 'primary': True}
[2017-05-04 10:53:52,669: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bb124996-6de1-403c-81ad-85bedf802f67
[2017-05-04 10:53:52,669: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bb124996-6de1-403c-81ad-85bedf802f67
[2017-05-04 10:53:55,758: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:53:55,758: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:53:55,759: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:53:55,759: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:53:55,759: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bb124996-6de1-403c-81ad-85bedf802f67
[2017-05-04 10:53:55,759: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bb124996-6de1-403c-81ad-85bedf802f67
[2017-05-04 10:53:55,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bb124996-6de1-403c-81ad-85bedf802f67
[2017-05-04 10:53:55,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bb124996-6de1-403c-81ad-85bedf802f67
[2017-05-04 10:53:55,786: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3/MGS', 'mount_point': u'/mnt/MGS', 'uuid': u'10805300282825369826', 'ha_label': u'MGS_9879fb', 'primary': False}
[2017-05-04 10:53:55,799: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ce0e5afd-e6af-49ed-809f-264ab4170908
[2017-05-04 10:53:55,799: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ce0e5afd-e6af-49ed-809f-264ab4170908
[2017-05-04 10:53:56,874: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:53:56,874: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:53:56,875: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:53:56,875: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:53:56,875: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ce0e5afd-e6af-49ed-809f-264ab4170908
[2017-05-04 10:53:56,875: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ce0e5afd-e6af-49ed-809f-264ab4170908
[2017-05-04 10:53:56,913: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ce0e5afd-e6af-49ed-809f-264ab4170908
[2017-05-04 10:53:56,913: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ce0e5afd-e6af-49ed-809f-264ab4170908
[2017-05-04 10:53:56,914: INFO/job_scheduler] Job 59 finished 6 steps successfully
[2017-05-04 10:53:56,935: INFO/job] StatefulObject.set_state MGS registered->unmounted (intentional=True) 139993453117840
[2017-05-04 10:53:56,940: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on MGS at severity 30
[2017-05-04 10:53:56,954: INFO/job] Job 59: StateChangeJob complete, setting state unmounted on MGS
[2017-05-04 10:53:56,955: INFO/job] Job 59: not_deleted=True
[2017-05-04 10:53:56,956: INFO/job_scheduler] Job 59: complete_job: Updating cache
[2017-05-04 10:53:56,957: INFO/job_scheduler] Job 59: locked_item 139993453117840 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-04 10:53:56,962: INFO/job_scheduler] Job 59 completing (errored=False, cancelled=False)
[2017-05-04 10:53:56,985: INFO/job_scheduler] run_next: 1 runnable jobs of (11 pending, 0 tasked)
[2017-05-04 10:53:57,026: INFO/job_scheduler] Job 60: RunJobThread.run
[2017-05-04 10:53:57,029: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3/MGS', 'pacemaker_ha_operation': False, 'device_type': 'zfs'}
[2017-05-04 10:53:57,029: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3/MGS', 'pacemaker_ha_operation': False, 'device_type': 'zfs'}
[2017-05-04 10:53:57,037: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3/MGS', 'pacemaker_ha_operation': False, 'device_type': 'zfs'}
[2017-05-04 10:53:57,047: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 40ee80cf-760a-47e1-b6d9-f6af6b1062e8
[2017-05-04 10:53:57,047: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 40ee80cf-760a-47e1-b6d9-f6af6b1062e8
[2017-05-04 10:53:57,062: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 790e1df5-e2ec-4fa2-93d5-235177ad6372
[2017-05-04 10:53:57,062: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 790e1df5-e2ec-4fa2-93d5-235177ad6372
[2017-05-04 10:53:57,075: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 704eb732-5a9a-4a9b-896c-2520024d90ee
[2017-05-04 10:53:57,075: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 704eb732-5a9a-4a9b-896c-2520024d90ee
[2017-05-04 10:53:58,946: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 1 tasked)
[2017-05-04 10:53:59,176: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:53:59,176: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:53:59,177: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:53:59,177: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:53:59,177: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 704eb732-5a9a-4a9b-896c-2520024d90ee
[2017-05-04 10:53:59,177: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 704eb732-5a9a-4a9b-896c-2520024d90ee
[2017-05-04 10:53:59,190: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 704eb732-5a9a-4a9b-896c-2520024d90ee
[2017-05-04 10:53:59,190: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 704eb732-5a9a-4a9b-896c-2520024d90ee
[2017-05-04 10:53:59,193: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 1 tasked)
[2017-05-04 10:54:00,141: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:00,141: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:00,142: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:00,142: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:00,142: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 40ee80cf-760a-47e1-b6d9-f6af6b1062e8
[2017-05-04 10:54:00,142: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 40ee80cf-760a-47e1-b6d9-f6af6b1062e8
[2017-05-04 10:54:00,172: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 40ee80cf-760a-47e1-b6d9-f6af6b1062e8
[2017-05-04 10:54:00,172: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 40ee80cf-760a-47e1-b6d9-f6af6b1062e8
[2017-05-04 10:54:01,213: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:01,213: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:01,213: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:01,213: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:01,213: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 790e1df5-e2ec-4fa2-93d5-235177ad6372
[2017-05-04 10:54:01,213: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 790e1df5-e2ec-4fa2-93d5-235177ad6372
[2017-05-04 10:54:01,226: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 790e1df5-e2ec-4fa2-93d5-235177ad6372
[2017-05-04 10:54:01,226: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 790e1df5-e2ec-4fa2-93d5-235177ad6372
[2017-05-04 10:54:01,227: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com import_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3/MGS', 'pacemaker_ha_operation': False, 'device_type': 'zfs', 'validate_importable': True}
[2017-05-04 10:54:01,247: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fb7d5dbf-42f8-4430-9a78-fc2c05edceae
[2017-05-04 10:54:01,247: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fb7d5dbf-42f8-4430-9a78-fc2c05edceae
[2017-05-04 10:54:04,434: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:54:04,434: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:54:04,434: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:54:04,434: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:54:04,435: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fb7d5dbf-42f8-4430-9a78-fc2c05edceae
[2017-05-04 10:54:04,435: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fb7d5dbf-42f8-4430-9a78-fc2c05edceae
[2017-05-04 10:54:04,462: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fb7d5dbf-42f8-4430-9a78-fc2c05edceae
[2017-05-04 10:54:04,462: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fb7d5dbf-42f8-4430-9a78-fc2c05edceae
[2017-05-04 10:54:04,462: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_target {'ha_label': u'MGS_9879fb'}
[2017-05-04 10:54:04,475: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f8ed5fac-b2e8-414d-a495-56d45fb740b0
[2017-05-04 10:54:04,475: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f8ed5fac-b2e8-414d-a495-56d45fb740b0
[2017-05-04 10:54:19,197: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 1 tasked)
[2017-05-04 10:54:20,650: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:54:20,650: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:54:20,651: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:54:20,651: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:54:20,651: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f8ed5fac-b2e8-414d-a495-56d45fb740b0
[2017-05-04 10:54:20,651: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f8ed5fac-b2e8-414d-a495-56d45fb740b0
[2017-05-04 10:54:20,693: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f8ed5fac-b2e8-414d-a495-56d45fb740b0
[2017-05-04 10:54:20,693: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f8ed5fac-b2e8-414d-a495-56d45fb740b0
[2017-05-04 10:54:20,694: INFO/job_scheduler] Job 60 finished 1 steps successfully
[2017-05-04 10:54:20,885: INFO/job] StatefulObject.set_state MGS unmounted->mounted (intentional=True) 139993453117840
[2017-05-04 10:54:20,928: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 20
[2017-05-04 10:54:20,942: INFO/job] Job 60: StateChangeJob complete, setting state mounted on MGS
[2017-05-04 10:54:20,942: INFO/job] Job 60: not_deleted=True
[2017-05-04 10:54:20,942: INFO/job_scheduler] Job 60: complete_job: Updating cache
[2017-05-04 10:54:20,942: INFO/job_scheduler] Job 60: locked_item 139993453117840 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-04 10:54:20,945: INFO/job_scheduler] Job 60 completing (errored=False, cancelled=False)
[2017-05-04 10:54:20,994: INFO/job_scheduler] run_next: 1 runnable jobs of (10 pending, 0 tasked)
[2017-05-04 10:54:21,027: INFO/job_scheduler] Job 61: RunJobThread.run
[2017-05-04 10:54:21,037: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com register_target {'device_path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'mount_point': u'/mnt/testfs-MDT0000', 'backfstype': 'ldiskfs'}
[2017-05-04 10:54:21,061: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bede1c54-d62d-4919-b952-49efda5470c2
[2017-05-04 10:54:21,061: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bede1c54-d62d-4919-b952-49efda5470c2
[2017-05-04 10:54:22,042: INFO/job_scheduler] notify: Dropping update of 1 (MGS) because it has been updated since
[2017-05-04 10:54:22,043: INFO/job_scheduler] run_next: 0 runnable jobs of (9 pending, 1 tasked)
[2017-05-04 10:54:26,166: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:54:26,166: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:54:26,167: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:54:26,167: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:54:26,167: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bede1c54-d62d-4919-b952-49efda5470c2
[2017-05-04 10:54:26,167: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bede1c54-d62d-4919-b952-49efda5470c2
[2017-05-04 10:54:26,175: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bede1c54-d62d-4919-b952-49efda5470c2
[2017-05-04 10:54:26,175: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bede1c54-d62d-4919-b952-49efda5470c2
[2017-05-04 10:54:26,177: INFO/job_scheduler] Job 61 finished 2 steps successfully
[2017-05-04 10:54:26,268: INFO/job] StatefulObject.set_state testfs-MDT0000 formatted->registered (intentional=True) 139993922403088
[2017-05-04 10:54:26,273: INFO/job] Job 61: StateChangeJob complete, setting state registered on testfs-MDT0000
[2017-05-04 10:54:26,273: INFO/job] Job 61: not_deleted=True
[2017-05-04 10:54:26,273: INFO/job_scheduler] Job 61: complete_job: Updating cache
[2017-05-04 10:54:26,274: INFO/job_scheduler] Job 61: locked_item 139993922403088 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-04 10:54:26,277: INFO/job_scheduler] Job 61 completing (errored=False, cancelled=False)
[2017-05-04 10:54:26,302: INFO/job_scheduler] run_next: 1 runnable jobs of (9 pending, 0 tasked)
[2017-05-04 10:54:26,352: INFO/job_scheduler] Job 62: RunJobThread.run
[2017-05-04 10:54:26,352: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-05-04 10:54:26,368: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 17d940a9-e636-4baa-b638-02f850286ad2
[2017-05-04 10:54:26,368: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 17d940a9-e636-4baa-b638-02f850286ad2
[2017-05-04 10:54:27,470: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:54:27,470: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:54:27,471: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:54:27,471: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:54:27,471: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 17d940a9-e636-4baa-b638-02f850286ad2
[2017-05-04 10:54:27,471: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 17d940a9-e636-4baa-b638-02f850286ad2
[2017-05-04 10:54:27,482: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 17d940a9-e636-4baa-b638-02f850286ad2
[2017-05-04 10:54:27,482: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 17d940a9-e636-4baa-b638-02f850286ad2
[2017-05-04 10:54:27,483: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-05-04 10:54:27,502: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 66c7d881-aec0-406b-a393-89ac85d95a7a
[2017-05-04 10:54:27,502: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 66c7d881-aec0-406b-a393-89ac85d95a7a
[2017-05-04 10:54:28,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:54:28,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:54:28,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:54:28,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:54:28,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 66c7d881-aec0-406b-a393-89ac85d95a7a
[2017-05-04 10:54:28,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 66c7d881-aec0-406b-a393-89ac85d95a7a
[2017-05-04 10:54:28,616: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 66c7d881-aec0-406b-a393-89ac85d95a7a
[2017-05-04 10:54:28,616: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 66c7d881-aec0-406b-a393-89ac85d95a7a
[2017-05-04 10:54:28,617: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_target_store {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux', 'mount_point': u'/mnt/testfs-MDT0000', 'uuid': u'2c04a2d1-779d-4874-a107-29716e4d9907', 'backfstype': 'ldiskfs'}
[2017-05-04 10:54:28,630: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ea9cdf31-3967-4ae7-9822-7783572b13e6
[2017-05-04 10:54:28,630: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ea9cdf31-3967-4ae7-9822-7783572b13e6
[2017-05-04 10:54:28,730: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:54:28,730: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:54:28,731: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:54:28,731: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:54:28,731: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ea9cdf31-3967-4ae7-9822-7783572b13e6
[2017-05-04 10:54:28,731: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ea9cdf31-3967-4ae7-9822-7783572b13e6
[2017-05-04 10:54:28,746: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ea9cdf31-3967-4ae7-9822-7783572b13e6
[2017-05-04 10:54:28,746: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ea9cdf31-3967-4ae7-9822-7783572b13e6
[2017-05-04 10:54:28,748: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_target_store {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux', 'mount_point': u'/mnt/testfs-MDT0000', 'uuid': u'2c04a2d1-779d-4874-a107-29716e4d9907', 'backfstype': 'ldiskfs'}
[2017-05-04 10:54:28,760: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 12aee0cf-5a1f-458d-a3b3-e23ba4a1af58
[2017-05-04 10:54:28,760: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 12aee0cf-5a1f-458d-a3b3-e23ba4a1af58
[2017-05-04 10:54:28,848: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:54:28,848: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:54:28,848: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:54:28,848: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:54:28,849: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 12aee0cf-5a1f-458d-a3b3-e23ba4a1af58
[2017-05-04 10:54:28,849: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 12aee0cf-5a1f-458d-a3b3-e23ba4a1af58
[2017-05-04 10:54:28,875: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 12aee0cf-5a1f-458d-a3b3-e23ba4a1af58
[2017-05-04 10:54:28,875: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 12aee0cf-5a1f-458d-a3b3-e23ba4a1af58
[2017-05-04 10:54:28,877: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'mount_point': u'/mnt/testfs-MDT0000', 'uuid': u'2c04a2d1-779d-4874-a107-29716e4d9907', 'ha_label': u'testfs-MDT0000_3b90dd', 'primary': True}
[2017-05-04 10:54:28,888: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 82492316-ea19-4626-9c2f-ec581dd89270
[2017-05-04 10:54:28,888: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 82492316-ea19-4626-9c2f-ec581dd89270
[2017-05-04 10:54:31,987: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:54:31,987: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:54:31,987: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:54:31,987: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:54:31,987: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 82492316-ea19-4626-9c2f-ec581dd89270
[2017-05-04 10:54:31,987: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 82492316-ea19-4626-9c2f-ec581dd89270
[2017-05-04 10:54:32,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 82492316-ea19-4626-9c2f-ec581dd89270
[2017-05-04 10:54:32,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 82492316-ea19-4626-9c2f-ec581dd89270
[2017-05-04 10:54:32,006: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'mount_point': u'/mnt/testfs-MDT0000', 'uuid': u'2c04a2d1-779d-4874-a107-29716e4d9907', 'ha_label': u'testfs-MDT0000_3b90dd', 'primary': False}
[2017-05-04 10:54:32,026: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a53db3d1-f666-46c7-9180-6069055dde1c
[2017-05-04 10:54:32,026: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a53db3d1-f666-46c7-9180-6069055dde1c
[2017-05-04 10:54:33,090: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:54:33,090: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:54:33,090: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:54:33,090: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:54:33,090: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a53db3d1-f666-46c7-9180-6069055dde1c
[2017-05-04 10:54:33,090: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a53db3d1-f666-46c7-9180-6069055dde1c
[2017-05-04 10:54:33,091: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a53db3d1-f666-46c7-9180-6069055dde1c
[2017-05-04 10:54:33,091: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a53db3d1-f666-46c7-9180-6069055dde1c
[2017-05-04 10:54:33,091: INFO/job_scheduler] Job 62 finished 6 steps successfully
[2017-05-04 10:54:33,110: INFO/job] StatefulObject.set_state testfs-MDT0000 registered->unmounted (intentional=True) 139993922403088
[2017-05-04 10:54:33,115: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on testfs-MDT0000 at severity 30
[2017-05-04 10:54:33,121: INFO/job] Job 62: StateChangeJob complete, setting state unmounted on testfs-MDT0000
[2017-05-04 10:54:33,121: INFO/job] Job 62: not_deleted=True
[2017-05-04 10:54:33,121: INFO/job_scheduler] Job 62: complete_job: Updating cache
[2017-05-04 10:54:33,121: INFO/job_scheduler] Job 62: locked_item 139993922403088 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-04 10:54:33,130: INFO/job_scheduler] Job 62 completing (errored=False, cancelled=False)
[2017-05-04 10:54:33,168: INFO/job_scheduler] run_next: 1 runnable jobs of (8 pending, 0 tasked)
[2017-05-04 10:54:33,211: INFO/job_scheduler] Job 63: RunJobThread.run
[2017-05-04 10:54:33,213: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'pacemaker_ha_operation': False, 'device_type': 'linux'}
[2017-05-04 10:54:33,213: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'pacemaker_ha_operation': False, 'device_type': 'linux'}
[2017-05-04 10:54:33,224: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'pacemaker_ha_operation': False, 'device_type': 'linux'}
[2017-05-04 10:54:33,231: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9068ee01-2422-4d22-bdc9-3f3fb0cbdf90
[2017-05-04 10:54:33,231: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9068ee01-2422-4d22-bdc9-3f3fb0cbdf90
[2017-05-04 10:54:33,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 046a00e6-b219-49ed-b022-2a0f80656ad8
[2017-05-04 10:54:33,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 046a00e6-b219-49ed-b022-2a0f80656ad8
[2017-05-04 10:54:33,256: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc be8899a2-3cde-4be5-ab1a-af0e78b13d7d
[2017-05-04 10:54:33,256: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc be8899a2-3cde-4be5-ab1a-af0e78b13d7d
[2017-05-04 10:54:33,310: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:33,310: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:33,310: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:33,310: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:33,310: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 046a00e6-b219-49ed-b022-2a0f80656ad8
[2017-05-04 10:54:33,310: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 046a00e6-b219-49ed-b022-2a0f80656ad8
[2017-05-04 10:54:33,312: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:33,312: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:33,312: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:33,312: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:33,312: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9068ee01-2422-4d22-bdc9-3f3fb0cbdf90
[2017-05-04 10:54:33,312: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9068ee01-2422-4d22-bdc9-3f3fb0cbdf90
[2017-05-04 10:54:33,313: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9068ee01-2422-4d22-bdc9-3f3fb0cbdf90
[2017-05-04 10:54:33,313: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9068ee01-2422-4d22-bdc9-3f3fb0cbdf90
[2017-05-04 10:54:33,314: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 046a00e6-b219-49ed-b022-2a0f80656ad8
[2017-05-04 10:54:33,314: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 046a00e6-b219-49ed-b022-2a0f80656ad8
[2017-05-04 10:54:33,332: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:54:33,332: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:54:33,332: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:54:33,332: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:54:33,332: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc be8899a2-3cde-4be5-ab1a-af0e78b13d7d
[2017-05-04 10:54:33,332: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc be8899a2-3cde-4be5-ab1a-af0e78b13d7d
[2017-05-04 10:54:33,370: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc be8899a2-3cde-4be5-ab1a-af0e78b13d7d
[2017-05-04 10:54:33,370: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc be8899a2-3cde-4be5-ab1a-af0e78b13d7d
[2017-05-04 10:54:33,371: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com import_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'pacemaker_ha_operation': False, 'device_type': 'linux', 'validate_importable': True}
[2017-05-04 10:54:33,383: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bbbd6a8c-e4aa-4e44-8e1a-befd25763f77
[2017-05-04 10:54:33,383: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bbbd6a8c-e4aa-4e44-8e1a-befd25763f77
[2017-05-04 10:54:33,450: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:54:33,450: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:54:33,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:54:33,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:54:33,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bbbd6a8c-e4aa-4e44-8e1a-befd25763f77
[2017-05-04 10:54:33,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bbbd6a8c-e4aa-4e44-8e1a-befd25763f77
[2017-05-04 10:54:33,497: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bbbd6a8c-e4aa-4e44-8e1a-befd25763f77
[2017-05-04 10:54:33,497: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bbbd6a8c-e4aa-4e44-8e1a-befd25763f77
[2017-05-04 10:54:33,497: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_target {'ha_label': u'testfs-MDT0000_3b90dd'}
[2017-05-04 10:54:33,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3bea6192-58d7-4824-b2e6-fca6e3194f9b
[2017-05-04 10:54:33,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3bea6192-58d7-4824-b2e6-fca6e3194f9b
[2017-05-04 10:54:39,075: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 1 tasked)
[2017-05-04 10:54:39,096: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 1 tasked)
[2017-05-04 10:54:39,649: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:54:39,649: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:54:39,649: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:54:39,649: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:54:39,650: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3bea6192-58d7-4824-b2e6-fca6e3194f9b
[2017-05-04 10:54:39,650: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3bea6192-58d7-4824-b2e6-fca6e3194f9b
[2017-05-04 10:54:39,678: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3bea6192-58d7-4824-b2e6-fca6e3194f9b
[2017-05-04 10:54:39,678: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3bea6192-58d7-4824-b2e6-fca6e3194f9b
[2017-05-04 10:54:39,679: INFO/job_scheduler] Job 63 finished 1 steps successfully
[2017-05-04 10:54:39,808: INFO/job] StatefulObject.set_state testfs-MDT0000 unmounted->mounted (intentional=True) 139993922403088
[2017-05-04 10:54:39,846: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm6.lotus.hpdd.lab.intel.com at severity 20
[2017-05-04 10:54:39,853: INFO/job] Job 63: StateChangeJob complete, setting state mounted on testfs-MDT0000
[2017-05-04 10:54:39,853: INFO/job] Job 63: not_deleted=True
[2017-05-04 10:54:39,853: INFO/job_scheduler] Job 63: complete_job: Updating cache
[2017-05-04 10:54:39,853: INFO/job_scheduler] Job 63: locked_item 139993922403088 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-04 10:54:39,856: INFO/job_scheduler] Job 63 completing (errored=False, cancelled=False)
[2017-05-04 10:54:40,025: INFO/job_scheduler] run_next: 2 runnable jobs of (7 pending, 0 tasked)
[2017-05-04 10:54:40,130: INFO/job_scheduler] Job 64: RunJobThread.run
[2017-05-04 10:54:40,132: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com register_target {'device_path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/testfs-OST0001', 'mount_point': u'/mnt/testfs-OST0001', 'backfstype': 'zfs'}
[2017-05-04 10:54:40,131: INFO/job_scheduler] Job 65: RunJobThread.run
[2017-05-04 10:54:40,152: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com register_target {'device_path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/testfs-OST0000', 'mount_point': u'/mnt/testfs-OST0000', 'backfstype': 'zfs'}
[2017-05-04 10:54:40,153: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e64357dc-f464-4abd-a65c-7345f7c58649
[2017-05-04 10:54:40,153: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e64357dc-f464-4abd-a65c-7345f7c58649
[2017-05-04 10:54:40,165: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ac71919d-7995-4513-9936-4c5872ac965e
[2017-05-04 10:54:40,165: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ac71919d-7995-4513-9936-4c5872ac965e
[2017-05-04 10:54:41,310: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 2 tasked)
[2017-05-04 10:54:41,327: INFO/job_scheduler] notify: Dropping update of 2 (testfs-MDT0000) because it has been updated since
[2017-05-04 10:54:41,328: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 2 tasked)
[2017-05-04 10:54:51,297: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:51,297: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:51,298: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:51,298: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:51,298: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ac71919d-7995-4513-9936-4c5872ac965e
[2017-05-04 10:54:51,298: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ac71919d-7995-4513-9936-4c5872ac965e
[2017-05-04 10:54:51,331: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ac71919d-7995-4513-9936-4c5872ac965e
[2017-05-04 10:54:51,331: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ac71919d-7995-4513-9936-4c5872ac965e
[2017-05-04 10:54:51,332: INFO/job_scheduler] Job 65 finished 2 steps successfully
[2017-05-04 10:54:51,456: INFO/job] StatefulObject.set_state testfs-OST0000 formatted->registered (intentional=True) 139993452763920
[2017-05-04 10:54:51,461: INFO/job] Job 65: StateChangeJob complete, setting state registered on testfs-OST0000
[2017-05-04 10:54:51,461: INFO/job] Job 65: not_deleted=True
[2017-05-04 10:54:51,461: INFO/job_scheduler] Job 65: complete_job: Updating cache
[2017-05-04 10:54:51,461: INFO/job_scheduler] Job 65: locked_item 139993452763920 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-04 10:54:51,464: INFO/job_scheduler] Job 65 completing (errored=False, cancelled=False)
[2017-05-04 10:54:51,482: INFO/job_scheduler] run_next: 1 runnable jobs of (5 pending, 1 tasked)
[2017-05-04 10:54:51,529: INFO/job_scheduler] Job 66: RunJobThread.run
[2017-05-04 10:54:51,530: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-05-04 10:54:51,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2da99310-d3bb-4a4c-b888-2d80d186b57a
[2017-05-04 10:54:51,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2da99310-d3bb-4a4c-b888-2d80d186b57a
[2017-05-04 10:54:52,321: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:52,321: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:52,321: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:52,321: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:52,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e64357dc-f464-4abd-a65c-7345f7c58649
[2017-05-04 10:54:52,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e64357dc-f464-4abd-a65c-7345f7c58649
[2017-05-04 10:54:52,369: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e64357dc-f464-4abd-a65c-7345f7c58649
[2017-05-04 10:54:52,369: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e64357dc-f464-4abd-a65c-7345f7c58649
[2017-05-04 10:54:52,371: INFO/job_scheduler] Job 64 finished 2 steps successfully
[2017-05-04 10:54:52,492: INFO/job] StatefulObject.set_state testfs-OST0001 formatted->registered (intentional=True) 139993922361296
[2017-05-04 10:54:52,497: INFO/job] Job 64: StateChangeJob complete, setting state registered on testfs-OST0001
[2017-05-04 10:54:52,497: INFO/job] Job 64: not_deleted=True
[2017-05-04 10:54:52,497: INFO/job_scheduler] Job 64: complete_job: Updating cache
[2017-05-04 10:54:52,497: INFO/job_scheduler] Job 64: locked_item 139993922361296 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-04 10:54:52,500: INFO/job_scheduler] Job 64 completing (errored=False, cancelled=False)
[2017-05-04 10:54:52,526: INFO/job_scheduler] run_next: 1 runnable jobs of (4 pending, 1 tasked)
[2017-05-04 10:54:52,577: INFO/job_scheduler] Job 67: RunJobThread.run
[2017-05-04 10:54:52,577: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-05-04 10:54:52,592: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6201b9dd-36e7-430d-89d6-76165a7b597f
[2017-05-04 10:54:52,592: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6201b9dd-36e7-430d-89d6-76165a7b597f
[2017-05-04 10:54:52,642: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:52,642: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:52,642: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:52,642: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:52,642: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2da99310-d3bb-4a4c-b888-2d80d186b57a
[2017-05-04 10:54:52,642: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2da99310-d3bb-4a4c-b888-2d80d186b57a
[2017-05-04 10:54:52,667: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2da99310-d3bb-4a4c-b888-2d80d186b57a
[2017-05-04 10:54:52,667: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2da99310-d3bb-4a4c-b888-2d80d186b57a
[2017-05-04 10:54:52,667: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-05-04 10:54:52,680: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dc17fc2f-fe04-4a55-97f7-95afa0ea2f64
[2017-05-04 10:54:52,680: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dc17fc2f-fe04-4a55-97f7-95afa0ea2f64
[2017-05-04 10:54:53,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:53,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:53,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:53,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:53,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6201b9dd-36e7-430d-89d6-76165a7b597f
[2017-05-04 10:54:53,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6201b9dd-36e7-430d-89d6-76165a7b597f
[2017-05-04 10:54:53,706: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6201b9dd-36e7-430d-89d6-76165a7b597f
[2017-05-04 10:54:53,706: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6201b9dd-36e7-430d-89d6-76165a7b597f
[2017-05-04 10:54:53,707: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-05-04 10:54:53,722: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 103b855d-624e-4e5f-8d6b-e67789deb882
[2017-05-04 10:54:53,722: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 103b855d-624e-4e5f-8d6b-e67789deb882
[2017-05-04 10:54:53,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:53,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:53,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:53,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:53,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dc17fc2f-fe04-4a55-97f7-95afa0ea2f64
[2017-05-04 10:54:53,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dc17fc2f-fe04-4a55-97f7-95afa0ea2f64
[2017-05-04 10:54:53,799: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dc17fc2f-fe04-4a55-97f7-95afa0ea2f64
[2017-05-04 10:54:53,799: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dc17fc2f-fe04-4a55-97f7-95afa0ea2f64
[2017-05-04 10:54:53,801: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_target_store {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/testfs-OST0000', 'device_type': 'zfs', 'mount_point': u'/mnt/testfs-OST0000', 'uuid': u'17257397556860038609', 'backfstype': 'zfs'}
[2017-05-04 10:54:53,814: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e7adf2b1-ca4b-49f0-bc22-fc3f41842909
[2017-05-04 10:54:53,814: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e7adf2b1-ca4b-49f0-bc22-fc3f41842909
[2017-05-04 10:54:53,899: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:53,899: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:53,900: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:53,900: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:53,900: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e7adf2b1-ca4b-49f0-bc22-fc3f41842909
[2017-05-04 10:54:53,900: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e7adf2b1-ca4b-49f0-bc22-fc3f41842909
[2017-05-04 10:54:53,929: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e7adf2b1-ca4b-49f0-bc22-fc3f41842909
[2017-05-04 10:54:53,929: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e7adf2b1-ca4b-49f0-bc22-fc3f41842909
[2017-05-04 10:54:53,930: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_target_store {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/testfs-OST0000', 'device_type': 'zfs', 'mount_point': u'/mnt/testfs-OST0000', 'uuid': u'17257397556860038609', 'backfstype': 'zfs'}
[2017-05-04 10:54:53,953: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f6d06fc2-075a-4c2b-9447-092131b7d58e
[2017-05-04 10:54:53,953: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f6d06fc2-075a-4c2b-9447-092131b7d58e
[2017-05-04 10:54:54,018: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:54,018: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:54,018: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:54,018: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:54,018: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f6d06fc2-075a-4c2b-9447-092131b7d58e
[2017-05-04 10:54:54,018: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f6d06fc2-075a-4c2b-9447-092131b7d58e
[2017-05-04 10:54:54,068: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f6d06fc2-075a-4c2b-9447-092131b7d58e
[2017-05-04 10:54:54,068: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f6d06fc2-075a-4c2b-9447-092131b7d58e
[2017-05-04 10:54:54,070: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/testfs-OST0000', 'mount_point': u'/mnt/testfs-OST0000', 'uuid': u'17257397556860038609', 'ha_label': u'testfs-OST0000_0a28c6', 'primary': True}
[2017-05-04 10:54:54,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e7c9df49-3f7f-467a-879b-02af8aa6259d
[2017-05-04 10:54:54,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e7c9df49-3f7f-467a-879b-02af8aa6259d
[2017-05-04 10:54:54,794: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:54,794: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:54,795: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:54,795: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:54,795: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 103b855d-624e-4e5f-8d6b-e67789deb882
[2017-05-04 10:54:54,795: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 103b855d-624e-4e5f-8d6b-e67789deb882
[2017-05-04 10:54:54,838: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 103b855d-624e-4e5f-8d6b-e67789deb882
[2017-05-04 10:54:54,838: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 103b855d-624e-4e5f-8d6b-e67789deb882
[2017-05-04 10:54:54,840: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_target_store {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/testfs-OST0001', 'device_type': 'zfs', 'mount_point': u'/mnt/testfs-OST0001', 'uuid': u'13848238846632045515', 'backfstype': 'zfs'}
[2017-05-04 10:54:54,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5285a4cc-c2ee-43ee-a64b-d8553e7dde7e
[2017-05-04 10:54:54,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5285a4cc-c2ee-43ee-a64b-d8553e7dde7e
[2017-05-04 10:54:54,913: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:54,913: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:54,913: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:54,913: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:54,913: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5285a4cc-c2ee-43ee-a64b-d8553e7dde7e
[2017-05-04 10:54:54,913: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5285a4cc-c2ee-43ee-a64b-d8553e7dde7e
[2017-05-04 10:54:54,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5285a4cc-c2ee-43ee-a64b-d8553e7dde7e
[2017-05-04 10:54:54,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5285a4cc-c2ee-43ee-a64b-d8553e7dde7e
[2017-05-04 10:54:54,924: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_target_store {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/testfs-OST0001', 'device_type': 'zfs', 'mount_point': u'/mnt/testfs-OST0001', 'uuid': u'13848238846632045515', 'backfstype': 'zfs'}
[2017-05-04 10:54:54,940: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 43e8d23d-b396-40c4-9998-36f537fbca45
[2017-05-04 10:54:54,940: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 43e8d23d-b396-40c4-9998-36f537fbca45
[2017-05-04 10:54:54,980: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:54,980: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:54,980: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:54,980: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:54,981: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 43e8d23d-b396-40c4-9998-36f537fbca45
[2017-05-04 10:54:54,981: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 43e8d23d-b396-40c4-9998-36f537fbca45
[2017-05-04 10:54:55,004: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 43e8d23d-b396-40c4-9998-36f537fbca45
[2017-05-04 10:54:55,004: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 43e8d23d-b396-40c4-9998-36f537fbca45
[2017-05-04 10:54:55,005: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/testfs-OST0001', 'mount_point': u'/mnt/testfs-OST0001', 'uuid': u'13848238846632045515', 'ha_label': u'testfs-OST0001_6ff542', 'primary': True}
[2017-05-04 10:54:55,017: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 43641d1a-b6ff-4e3f-8eb7-e4f3e3784923
[2017-05-04 10:54:55,017: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 43641d1a-b6ff-4e3f-8eb7-e4f3e3784923
[2017-05-04 10:54:57,167: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:57,167: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:57,167: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:57,167: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:57,167: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e7c9df49-3f7f-467a-879b-02af8aa6259d
[2017-05-04 10:54:57,167: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e7c9df49-3f7f-467a-879b-02af8aa6259d
[2017-05-04 10:54:57,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e7c9df49-3f7f-467a-879b-02af8aa6259d
[2017-05-04 10:54:57,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e7c9df49-3f7f-467a-879b-02af8aa6259d
[2017-05-04 10:54:57,200: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/testfs-OST0000', 'mount_point': u'/mnt/testfs-OST0000', 'uuid': u'17257397556860038609', 'ha_label': u'testfs-OST0000_0a28c6', 'primary': False}
[2017-05-04 10:54:57,218: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 18b84c97-7f90-4547-a902-eec22219af90
[2017-05-04 10:54:57,218: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 18b84c97-7f90-4547-a902-eec22219af90
[2017-05-04 10:54:58,096: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:58,096: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:58,096: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:58,096: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:58,096: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 43641d1a-b6ff-4e3f-8eb7-e4f3e3784923
[2017-05-04 10:54:58,096: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 43641d1a-b6ff-4e3f-8eb7-e4f3e3784923
[2017-05-04 10:54:58,132: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 43641d1a-b6ff-4e3f-8eb7-e4f3e3784923
[2017-05-04 10:54:58,132: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 43641d1a-b6ff-4e3f-8eb7-e4f3e3784923
[2017-05-04 10:54:58,134: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/testfs-OST0001', 'mount_point': u'/mnt/testfs-OST0001', 'uuid': u'13848238846632045515', 'ha_label': u'testfs-OST0001_6ff542', 'primary': False}
[2017-05-04 10:54:58,146: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6e7e9705-06bf-4d3e-988c-b084c387a8b0
[2017-05-04 10:54:58,146: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6e7e9705-06bf-4d3e-988c-b084c387a8b0
[2017-05-04 10:54:58,319: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:58,319: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:58,320: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:58,320: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:58,320: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 18b84c97-7f90-4547-a902-eec22219af90
[2017-05-04 10:54:58,320: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 18b84c97-7f90-4547-a902-eec22219af90
[2017-05-04 10:54:58,332: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 18b84c97-7f90-4547-a902-eec22219af90
[2017-05-04 10:54:58,332: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 18b84c97-7f90-4547-a902-eec22219af90
[2017-05-04 10:54:58,333: INFO/job_scheduler] Job 66 finished 6 steps successfully
[2017-05-04 10:54:58,409: INFO/job] StatefulObject.set_state testfs-OST0000 registered->unmounted (intentional=True) 139993452763920
[2017-05-04 10:54:58,415: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on testfs-OST0000 at severity 30
[2017-05-04 10:54:58,421: INFO/job] Job 66: StateChangeJob complete, setting state unmounted on testfs-OST0000
[2017-05-04 10:54:58,421: INFO/job] Job 66: not_deleted=True
[2017-05-04 10:54:58,421: INFO/job_scheduler] Job 66: complete_job: Updating cache
[2017-05-04 10:54:58,422: INFO/job_scheduler] Job 66: locked_item 139993452763920 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-04 10:54:58,430: INFO/job_scheduler] Job 66 completing (errored=False, cancelled=False)
[2017-05-04 10:54:58,485: INFO/job_scheduler] run_next: 1 runnable jobs of (3 pending, 1 tasked)
[2017-05-04 10:54:58,528: INFO/job_scheduler] Job 68: RunJobThread.run
[2017-05-04 10:54:58,530: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/testfs-OST0000', 'pacemaker_ha_operation': False, 'device_type': 'zfs'}
[2017-05-04 10:54:58,531: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/testfs-OST0000', 'pacemaker_ha_operation': False, 'device_type': 'zfs'}
[2017-05-04 10:54:58,539: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/testfs-OST0000', 'pacemaker_ha_operation': False, 'device_type': 'zfs'}
[2017-05-04 10:54:58,544: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 856b966a-a40e-4d42-bb15-9ab49e03175a
[2017-05-04 10:54:58,544: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 856b966a-a40e-4d42-bb15-9ab49e03175a
[2017-05-04 10:54:58,561: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0f02e158-c60f-48bd-82bd-b2ebdd3fd1ca
[2017-05-04 10:54:58,561: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0f02e158-c60f-48bd-82bd-b2ebdd3fd1ca
[2017-05-04 10:54:58,574: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6e2243bf-d8ca-4a81-9662-5aaeec82bf3f
[2017-05-04 10:54:58,574: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6e2243bf-d8ca-4a81-9662-5aaeec82bf3f
[2017-05-04 10:54:58,695: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:54:58,695: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:54:58,695: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:54:58,695: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:54:58,696: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 856b966a-a40e-4d42-bb15-9ab49e03175a
[2017-05-04 10:54:58,696: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 856b966a-a40e-4d42-bb15-9ab49e03175a
[2017-05-04 10:54:58,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 856b966a-a40e-4d42-bb15-9ab49e03175a
[2017-05-04 10:54:58,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 856b966a-a40e-4d42-bb15-9ab49e03175a
[2017-05-04 10:54:59,240: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:59,240: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:59,240: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:59,240: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:59,241: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6e7e9705-06bf-4d3e-988c-b084c387a8b0
[2017-05-04 10:54:59,241: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6e7e9705-06bf-4d3e-988c-b084c387a8b0
[2017-05-04 10:54:59,261: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6e7e9705-06bf-4d3e-988c-b084c387a8b0
[2017-05-04 10:54:59,261: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6e7e9705-06bf-4d3e-988c-b084c387a8b0
[2017-05-04 10:54:59,262: INFO/job_scheduler] Job 67 finished 6 steps successfully
[2017-05-04 10:54:59,327: INFO/job] StatefulObject.set_state testfs-OST0001 registered->unmounted (intentional=True) 139993922361296
[2017-05-04 10:54:59,332: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on testfs-OST0001 at severity 30
[2017-05-04 10:54:59,347: INFO/job] Job 67: StateChangeJob complete, setting state unmounted on testfs-OST0001
[2017-05-04 10:54:59,347: INFO/job] Job 67: not_deleted=True
[2017-05-04 10:54:59,347: INFO/job_scheduler] Job 67: complete_job: Updating cache
[2017-05-04 10:54:59,348: INFO/job_scheduler] Job 67: locked_item 139993922361296 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-04 10:54:59,350: INFO/job_scheduler] Job 67 completing (errored=False, cancelled=False)
[2017-05-04 10:54:59,377: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 1 tasked)
[2017-05-04 10:54:59,423: INFO/job_scheduler] Job 69: RunJobThread.run
[2017-05-04 10:54:59,425: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/testfs-OST0001', 'pacemaker_ha_operation': False, 'device_type': 'zfs'}
[2017-05-04 10:54:59,436: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/testfs-OST0001', 'pacemaker_ha_operation': False, 'device_type': 'zfs'}
[2017-05-04 10:54:59,440: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/testfs-OST0001', 'pacemaker_ha_operation': False, 'device_type': 'zfs'}
[2017-05-04 10:54:59,445: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f7f1a901-7a41-461c-976b-bc034d2c82f2
[2017-05-04 10:54:59,445: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f7f1a901-7a41-461c-976b-bc034d2c82f2
[2017-05-04 10:54:59,457: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ad3d5ca6-c7d2-4ca9-9727-0b3655278dad
[2017-05-04 10:54:59,457: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ad3d5ca6-c7d2-4ca9-9727-0b3655278dad
[2017-05-04 10:54:59,469: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d26bc36d-173a-421e-b389-e987612a7e88
[2017-05-04 10:54:59,469: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d26bc36d-173a-421e-b389-e987612a7e88
[2017-05-04 10:54:59,590: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:59,590: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:59,590: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:59,590: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:54:59,590: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ad3d5ca6-c7d2-4ca9-9727-0b3655278dad
[2017-05-04 10:54:59,590: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ad3d5ca6-c7d2-4ca9-9727-0b3655278dad
[2017-05-04 10:54:59,630: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ad3d5ca6-c7d2-4ca9-9727-0b3655278dad
[2017-05-04 10:54:59,630: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ad3d5ca6-c7d2-4ca9-9727-0b3655278dad
[2017-05-04 10:54:59,658: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:59,658: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:59,658: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:59,658: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:54:59,658: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0f02e158-c60f-48bd-82bd-b2ebdd3fd1ca
[2017-05-04 10:54:59,658: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0f02e158-c60f-48bd-82bd-b2ebdd3fd1ca
[2017-05-04 10:54:59,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:54:59,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:54:59,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:54:59,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:54:59,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6e2243bf-d8ca-4a81-9662-5aaeec82bf3f
[2017-05-04 10:54:59,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6e2243bf-d8ca-4a81-9662-5aaeec82bf3f
[2017-05-04 10:54:59,683: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0f02e158-c60f-48bd-82bd-b2ebdd3fd1ca
[2017-05-04 10:54:59,683: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0f02e158-c60f-48bd-82bd-b2ebdd3fd1ca
[2017-05-04 10:54:59,689: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6e2243bf-d8ca-4a81-9662-5aaeec82bf3f
[2017-05-04 10:54:59,689: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6e2243bf-d8ca-4a81-9662-5aaeec82bf3f
[2017-05-04 10:54:59,689: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com import_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/testfs-OST0000', 'pacemaker_ha_operation': False, 'device_type': 'zfs', 'validate_importable': True}
[2017-05-04 10:54:59,701: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9ca6222d-c5ad-445f-807d-79d3d232e9e3
[2017-05-04 10:54:59,701: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9ca6222d-c5ad-445f-807d-79d3d232e9e3
[2017-05-04 10:54:59,857: INFO/job_scheduler] notify: Dropping update of 4 (testfs-OST0001) because it has been updated since
[2017-05-04 10:54:59,857: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 10:54:59,871: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 10:55:00,282: INFO/job_scheduler] notify: Dropping update of 4 (testfs-OST0001) because it has been updated since
[2017-05-04 10:55:00,283: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 10:55:00,295: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 10:55:00,587: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:55:00,587: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:55:00,587: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:55:00,587: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:55:00,587: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f7f1a901-7a41-461c-976b-bc034d2c82f2
[2017-05-04 10:55:00,587: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f7f1a901-7a41-461c-976b-bc034d2c82f2
[2017-05-04 10:55:00,588: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:55:00,588: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:55:00,588: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:55:00,588: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:55:00,588: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d26bc36d-173a-421e-b389-e987612a7e88
[2017-05-04 10:55:00,588: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d26bc36d-173a-421e-b389-e987612a7e88
[2017-05-04 10:55:00,613: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f7f1a901-7a41-461c-976b-bc034d2c82f2
[2017-05-04 10:55:00,613: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f7f1a901-7a41-461c-976b-bc034d2c82f2
[2017-05-04 10:55:00,634: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d26bc36d-173a-421e-b389-e987612a7e88
[2017-05-04 10:55:00,634: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d26bc36d-173a-421e-b389-e987612a7e88
[2017-05-04 10:55:00,634: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com import_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/testfs-OST0001', 'pacemaker_ha_operation': False, 'device_type': 'zfs', 'validate_importable': True}
[2017-05-04 10:55:00,654: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c75f594d-ad68-46f6-af33-9203e71cd3f2
[2017-05-04 10:55:00,654: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c75f594d-ad68-46f6-af33-9203e71cd3f2
[2017-05-04 10:55:03,809: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:55:03,809: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:55:03,810: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:55:03,810: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:55:03,810: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c75f594d-ad68-46f6-af33-9203e71cd3f2
[2017-05-04 10:55:03,810: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c75f594d-ad68-46f6-af33-9203e71cd3f2
[2017-05-04 10:55:03,818: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c75f594d-ad68-46f6-af33-9203e71cd3f2
[2017-05-04 10:55:03,818: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c75f594d-ad68-46f6-af33-9203e71cd3f2
[2017-05-04 10:55:03,819: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_target {'ha_label': u'testfs-OST0001_6ff542'}
[2017-05-04 10:55:03,831: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cd28ccbb-9227-4290-94ca-bd83d3d709b0
[2017-05-04 10:55:03,831: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cd28ccbb-9227-4290-94ca-bd83d3d709b0
[2017-05-04 10:55:07,870: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:55:07,870: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:55:07,871: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:55:07,871: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:55:07,871: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9ca6222d-c5ad-445f-807d-79d3d232e9e3
[2017-05-04 10:55:07,871: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9ca6222d-c5ad-445f-807d-79d3d232e9e3
[2017-05-04 10:55:07,917: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9ca6222d-c5ad-445f-807d-79d3d232e9e3
[2017-05-04 10:55:07,917: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9ca6222d-c5ad-445f-807d-79d3d232e9e3
[2017-05-04 10:55:07,918: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_target {'ha_label': u'testfs-OST0000_0a28c6'}
[2017-05-04 10:55:07,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc af7fea08-706e-41d5-a40c-15b6c1ecc8c3
[2017-05-04 10:55:07,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc af7fea08-706e-41d5-a40c-15b6c1ecc8c3
[2017-05-04 10:55:19,945: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 10:55:19,960: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 10:55:20,045: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:55:20,045: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:55:20,045: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:55:20,045: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 10:55:20,045: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cd28ccbb-9227-4290-94ca-bd83d3d709b0
[2017-05-04 10:55:20,045: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cd28ccbb-9227-4290-94ca-bd83d3d709b0
[2017-05-04 10:55:20,049: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cd28ccbb-9227-4290-94ca-bd83d3d709b0
[2017-05-04 10:55:20,049: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cd28ccbb-9227-4290-94ca-bd83d3d709b0
[2017-05-04 10:55:20,051: INFO/job_scheduler] Job 69 finished 1 steps successfully
[2017-05-04 10:55:20,313: INFO/job] StatefulObject.set_state testfs-OST0001 unmounted->mounted (intentional=True) 139993922361296
[2017-05-04 10:55:20,344: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm8.lotus.hpdd.lab.intel.com at severity 20
[2017-05-04 10:55:20,364: INFO/job] Job 69: StateChangeJob complete, setting state mounted on testfs-OST0001
[2017-05-04 10:55:20,365: INFO/job] Job 69: not_deleted=True
[2017-05-04 10:55:20,366: INFO/job_scheduler] Job 69: complete_job: Updating cache
[2017-05-04 10:55:20,367: INFO/job_scheduler] Job 69: locked_item 139993922361296 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-04 10:55:20,372: INFO/job_scheduler] Job 69 completing (errored=False, cancelled=False)
[2017-05-04 10:55:20,530: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-04 10:55:24,399: INFO/job_scheduler] notify: Dropping update of 4 (testfs-OST0001) because it has been updated since
[2017-05-04 10:55:24,399: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-04 10:55:24,416: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-04 10:55:31,425: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-04 10:55:31,443: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-04 10:55:32,204: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:55:32,204: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:55:32,204: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:55:32,204: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 10:55:32,204: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc af7fea08-706e-41d5-a40c-15b6c1ecc8c3
[2017-05-04 10:55:32,204: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc af7fea08-706e-41d5-a40c-15b6c1ecc8c3
[2017-05-04 10:55:32,210: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc af7fea08-706e-41d5-a40c-15b6c1ecc8c3
[2017-05-04 10:55:32,210: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc af7fea08-706e-41d5-a40c-15b6c1ecc8c3
[2017-05-04 10:55:32,211: INFO/job_scheduler] Job 68 finished 1 steps successfully
[2017-05-04 10:55:32,571: INFO/job] StatefulObject.set_state testfs-OST0000 unmounted->mounted (intentional=True) 139993452763920
[2017-05-04 10:55:32,608: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 20
[2017-05-04 10:55:32,627: INFO/job] Job 68: StateChangeJob complete, setting state mounted on testfs-OST0000
[2017-05-04 10:55:32,629: INFO/job] Job 68: not_deleted=True
[2017-05-04 10:55:32,629: INFO/job_scheduler] Job 68: complete_job: Updating cache
[2017-05-04 10:55:32,630: INFO/job_scheduler] Job 68: locked_item 139993452763920 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-04 10:55:32,638: INFO/job_scheduler] Job 68 completing (errored=False, cancelled=False)
[2017-05-04 10:55:32,696: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-04 10:55:32,746: INFO/job_scheduler] Job 70: open DB connection during completion
[2017-05-04 10:55:32,754: INFO/job] StatefulObject.set_state testfs unavailable->available (intentional=True) 139993453457232
[2017-05-04 10:55:32,758: INFO/job] Job 70: StateChangeJob complete, setting state available on testfs
[2017-05-04 10:55:32,758: INFO/job] Job 70: not_deleted=True
[2017-05-04 10:55:32,758: INFO/job_scheduler] Job 70: complete_job: Updating cache
[2017-05-04 10:55:32,758: INFO/job_scheduler] Job 70: locked_item 139993453457232 <class 'chroma_core.models.filesystem.ManagedFilesystem'> False True
[2017-05-04 10:55:32,761: INFO/job_scheduler] Job 70 completing (errored=False, cancelled=False)
[2017-05-04 10:55:32,799: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 10:55:33,012: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 10:55:33,030: INFO/job_scheduler] notify: Dropping update of 3 (testfs-OST0000) because it has been updated since
[2017-05-04 10:55:33,030: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 10:56:09,320: INFO/job_scheduler] _notify: Updating .state of item 1 (MGS) from mounted to unmounted
[2017-05-04 10:56:09,321: INFO/job] StatefulObject.set_state MGS mounted->unmounted (intentional=False) 139993453881360
[2017-05-04 10:56:09,330: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on MGS at severity 40
[2017-05-04 10:56:09,330: INFO/job_scheduler] _notify: Updating .active_mount_id of item 1 (MGS) from 1 to None
[2017-05-04 10:56:09,345: INFO/job_scheduler] _notify: Set active_mount_id=None on MGS (ManagedTarget-1) and saved
[2017-05-04 10:56:09,353: INFO/job_scheduler] _notify: Updating .state of item 1 (testfs) from available to unavailable
[2017-05-04 10:56:09,353: INFO/job] StatefulObject.set_state testfs available->unavailable (intentional=False) 139993453854096
[2017-05-04 10:56:09,362: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 10:56:09,396: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 10:56:09,558: INFO/job_scheduler] notify: Dropping update of 1 (MGS) because it has been updated since
[2017-05-04 10:56:09,558: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 10:56:09,606: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 10:56:29,613: INFO/job_scheduler] _notify: Updating .state of item 1 (MGS) from unmounted to mounted
[2017-05-04 10:56:29,614: INFO/job] StatefulObject.set_state MGS unmounted->mounted (intentional=False) 139993453454672
[2017-05-04 10:56:29,643: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 20
[2017-05-04 10:56:29,647: INFO/job_scheduler] _notify: Updating .active_mount_id of item 1 (MGS) from None to 1
[2017-05-04 10:56:29,649: INFO/job_scheduler] _notify: Set active_mount_id=1 on MGS (ManagedTarget-1) and saved
[2017-05-04 10:56:29,660: INFO/job_scheduler] _notify: Updating .state of item 1 (testfs) from unavailable to available
[2017-05-04 10:56:29,660: INFO/job] StatefulObject.set_state testfs unavailable->available (intentional=False) 139993453881232
[2017-05-04 10:56:29,673: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 10:56:29,740: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 10:56:31,237: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 10:56:31,243: INFO/command_plan] set_state: <class 'chroma_core.models.target.ManagedTarget'>-1 transitioning from mounted to unmounted
[2017-05-04 10:56:31,279: INFO/command_plan] Created command 19 (Stop target MGS) with 1 jobs
[2017-05-04 10:56:31,340: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-04 10:56:31,355: INFO/job_scheduler] Job 75: RunJobThread.run
[2017-05-04 10:56:31,357: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_target {'ha_label': u'MGS_9879fb'}
[2017-05-04 10:56:31,382: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 480e6cda-c1a9-4d73-a90c-903f486ba199
[2017-05-04 10:56:31,382: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 480e6cda-c1a9-4d73-a90c-903f486ba199
[2017-05-04 10:56:31,421: INFO/job_scheduler] notify: Dropping update of 1 (MGS) because it has been updated since
[2017-05-04 10:56:31,421: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-04 10:56:31,456: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-04 10:56:49,474: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-04 10:56:49,510: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-04 10:56:49,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:56:49,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:56:49,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:56:49,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/532599b9-9283-4f58-97fc-d942fb60e255
[2017-05-04 10:56:49,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 480e6cda-c1a9-4d73-a90c-903f486ba199
[2017-05-04 10:56:49,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 480e6cda-c1a9-4d73-a90c-903f486ba199
[2017-05-04 10:56:49,548: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 480e6cda-c1a9-4d73-a90c-903f486ba199
[2017-05-04 10:56:49,548: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 480e6cda-c1a9-4d73-a90c-903f486ba199
[2017-05-04 10:56:49,549: INFO/job_scheduler] Job 75 finished 1 steps successfully
[2017-05-04 10:56:49,660: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-04 10:56:49,770: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-04 10:56:49,807: INFO/job] StatefulObject.set_state MGS mounted->unmounted (intentional=True) 139993453500624
[2017-05-04 10:56:49,814: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on MGS at severity 30
[2017-05-04 10:56:49,821: INFO/job] Job 75: StateChangeJob complete, setting state unmounted on MGS
[2017-05-04 10:56:49,823: INFO/job] Job 75: not_deleted=True
[2017-05-04 10:56:49,827: INFO/job_scheduler] Job 75: complete_job: Updating cache
[2017-05-04 10:56:49,827: INFO/job_scheduler] Job 75: locked_item 139993453500624 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-04 10:56:49,841: INFO/job_scheduler] Job 75 completing (errored=False, cancelled=False)
[2017-05-04 10:56:49,853: INFO/job_scheduler] _notify: Updating .state of item 1 (testfs) from available to unavailable
[2017-05-04 10:56:49,853: INFO/job] StatefulObject.set_state testfs available->unavailable (intentional=False) 139993318322384
[2017-05-04 10:56:49,908: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 10:56:51,633: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 10:56:51,638: INFO/command_plan] set_state: <class 'chroma_core.models.filesystem.ManagedFilesystem'>-1 transitioning from unavailable to stopped
[2017-05-04 10:56:51,699: INFO/command_plan] Created command 20 (Stop file system testfs) with 4 jobs
[2017-05-04 10:56:51,732: INFO/job_scheduler] run_next: 3 runnable jobs of (4 pending, 0 tasked)
[2017-05-04 10:56:51,775: INFO/job_scheduler] Job 76: RunJobThread.run
[2017-05-04 10:56:51,775: INFO/job_scheduler] Job 77: RunJobThread.run
[2017-05-04 10:56:51,778: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com stop_target {'ha_label': u'testfs-OST0001_6ff542'}
[2017-05-04 10:56:51,788: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com stop_target {'ha_label': u'testfs-MDT0000_3b90dd'}
[2017-05-04 10:56:51,776: INFO/job_scheduler] Job 78: RunJobThread.run
[2017-05-04 10:56:51,791: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com stop_target {'ha_label': u'testfs-OST0000_0a28c6'}
[2017-05-04 10:56:51,800: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4c246eab-4312-4cb8-834e-852e6d6d01a9
[2017-05-04 10:56:51,800: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4c246eab-4312-4cb8-834e-852e6d6d01a9
[2017-05-04 10:56:51,815: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8e31438e-c01e-4605-b50c-6e24a0a8cc35
[2017-05-04 10:56:51,815: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8e31438e-c01e-4605-b50c-6e24a0a8cc35
[2017-05-04 10:56:51,834: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1cbe42f7-5d66-44a4-aeff-51e65b48dea8
[2017-05-04 10:56:51,834: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1cbe42f7-5d66-44a4-aeff-51e65b48dea8
[2017-05-04 10:57:02,299: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:57:02,299: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:57:02,299: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:57:02,299: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/33a06138-aea9-4e65-8dc8-94950ab0902f
[2017-05-04 10:57:02,299: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8e31438e-c01e-4605-b50c-6e24a0a8cc35
[2017-05-04 10:57:02,299: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8e31438e-c01e-4605-b50c-6e24a0a8cc35
[2017-05-04 10:57:02,334: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8e31438e-c01e-4605-b50c-6e24a0a8cc35
[2017-05-04 10:57:02,334: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8e31438e-c01e-4605-b50c-6e24a0a8cc35
[2017-05-04 10:57:02,334: INFO/job_scheduler] Job 76 finished 1 steps successfully
[2017-05-04 10:57:02,411: INFO/job] StatefulObject.set_state testfs-MDT0000 mounted->unmounted (intentional=True) 139993452758480
[2017-05-04 10:57:02,416: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on testfs-MDT0000 at severity 30
[2017-05-04 10:57:02,421: INFO/job] Job 76: StateChangeJob complete, setting state unmounted on testfs-MDT0000
[2017-05-04 10:57:02,423: INFO/job] Job 76: not_deleted=True
[2017-05-04 10:57:02,423: INFO/job_scheduler] Job 76: complete_job: Updating cache
[2017-05-04 10:57:02,426: INFO/job_scheduler] Job 76: locked_item 139993452758480 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-04 10:57:02,442: INFO/job_scheduler] Job 76 completing (errored=False, cancelled=False)
[2017-05-04 10:57:02,461: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 10:57:09,705: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 10:57:09,720: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 10:57:10,502: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 10:57:10,516: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 10:57:29,732: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 10:57:29,747: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 10:57:30,413: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 10:57:30,429: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 10:57:30,800: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 10:57:30,822: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 10:57:31,701: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 10:57:31,721: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 10:57:32,175: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 10:57:32,184: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 10:57:32,798: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 10:57:32,812: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 10:57:59,651: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 10:57:59,665: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 10:58:01,452: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 10:58:01,468: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 11:07:41,193: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 11:07:41,209: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 11:07:42,243: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 11:07:42,260: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 11:07:42,515: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 11:07:42,533: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 11:07:43,564: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 11:07:43,578: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 11:07:44,341: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 11:07:44,352: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 11:07:44,665: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 11:07:44,678: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 11:08:11,162: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 11:08:11,176: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 11:08:13,363: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 11:08:13,379: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 11:13:01,290: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 11:13:01,290: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 11:13:01,290: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 11:13:01,290: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/936cca21-a605-4122-8f1f-49268c48a439
[2017-05-04 11:13:01,290: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1cbe42f7-5d66-44a4-aeff-51e65b48dea8
[2017-05-04 11:13:01,290: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1cbe42f7-5d66-44a4-aeff-51e65b48dea8
[2017-05-04 11:13:01,298: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1cbe42f7-5d66-44a4-aeff-51e65b48dea8
[2017-05-04 11:13:01,298: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1cbe42f7-5d66-44a4-aeff-51e65b48dea8
[2017-05-04 11:13:01,335: INFO/job] Job 78 UnmountStep: failed to stop target testfs-OST0000_0a28c6
[2017-05-04 11:13:01,335: ERROR/job_scheduler] Job 78 step 0 encountered an agent error: failed to stop target testfs-OST0000_0a28c6
[2017-05-04 11:13:01,770: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 11:13:01,770: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 11:13:01,770: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 11:13:01,770: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/81ff8c8c-bcae-44f4-919e-aa9ce1c52772
[2017-05-04 11:13:01,770: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4c246eab-4312-4cb8-834e-852e6d6d01a9
[2017-05-04 11:13:01,770: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4c246eab-4312-4cb8-834e-852e6d6d01a9
[2017-05-04 11:13:01,782: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4c246eab-4312-4cb8-834e-852e6d6d01a9
[2017-05-04 11:13:01,782: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4c246eab-4312-4cb8-834e-852e6d6d01a9
[2017-05-04 11:13:01,825: INFO/job] Job 77 UnmountStep: failed to stop target testfs-OST0001_6ff542
[2017-05-04 11:13:01,825: ERROR/job_scheduler] Job 77 step 0 encountered an agent error: failed to stop target testfs-OST0001_6ff542
[2017-05-04 11:13:10,573: INFO/job_scheduler] Job 78: complete_job: Updating cache
[2017-05-04 11:13:10,573: INFO/job_scheduler] Job 78: locked_item 139993453906384 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-04 11:13:10,576: INFO/job_scheduler] Job 78 completing (errored=True, cancelled=False)
[2017-05-04 11:13:10,598: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-04 11:13:18,935: INFO/job_scheduler] Job 77: complete_job: Updating cache
[2017-05-04 11:13:18,935: INFO/job_scheduler] Job 77: locked_item 139993454125264 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-04 11:13:18,938: INFO/job_scheduler] Job 77 completing (errored=True, cancelled=False)
[2017-05-04 11:13:18,959: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-04 11:13:18,983: WARNING/job] DependOn not satisfied: testfs-OST0000 in state mounted, not one of ['registered', 'formatted', 'unmounted', 'forgotten', 'unformatted', 'removed'] (preferred unmounted)
[2017-05-04 11:13:18,997: WARNING/job] DependOn not satisfied: testfs-OST0000 in state mounted, not one of ['registered', 'formatted', 'unmounted', 'forgotten', 'unformatted', 'removed'] (preferred unmounted)
[2017-05-04 11:13:18,998: WARNING/job] DependOn not satisfied: testfs-OST0001 in state mounted, not one of ['registered', 'formatted', 'unmounted', 'forgotten', 'unformatted', 'removed'] (preferred unmounted)
[2017-05-04 11:13:19,000: WARNING/job_scheduler] Job 79: cancelling because of failed dependency
[2017-05-04 11:13:19,000: INFO/job_scheduler] Job 79 completing (errored=False, cancelled=True)
[2017-05-04 11:13:19,060: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:17:52,585: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:17:52,617: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:17:54,160: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:17:54,199: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:17:54,268: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:17:54,292: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:17:55,420: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:17:55,439: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:17:56,641: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:17:56,661: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:17:56,703: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:17:56,723: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:18:22,607: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:18:22,639: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:18:25,288: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:18:25,309: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:28:03,982: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:28:04,008: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:28:06,954: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:28:06,991: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:28:07,100: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:28:07,127: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:28:07,213: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:28:07,235: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:28:08,756: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:28:08,772: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:28:08,803: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:28:08,819: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:28:33,995: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:28:34,026: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:28:37,174: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:28:37,216: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:38:16,414: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:38:16,438: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:38:17,763: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:38:17,799: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:38:17,855: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:38:17,883: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:38:18,977: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:38:18,996: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:38:20,776: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:38:20,795: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:38:20,949: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:38:20,971: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:38:45,364: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:38:45,399: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:38:49,133: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:38:49,148: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:48:26,863: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:48:26,906: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:48:29,511: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:48:29,546: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:48:29,601: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:48:29,631: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:48:30,963: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:48:30,976: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:48:32,782: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:48:32,793: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:48:33,103: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:48:33,124: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:48:57,025: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:48:57,053: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:49:01,291: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:49:01,304: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:58:39,282: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:58:39,309: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:58:41,220: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:58:41,252: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:58:41,311: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:58:41,332: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:58:42,798: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:58:42,821: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:58:44,671: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:58:44,688: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:58:45,401: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:58:45,414: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:59:08,489: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:59:08,514: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:59:13,153: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 11:59:13,173: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:08:49,696: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:08:49,717: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:08:52,865: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:08:52,894: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:08:54,120: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:08:54,158: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:08:54,572: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:08:54,588: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:08:56,833: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:08:56,849: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:08:57,550: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:08:57,564: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:09:20,098: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:09:20,132: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:09:25,138: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:09:25,157: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:19:01,177: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:19:01,241: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:19:04,592: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:19:04,618: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:19:05,014: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:19:05,050: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:19:06,420: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:19:06,434: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:19:08,919: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:19:08,938: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:19:09,712: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:19:09,731: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:19:31,877: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:19:31,904: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:19:37,084: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:19:37,102: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:29:13,690: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:29:13,711: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:29:16,387: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:29:16,423: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:29:16,911: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:29:16,945: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:29:18,321: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:29:18,338: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:29:21,013: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:29:21,029: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:29:21,839: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:29:21,856: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:29:43,368: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:29:43,388: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:29:48,924: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:29:48,940: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:39:24,232: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:39:24,258: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:39:28,120: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:39:28,153: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:39:28,815: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:39:28,848: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:39:30,218: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:39:30,236: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:39:32,913: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:39:32,929: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:39:34,073: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:39:34,093: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:39:54,859: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:39:54,882: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:40:00,693: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:40:00,712: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:49:36,835: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:49:36,864: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:49:39,931: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:49:39,960: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:49:41,560: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:49:41,597: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:49:42,019: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:49:42,033: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:49:44,918: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:49:44,937: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:49:46,329: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:49:46,345: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:50:06,390: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:50:06,418: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:50:12,638: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:50:12,657: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:59:47,307: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:59:47,333: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:59:51,732: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:59:51,761: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:59:52,563: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:59:52,599: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:59:53,817: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:59:53,832: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:59:56,971: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:59:56,991: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:59:58,539: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 12:59:58,566: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 13:00:17,862: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 13:00:17,886: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 13:00:24,563: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 13:00:24,585: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 13:09:59,906: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 13:09:59,975: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 13:10:04,558: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 13:10:04,582: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 13:10:05,395: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 13:10:05,430: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 13:10:05,860: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 13:10:05,876: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 13:10:09,303: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 13:10:09,325: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 13:10:10,897: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 13:10:10,921: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 13:10:29,421: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 13:10:29,445: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 13:10:36,406: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 13:10:36,434: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 13:11:47,755: INFO/chroma_service] Stopping job_scheduler
[2017-05-04 13:11:47,755: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-05-04 13:11:47,755: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-05-04 13:11:47,756: INFO/queue] Stopping ServiceQueue agent_action_runner_rx
[2017-05-04 13:11:48,463: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-05-04 13:11:48,463: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-05-04 13:11:48,464: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-05-04 13:11:48,464: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-05-04 13:11:48,464: INFO/job_scheduler] Stopping...
[2017-05-04 13:11:48,464: INFO/queue] Stopping ServiceQueue job_scheduler_notifications
[2017-05-04 13:11:48,464: INFO/job_scheduler] Joining...
[2017-05-04 13:11:49,415: INFO/job_scheduler] Complete.
[2017-05-04 13:11:49,415: INFO/chroma_service] Joining job_scheduler
[2017-05-04 13:11:49,415: INFO/job_scheduler] Cancelling outstanding jobs...
[2017-05-04 16:00:02,441: INFO/queue] Purged 0 messages from 'job_scheduler_notifications' queue
[2017-05-04 16:00:02,495: INFO/queue] Purged 0 messages from 'agent_action_runner_rx' queue
[2017-05-04 16:00:02,718: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: None/None
[2017-05-04 16:00:02,718: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: None/None
[2017-05-04 16:03:03,801: INFO/chroma_service] Stopping job_scheduler
[2017-05-04 16:03:03,802: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-05-04 16:03:03,802: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-05-04 16:03:03,802: INFO/queue] Stopping ServiceQueue agent_action_runner_rx
[2017-05-04 16:03:03,918: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-05-04 16:03:03,918: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-05-04 16:03:03,918: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-05-04 16:03:03,918: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-05-04 16:03:03,918: INFO/job_scheduler] Stopping...
[2017-05-04 16:03:03,918: INFO/queue] Stopping ServiceQueue job_scheduler_notifications
[2017-05-04 16:03:03,918: INFO/job_scheduler] Joining...
[2017-05-04 16:03:04,884: INFO/job_scheduler] Complete.
[2017-05-04 16:03:04,885: INFO/chroma_service] Joining job_scheduler
[2017-05-04 16:03:04,885: INFO/job_scheduler] Cancelling outstanding jobs...
[2017-05-04 16:04:51,275: INFO/queue] Purged 0 messages from 'job_scheduler_notifications' queue
[2017-05-04 16:04:51,313: INFO/queue] Purged 0 messages from 'agent_action_runner_rx' queue
[2017-05-04 16:09:51,742: WARNING/chroma_core.services.job_scheduler.agent_rpc] Unable to reset action_runner sessions
[2017-05-04 16:09:51,742: WARNING/chroma_core.services.job_scheduler.agent_rpc] Unable to reset action_runner sessions
[2017-05-04 16:09:51,755: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: None/None
[2017-05-04 16:09:51,755: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: None/None
[2017-05-04 16:10:29,590: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 16:10:29,591: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-04 16:10:29,611: INFO/command_plan] add_jobs: created Job 1 (Test for host connectivity)
[2017-05-04 16:10:29,636: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-04 16:10:29,642: INFO/job_scheduler] Job 1: RunJobThread.run
[2017-05-04 16:10:29,670: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 16:10:29,670: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-04 16:10:29,678: INFO/command_plan] add_jobs: created Job 2 (Test for host connectivity)
[2017-05-04 16:10:29,729: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-05-04 16:10:29,741: INFO/job_scheduler] Job 2: RunJobThread.run
[2017-05-04 16:10:29,785: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 16:10:29,786: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-04 16:10:29,795: INFO/command_plan] add_jobs: created Job 3 (Test for host connectivity)
[2017-05-04 16:10:29,842: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 16:10:29,870: INFO/job_scheduler] Job 3: RunJobThread.run
[2017-05-04 16:10:29,932: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 16:10:29,932: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-04 16:10:29,953: INFO/command_plan] add_jobs: created Job 4 (Test for host connectivity)
[2017-05-04 16:10:29,972: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-05-04 16:10:29,997: INFO/job_scheduler] Job 4: RunJobThread.run
[2017-05-04 16:10:33,221: INFO/job_scheduler] Job 1 finished 1 steps successfully
[2017-05-04 16:10:33,252: INFO/job_scheduler] Job 1: complete_job: Updating cache
[2017-05-04 16:10:33,252: INFO/job_scheduler] Job 1 completing (errored=False, cancelled=False)
[2017-05-04 16:10:33,285: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-04 16:10:33,371: INFO/job_scheduler] Job 2 finished 1 steps successfully
[2017-05-04 16:10:33,435: INFO/job_scheduler] Job 2: complete_job: Updating cache
[2017-05-04 16:10:33,435: INFO/job_scheduler] Job 2 completing (errored=False, cancelled=False)
[2017-05-04 16:10:33,468: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-05-04 16:10:33,679: INFO/job_scheduler] Job 4 finished 1 steps successfully
[2017-05-04 16:10:33,693: INFO/job_scheduler] Job 4: complete_job: Updating cache
[2017-05-04 16:10:33,693: INFO/job_scheduler] Job 4 completing (errored=False, cancelled=False)
[2017-05-04 16:10:33,728: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-04 16:10:33,918: INFO/job_scheduler] Job 3 finished 1 steps successfully
[2017-05-04 16:10:33,977: INFO/job_scheduler] Job 3: complete_job: Updating cache
[2017-05-04 16:10:33,977: INFO/job_scheduler] Job 3 completing (errored=False, cancelled=False)
[2017-05-04 16:10:34,002: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 16:10:38,227: INFO/job_scheduler] Getting FQDN for 'lotus-32vm5': lotus-32vm5
lotus-32vm5.lotus.hpdd.lab.intel.com

[2017-05-04 16:10:38,257: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 16:10:38,261: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-1 transitioning from undeployed to unconfigured
[2017-05-04 16:10:38,279: INFO/command_plan] Created command 5 (Deploying agent to host lotus-32vm5.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-05-04 16:10:38,327: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-04 16:10:38,486: INFO/job_scheduler] Job 5: RunJobThread.run
[2017-05-04 16:10:39,007: INFO/job_scheduler] Getting FQDN for 'lotus-32vm7.lotus.hpdd.lab.intel.com': lotus-32vm7
lotus-32vm7.lotus.hpdd.lab.intel.com

[2017-05-04 16:10:39,037: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 16:10:39,040: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-2 transitioning from undeployed to unconfigured
[2017-05-04 16:10:39,060: INFO/command_plan] Created command 6 (Deploying agent to host lotus-32vm7.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-05-04 16:10:39,105: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-05-04 16:10:39,150: INFO/job_scheduler] Job 6: RunJobThread.run
[2017-05-04 16:10:39,752: INFO/job_scheduler] Getting FQDN for 'lotus-32vm8.lotus.hpdd.lab.intel.com': lotus-32vm8
lotus-32vm8.lotus.hpdd.lab.intel.com

[2017-05-04 16:10:39,782: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 16:10:39,786: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-3 transitioning from undeployed to unconfigured
[2017-05-04 16:10:39,808: INFO/command_plan] Created command 7 (Deploying agent to host lotus-32vm8.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-05-04 16:10:39,887: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 16:10:39,941: INFO/job_scheduler] Job 7: RunJobThread.run
[2017-05-04 16:10:39,967: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-04 16:10:40,093: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-04 16:10:40,631: INFO/job_scheduler] Getting FQDN for 'lotus-32vm6.lotus.hpdd.lab.intel.com': lotus-32vm6
lotus-32vm6.lotus.hpdd.lab.intel.com

[2017-05-04 16:10:40,657: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 16:10:40,661: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-4 transitioning from undeployed to unconfigured
[2017-05-04 16:10:40,677: INFO/command_plan] Created command 8 (Deploying agent to host lotus-32vm6.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-05-04 16:10:40,725: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-05-04 16:10:40,782: INFO/job_scheduler] Job 8: RunJobThread.run
[2017-05-04 16:10:40,794: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-04 16:10:41,520: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-04 16:10:41,785: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 16:10:41,786: INFO/job_scheduler]   Job 5 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-04 16:10:41,786: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-04 16:10:41,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/cbf3af92-ef74-45a4-95e7-1b3837884337
[2017-05-04 16:10:41,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/cbf3af92-ef74-45a4-95e7-1b3837884337
[2017-05-04 16:10:41,810: INFO/job_scheduler] Job 5 finished 1 steps successfully
[2017-05-04 16:10:41,859: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 139657774235472
[2017-05-04 16:10:41,871: INFO/job] Job 5: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:41,873: INFO/job] Job 5: not_deleted=True
[2017-05-04 16:10:41,873: INFO/job_scheduler] Job 5: complete_job: Updating cache
[2017-05-04 16:10:41,873: INFO/job_scheduler] Job 5: locked_item 139657774235472 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-04 16:10:41,877: INFO/job_scheduler] Job 5 completing (errored=False, cancelled=False)
[2017-05-04 16:10:41,942: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-1
[2017-05-04 16:10:41,943: INFO/job_scheduler] _notify: Updating .boot_time of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com) from None to 2017-05-03 16:42:07+00:00
[2017-05-04 16:10:41,952: INFO/job_scheduler] _notify: Set boot_time=2017-05-03 16:42:07+00:00 on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-1) and saved
[2017-05-04 16:10:41,961: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-04 16:10:42,045: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 16:10:42,045: INFO/job_scheduler]   Job 6 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-04 16:10:42,045: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-04 16:10:42,073: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-04 16:10:42,073: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-04 16:10:42,080: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/316480ef-58c6-4762-8a5c-ac254026c06f
[2017-05-04 16:10:42,080: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/316480ef-58c6-4762-8a5c-ac254026c06f
[2017-05-04 16:10:42,759: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 16:10:42,759: INFO/job_scheduler]   Job 7 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-04 16:10:42,759: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-04 16:10:42,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/8cd0679b-28b4-479a-bdac-ffbb32db23f7
[2017-05-04 16:10:42,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/8cd0679b-28b4-479a-bdac-ffbb32db23f7
[2017-05-04 16:10:42,832: INFO/job_scheduler] Job 7 finished 1 steps successfully
[2017-05-04 16:10:42,897: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 139657774261072
[2017-05-04 16:10:42,905: INFO/job] Job 7: StateChangeJob complete, setting state unconfigured on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:42,905: INFO/job] Job 7: not_deleted=True
[2017-05-04 16:10:42,905: INFO/job_scheduler] Job 7: complete_job: Updating cache
[2017-05-04 16:10:42,905: INFO/job_scheduler] Job 7: locked_item 139657774261072 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-04 16:10:42,907: INFO/job_scheduler] Job 7 completing (errored=False, cancelled=False)
[2017-05-04 16:10:42,956: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-3
[2017-05-04 16:10:42,956: INFO/job_scheduler] _notify: Updating .boot_time of item 3 (lotus-32vm8.lotus.hpdd.lab.intel.com) from None to 2017-04-25 07:09:56+00:00
[2017-05-04 16:10:42,961: INFO/job_scheduler] _notify: Set boot_time=2017-04-25 07:09:56+00:00 on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-3) and saved
[2017-05-04 16:10:42,977: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-05-04 16:10:43,077: INFO/job_scheduler] Job 6 finished 1 steps successfully
[2017-05-04 16:10:43,154: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 139657908363792
[2017-05-04 16:10:43,161: INFO/job] Job 6: StateChangeJob complete, setting state unconfigured on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:43,161: INFO/job] Job 6: not_deleted=True
[2017-05-04 16:10:43,161: INFO/job_scheduler] Job 6: complete_job: Updating cache
[2017-05-04 16:10:43,161: INFO/job_scheduler] Job 6: locked_item 139657908363792 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-04 16:10:43,164: INFO/job_scheduler] Job 6 completing (errored=False, cancelled=False)
[2017-05-04 16:10:43,365: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-2
[2017-05-04 16:10:43,366: INFO/job_scheduler] _notify: Updating .boot_time of item 2 (lotus-32vm7.lotus.hpdd.lab.intel.com) from None to 2017-04-25 07:09:31+00:00
[2017-05-04 16:10:43,373: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-04 16:10:43,373: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-04 16:10:43,378: INFO/job_scheduler] _notify: Set boot_time=2017-04-25 07:09:31+00:00 on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-2) and saved
[2017-05-04 16:10:43,384: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-04 16:10:43,398: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/ec71d10c-93ee-4142-b94d-1c0407491462
[2017-05-04 16:10:43,398: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/ec71d10c-93ee-4142-b94d-1c0407491462
[2017-05-04 16:10:43,440: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 16:10:43,440: INFO/job_scheduler]   Job 8 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-04 16:10:43,440: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-04 16:10:44,374: INFO/job_scheduler] Job 8 finished 1 steps successfully
[2017-05-04 16:10:44,438: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 139657774323856
[2017-05-04 16:10:44,444: INFO/job] Job 8: StateChangeJob complete, setting state unconfigured on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:44,445: INFO/job] Job 8: not_deleted=True
[2017-05-04 16:10:44,445: INFO/job_scheduler] Job 8: complete_job: Updating cache
[2017-05-04 16:10:44,445: INFO/job_scheduler] Job 8: locked_item 139657774323856 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-04 16:10:44,448: INFO/job_scheduler] Job 8 completing (errored=False, cancelled=False)
[2017-05-04 16:10:44,505: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-4
[2017-05-04 16:10:44,505: INFO/job_scheduler] _notify: Updating .boot_time of item 4 (lotus-32vm6.lotus.hpdd.lab.intel.com) from None to 2017-04-25 07:09:19+00:00
[2017-05-04 16:10:44,510: INFO/job_scheduler] _notify: Set boot_time=2017-04-25 07:09:19+00:00 on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-4) and saved
[2017-05-04 16:10:44,519: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 16:10:51,261: INFO/job_scheduler] _notify: Updating .properties of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-05-04 16:10:51,292: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-1) and saved
[2017-05-04 16:10:51,297: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 16:10:51,442: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 16:10:51,615: INFO/job_scheduler] _notify: Updating .properties of item 2 (lotus-32vm7.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-05-04 16:10:51,620: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-2) and saved
[2017-05-04 16:10:51,633: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 16:10:51,816: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 16:10:52,747: INFO/job_scheduler] _notify: Updating .properties of item 4 (lotus-32vm6.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-05-04 16:10:52,752: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-4) and saved
[2017-05-04 16:10:52,757: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 16:10:52,870: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 16:10:53,309: INFO/job_scheduler] _notify: Updating .properties of item 3 (lotus-32vm8.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-05-04 16:10:53,315: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-3) and saved
[2017-05-04 16:10:53,329: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 16:10:53,488: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 16:10:56,330: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 16:10:56,334: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-1 transitioning from unconfigured to managed
[2017-05-04 16:10:56,568: INFO/command_plan] Created command 9 (Setup managed host lotus-32vm5.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-05-04 16:10:56,637: INFO/job_scheduler] run_next: 1 runnable jobs of (11 pending, 0 tasked)
[2017-05-04 16:10:56,696: INFO/job_scheduler] Job 9: RunJobThread.run
[2017-05-04 16:10:56,697: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-05-04 16:10:56,756: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 34e19c79-78f5-4961-8cf3-bb9a93cd894f
[2017-05-04 16:10:56,756: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 34e19c79-78f5-4961-8cf3-bb9a93cd894f
[2017-05-04 16:10:56,899: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 16:10:56,904: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-2 transitioning from unconfigured to managed
[2017-05-04 16:10:57,120: INFO/command_plan] Created command 10 (Setup managed host lotus-32vm7.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-05-04 16:10:57,221: INFO/job_scheduler] run_next: 1 runnable jobs of (21 pending, 1 tasked)
[2017-05-04 16:10:57,259: INFO/job_scheduler] Job 20: RunJobThread.run
[2017-05-04 16:10:57,260: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-05-04 16:10:57,291: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc aea5e515-77fe-40ee-9216-0dd2ca15de0d
[2017-05-04 16:10:57,291: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc aea5e515-77fe-40ee-9216-0dd2ca15de0d
[2017-05-04 16:10:57,411: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 16:10:57,417: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-3 transitioning from unconfigured to managed
[2017-05-04 16:10:57,563: INFO/command_plan] Created command 11 (Setup managed host lotus-32vm8.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-05-04 16:10:57,695: INFO/job_scheduler] run_next: 1 runnable jobs of (31 pending, 2 tasked)
[2017-05-04 16:10:57,724: INFO/job_scheduler] Job 31: RunJobThread.run
[2017-05-04 16:10:57,725: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-05-04 16:10:57,743: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f7849bdd-4f19-406f-b602-73c1f867d43b
[2017-05-04 16:10:57,743: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f7849bdd-4f19-406f-b602-73c1f867d43b
[2017-05-04 16:10:57,826: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 16:10:57,831: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-4 transitioning from unconfigured to managed
[2017-05-04 16:10:57,858: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/cbf3af92-ef74-45a4-95e7-1b3837884337
[2017-05-04 16:10:57,858: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/cbf3af92-ef74-45a4-95e7-1b3837884337
[2017-05-04 16:10:57,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/cbf3af92-ef74-45a4-95e7-1b3837884337
[2017-05-04 16:10:57,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/cbf3af92-ef74-45a4-95e7-1b3837884337
[2017-05-04 16:10:57,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 34e19c79-78f5-4961-8cf3-bb9a93cd894f
[2017-05-04 16:10:57,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 34e19c79-78f5-4961-8cf3-bb9a93cd894f
[2017-05-04 16:10:57,880: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 34e19c79-78f5-4961-8cf3-bb9a93cd894f
[2017-05-04 16:10:57,880: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 34e19c79-78f5-4961-8cf3-bb9a93cd894f
[2017-05-04 16:10:57,881: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-04 16:10:57.880968+00:00:
[2017-05-04 16:10:57,881: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-04 16:10:57.880968+00:00:
[2017-05-04 16:10:57,881: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-04 16:10:57,881: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-04 16:10:57,910: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-04 16:10:57.910874+00:00:
[2017-05-04 16:10:57,910: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-04 16:10:57.910874+00:00:
[2017-05-04 16:10:57,911: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-04 16:10:57,911: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-04 16:10:57,936: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com restart_agent {}
[2017-05-04 16:10:57,963: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bed2f86c-d2ff-46ff-99ed-bf20fd5ec6c1
[2017-05-04 16:10:57,963: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bed2f86c-d2ff-46ff-99ed-bf20fd5ec6c1
[2017-05-04 16:10:58,035: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/cbf3af92-ef74-45a4-95e7-1b3837884337
[2017-05-04 16:10:58,035: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/cbf3af92-ef74-45a4-95e7-1b3837884337
[2017-05-04 16:10:58,036: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/cbf3af92-ef74-45a4-95e7-1b3837884337
[2017-05-04 16:10:58,036: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/cbf3af92-ef74-45a4-95e7-1b3837884337
[2017-05-04 16:10:58,036: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bed2f86c-d2ff-46ff-99ed-bf20fd5ec6c1
[2017-05-04 16:10:58,036: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bed2f86c-d2ff-46ff-99ed-bf20fd5ec6c1
[2017-05-04 16:10:58,064: INFO/command_plan] Created command 12 (Setup managed host lotus-32vm6.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-05-04 16:10:58,083: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bed2f86c-d2ff-46ff-99ed-bf20fd5ec6c1
[2017-05-04 16:10:58,083: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bed2f86c-d2ff-46ff-99ed-bf20fd5ec6c1
[2017-05-04 16:10:58,083: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old cbf3af92-ef74-45a4-95e7-1b3837884337)
[2017-05-04 16:10:58,083: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old cbf3af92-ef74-45a4-95e7-1b3837884337)
[2017-05-04 16:10:58,084: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 16:10:58,084: INFO/job_scheduler]   Job 9 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-04 16:10:58,084: INFO/job_scheduler]   Job 10 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:58,085: INFO/job_scheduler]   Job 11 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:58,085: INFO/job_scheduler]   Job 12 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:58,085: INFO/job_scheduler]   Job 13 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:58,085: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:58,085: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 16:10:58,085: INFO/job_scheduler] run_next: 1 runnable jobs of (41 pending, 3 tasked)
[2017-05-04 16:10:58,131: INFO/job_scheduler] Job 42: RunJobThread.run
[2017-05-04 16:10:58,132: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-05-04 16:10:58,176: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e5a74769-7670-4d92-8cd6-d383b9ca31a0
[2017-05-04 16:10:58,176: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e5a74769-7670-4d92-8cd6-d383b9ca31a0
[2017-05-04 16:10:58,198: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-04 16:10:58,200: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 16:10:58,200: INFO/job_scheduler]   Job 9 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-04 16:10:58,200: INFO/job_scheduler]   Job 10 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:58,200: INFO/job_scheduler]   Job 11 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:58,201: INFO/job_scheduler]   Job 12 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:58,201: INFO/job_scheduler]   Job 13 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:58,201: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:58,201: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 16:10:58,201: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-04 16:10:58,359: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/316480ef-58c6-4762-8a5c-ac254026c06f
[2017-05-04 16:10:58,359: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/316480ef-58c6-4762-8a5c-ac254026c06f
[2017-05-04 16:10:58,359: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/316480ef-58c6-4762-8a5c-ac254026c06f
[2017-05-04 16:10:58,359: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/316480ef-58c6-4762-8a5c-ac254026c06f
[2017-05-04 16:10:58,359: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc aea5e515-77fe-40ee-9216-0dd2ca15de0d
[2017-05-04 16:10:58,359: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc aea5e515-77fe-40ee-9216-0dd2ca15de0d
[2017-05-04 16:10:58,361: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc aea5e515-77fe-40ee-9216-0dd2ca15de0d
[2017-05-04 16:10:58,361: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc aea5e515-77fe-40ee-9216-0dd2ca15de0d
[2017-05-04 16:10:58,361: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-05-04 16:10:58.361820+00:00:
[2017-05-04 16:10:58,361: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-05-04 16:10:58.361820+00:00:
[2017-05-04 16:10:58,362: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-04 16:10:58,362: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-04 16:10:58,391: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 16:10:58,391: INFO/job_scheduler]   Job 20 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-04 16:10:58,391: INFO/job_scheduler]   Job 21 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:58,392: INFO/job_scheduler]   Job 22 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:58,392: INFO/job_scheduler]   Job 23 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:58,392: INFO/job_scheduler]   Job 24 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:58,392: INFO/job_scheduler]   Job 27 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:58,392: INFO/job_scheduler]   Job 30 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 16:10:58,392: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-04 16:10:58,394: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-05-04 16:10:58.394209+00:00:
[2017-05-04 16:10:58,394: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-05-04 16:10:58.394209+00:00:
[2017-05-04 16:10:58,394: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-04 16:10:58,394: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-04 16:10:58,438: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 16:10:58,438: INFO/job_scheduler]   Job 20 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-04 16:10:58,438: INFO/job_scheduler]   Job 21 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:58,439: INFO/job_scheduler]   Job 22 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:58,439: INFO/job_scheduler]   Job 23 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:58,439: INFO/job_scheduler]   Job 24 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:58,439: INFO/job_scheduler]   Job 27 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:58,439: INFO/job_scheduler]   Job 30 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 16:10:58,439: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-04 16:10:58,444: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com restart_agent {}
[2017-05-04 16:10:58,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0cf46b8d-346e-4bbe-84d9-95902a05ac3f
[2017-05-04 16:10:58,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0cf46b8d-346e-4bbe-84d9-95902a05ac3f
[2017-05-04 16:10:58,542: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/316480ef-58c6-4762-8a5c-ac254026c06f
[2017-05-04 16:10:58,542: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/316480ef-58c6-4762-8a5c-ac254026c06f
[2017-05-04 16:10:58,543: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/316480ef-58c6-4762-8a5c-ac254026c06f
[2017-05-04 16:10:58,543: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/316480ef-58c6-4762-8a5c-ac254026c06f
[2017-05-04 16:10:58,543: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0cf46b8d-346e-4bbe-84d9-95902a05ac3f
[2017-05-04 16:10:58,543: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0cf46b8d-346e-4bbe-84d9-95902a05ac3f
[2017-05-04 16:10:58,590: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0cf46b8d-346e-4bbe-84d9-95902a05ac3f
[2017-05-04 16:10:58,590: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0cf46b8d-346e-4bbe-84d9-95902a05ac3f
[2017-05-04 16:10:58,591: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm7.lotus.hpdd.lab.intel.com (old 316480ef-58c6-4762-8a5c-ac254026c06f)
[2017-05-04 16:10:58,591: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm7.lotus.hpdd.lab.intel.com (old 316480ef-58c6-4762-8a5c-ac254026c06f)
[2017-05-04 16:10:58,819: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/8cd0679b-28b4-479a-bdac-ffbb32db23f7
[2017-05-04 16:10:58,819: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/8cd0679b-28b4-479a-bdac-ffbb32db23f7
[2017-05-04 16:10:58,819: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/8cd0679b-28b4-479a-bdac-ffbb32db23f7
[2017-05-04 16:10:58,819: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/8cd0679b-28b4-479a-bdac-ffbb32db23f7
[2017-05-04 16:10:58,820: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f7849bdd-4f19-406f-b602-73c1f867d43b
[2017-05-04 16:10:58,820: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f7849bdd-4f19-406f-b602-73c1f867d43b
[2017-05-04 16:10:58,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f7849bdd-4f19-406f-b602-73c1f867d43b
[2017-05-04 16:10:58,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f7849bdd-4f19-406f-b602-73c1f867d43b
[2017-05-04 16:10:58,867: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-05-04 16:10:58.867410+00:00:
[2017-05-04 16:10:58,867: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-05-04 16:10:58.867410+00:00:
[2017-05-04 16:10:58,867: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-04 16:10:58,867: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-04 16:10:58,896: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 16:10:58,896: INFO/job_scheduler]   Job 31 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-04 16:10:58,896: INFO/job_scheduler]   Job 32 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:58,897: INFO/job_scheduler]   Job 33 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:58,897: INFO/job_scheduler]   Job 34 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:58,897: INFO/job_scheduler]   Job 35 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:58,897: INFO/job_scheduler]   Job 38 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:58,897: INFO/job_scheduler]   Job 41 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 16:10:58,897: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-04 16:10:58,901: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-05-04 16:10:58.901790+00:00:
[2017-05-04 16:10:58,901: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-05-04 16:10:58.901790+00:00:
[2017-05-04 16:10:58,902: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-04 16:10:58,902: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-04 16:10:58,918: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com restart_agent {}
[2017-05-04 16:10:58,929: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 16:10:58,929: INFO/job_scheduler]   Job 31 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-04 16:10:58,929: INFO/job_scheduler]   Job 32 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:58,929: INFO/job_scheduler]   Job 33 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:58,929: INFO/job_scheduler]   Job 34 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:58,930: INFO/job_scheduler]   Job 35 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:58,930: INFO/job_scheduler]   Job 38 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:58,930: INFO/job_scheduler]   Job 41 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 16:10:58,930: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-04 16:10:58,944: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 27d0ca23-810b-4f27-b1f3-831fef86f5ba
[2017-05-04 16:10:58,944: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 27d0ca23-810b-4f27-b1f3-831fef86f5ba
[2017-05-04 16:10:58,992: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/8cd0679b-28b4-479a-bdac-ffbb32db23f7
[2017-05-04 16:10:58,992: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/8cd0679b-28b4-479a-bdac-ffbb32db23f7
[2017-05-04 16:10:58,992: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/8cd0679b-28b4-479a-bdac-ffbb32db23f7
[2017-05-04 16:10:58,992: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/8cd0679b-28b4-479a-bdac-ffbb32db23f7
[2017-05-04 16:10:58,992: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 27d0ca23-810b-4f27-b1f3-831fef86f5ba
[2017-05-04 16:10:58,992: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 27d0ca23-810b-4f27-b1f3-831fef86f5ba
[2017-05-04 16:10:59,009: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 27d0ca23-810b-4f27-b1f3-831fef86f5ba
[2017-05-04 16:10:59,009: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 27d0ca23-810b-4f27-b1f3-831fef86f5ba
[2017-05-04 16:10:59,009: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm8.lotus.hpdd.lab.intel.com (old 8cd0679b-28b4-479a-bdac-ffbb32db23f7)
[2017-05-04 16:10:59,009: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm8.lotus.hpdd.lab.intel.com (old 8cd0679b-28b4-479a-bdac-ffbb32db23f7)
[2017-05-04 16:10:59,316: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/ec71d10c-93ee-4142-b94d-1c0407491462
[2017-05-04 16:10:59,316: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/ec71d10c-93ee-4142-b94d-1c0407491462
[2017-05-04 16:10:59,317: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/ec71d10c-93ee-4142-b94d-1c0407491462
[2017-05-04 16:10:59,317: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/ec71d10c-93ee-4142-b94d-1c0407491462
[2017-05-04 16:10:59,317: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e5a74769-7670-4d92-8cd6-d383b9ca31a0
[2017-05-04 16:10:59,317: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e5a74769-7670-4d92-8cd6-d383b9ca31a0
[2017-05-04 16:10:59,343: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e5a74769-7670-4d92-8cd6-d383b9ca31a0
[2017-05-04 16:10:59,343: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e5a74769-7670-4d92-8cd6-d383b9ca31a0
[2017-05-04 16:10:59,343: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-05-04 16:10:59.343735+00:00:
[2017-05-04 16:10:59,343: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-05-04 16:10:59.343735+00:00:
[2017-05-04 16:10:59,343: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-04 16:10:59,343: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-04 16:10:59,359: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 16:10:59,359: INFO/job_scheduler]   Job 42 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-04 16:10:59,359: INFO/job_scheduler]   Job 43 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:59,359: INFO/job_scheduler]   Job 44 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:59,359: INFO/job_scheduler]   Job 45 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:59,359: INFO/job_scheduler]   Job 46 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:59,359: INFO/job_scheduler]   Job 50 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:59,359: INFO/job_scheduler]   Job 52 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 16:10:59,360: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-04 16:10:59,361: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-05-04 16:10:59.361489+00:00:
[2017-05-04 16:10:59,361: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-05-04 16:10:59.361489+00:00:
[2017-05-04 16:10:59,361: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-04 16:10:59,361: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-04 16:10:59,373: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 16:10:59,373: INFO/job_scheduler]   Job 42 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-04 16:10:59,373: INFO/job_scheduler]   Job 43 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:59,373: INFO/job_scheduler]   Job 44 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:59,373: INFO/job_scheduler]   Job 45 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:59,374: INFO/job_scheduler]   Job 46 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:59,374: INFO/job_scheduler]   Job 50 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 16:10:59,374: INFO/job_scheduler]   Job 52 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 16:10:59,374: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-04 16:10:59,375: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com restart_agent {}
[2017-05-04 16:10:59,396: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5324e293-e3f8-4a37-8be6-587c4887c259
[2017-05-04 16:10:59,396: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5324e293-e3f8-4a37-8be6-587c4887c259
[2017-05-04 16:10:59,446: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/ec71d10c-93ee-4142-b94d-1c0407491462
[2017-05-04 16:10:59,446: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/ec71d10c-93ee-4142-b94d-1c0407491462
[2017-05-04 16:10:59,446: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/ec71d10c-93ee-4142-b94d-1c0407491462
[2017-05-04 16:10:59,446: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/ec71d10c-93ee-4142-b94d-1c0407491462
[2017-05-04 16:10:59,447: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5324e293-e3f8-4a37-8be6-587c4887c259
[2017-05-04 16:10:59,447: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5324e293-e3f8-4a37-8be6-587c4887c259
[2017-05-04 16:10:59,469: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5324e293-e3f8-4a37-8be6-587c4887c259
[2017-05-04 16:10:59,469: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5324e293-e3f8-4a37-8be6-587c4887c259
[2017-05-04 16:10:59,469: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm6.lotus.hpdd.lab.intel.com (old ec71d10c-93ee-4142-b94d-1c0407491462)
[2017-05-04 16:10:59,469: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm6.lotus.hpdd.lab.intel.com (old ec71d10c-93ee-4142-b94d-1c0407491462)
[2017-05-04 16:11:28,117: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com timeout after 30.0s
[2017-05-04 16:11:28,117: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com timeout after 30.0s
[2017-05-04 16:11:28,624: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm7.lotus.hpdd.lab.intel.com timeout after 30.0s
[2017-05-04 16:11:28,624: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm7.lotus.hpdd.lab.intel.com timeout after 30.0s
[2017-05-04 16:11:28,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/8cd0679b-28b4-479a-bdac-ffbb32db23f7
[2017-05-04 16:11:28,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/8cd0679b-28b4-479a-bdac-ffbb32db23f7
[2017-05-04 16:11:28,859: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 8cd0679b-28b4-479a-bdac-ffbb32db23f7 because session terminated
[2017-05-04 16:11:28,859: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 8cd0679b-28b4-479a-bdac-ffbb32db23f7 because session terminated
[2017-05-04 16:11:28,861: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:11:28,861: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:11:29,048: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm8.lotus.hpdd.lab.intel.com new d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:11:29,048: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm8.lotus.hpdd.lab.intel.com new d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:11:29,086: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-04 16:11:29,102: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 53deb3c0-9f99-495d-badd-a43e4f0a0017
[2017-05-04 16:11:29,102: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 53deb3c0-9f99-495d-badd-a43e4f0a0017
[2017-05-04 16:11:29,118: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com timeout after 31.0s
[2017-05-04 16:11:29,118: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com timeout after 31.0s
[2017-05-04 16:11:29,185: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:11:29,185: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:11:29,186: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:11:29,186: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:11:29,186: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 53deb3c0-9f99-495d-badd-a43e4f0a0017
[2017-05-04 16:11:29,186: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 53deb3c0-9f99-495d-badd-a43e4f0a0017
[2017-05-04 16:11:29,217: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 53deb3c0-9f99-495d-badd-a43e4f0a0017
[2017-05-04 16:11:29,217: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 53deb3c0-9f99-495d-badd-a43e4f0a0017
[2017-05-04 16:11:29,218: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-05-04 16:11:29,253: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 128b69df-3f89-4cdf-9766-68fc288dba70
[2017-05-04 16:11:29,253: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 128b69df-3f89-4cdf-9766-68fc288dba70
[2017-05-04 16:11:29,500: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm6.lotus.hpdd.lab.intel.com timeout after 30.0s
[2017-05-04 16:11:29,500: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm6.lotus.hpdd.lab.intel.com timeout after 30.0s
[2017-05-04 16:11:29,625: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm7.lotus.hpdd.lab.intel.com timeout after 31.0s
[2017-05-04 16:11:29,625: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm7.lotus.hpdd.lab.intel.com timeout after 31.0s
[2017-05-04 16:11:29,792: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/cbf3af92-ef74-45a4-95e7-1b3837884337
[2017-05-04 16:11:29,792: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/cbf3af92-ef74-45a4-95e7-1b3837884337
[2017-05-04 16:11:29,792: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session cbf3af92-ef74-45a4-95e7-1b3837884337 because session terminated
[2017-05-04 16:11:29,792: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session cbf3af92-ef74-45a4-95e7-1b3837884337 because session terminated
[2017-05-04 16:11:29,793: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:11:29,793: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:11:30,120: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:11:30,120: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:11:30,120: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-04 16:11:30,133: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d5d05677-58b2-4fe8-88b9-dcc386f73363
[2017-05-04 16:11:30,133: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d5d05677-58b2-4fe8-88b9-dcc386f73363
[2017-05-04 16:11:30,182: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/ec71d10c-93ee-4142-b94d-1c0407491462
[2017-05-04 16:11:30,182: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/ec71d10c-93ee-4142-b94d-1c0407491462
[2017-05-04 16:11:30,182: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session ec71d10c-93ee-4142-b94d-1c0407491462 because session terminated
[2017-05-04 16:11:30,182: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session ec71d10c-93ee-4142-b94d-1c0407491462 because session terminated
[2017-05-04 16:11:30,185: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:11:30,185: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:11:30,192: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:11:30,192: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:11:30,192: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:11:30,192: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:11:30,192: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d5d05677-58b2-4fe8-88b9-dcc386f73363
[2017-05-04 16:11:30,192: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d5d05677-58b2-4fe8-88b9-dcc386f73363
[2017-05-04 16:11:30,202: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d5d05677-58b2-4fe8-88b9-dcc386f73363
[2017-05-04 16:11:30,202: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d5d05677-58b2-4fe8-88b9-dcc386f73363
[2017-05-04 16:11:30,202: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-05-04 16:11:30,214: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 746d7e07-f7cd-4b3a-8029-513d17ff5c83
[2017-05-04 16:11:30,214: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 746d7e07-f7cd-4b3a-8029-513d17ff5c83
[2017-05-04 16:11:30,417: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/316480ef-58c6-4762-8a5c-ac254026c06f
[2017-05-04 16:11:30,417: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/316480ef-58c6-4762-8a5c-ac254026c06f
[2017-05-04 16:11:30,417: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 316480ef-58c6-4762-8a5c-ac254026c06f because session terminated
[2017-05-04 16:11:30,417: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 316480ef-58c6-4762-8a5c-ac254026c06f because session terminated
[2017-05-04 16:11:30,420: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:11:30,420: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:11:30,501: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm6.lotus.hpdd.lab.intel.com new 2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:11:30,501: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm6.lotus.hpdd.lab.intel.com new 2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:11:30,502: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-04 16:11:30,516: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6147e7fa-a221-4f19-a0f5-d33da0b88bb2
[2017-05-04 16:11:30,516: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6147e7fa-a221-4f19-a0f5-d33da0b88bb2
[2017-05-04 16:11:30,627: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm7.lotus.hpdd.lab.intel.com new efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:11:30,627: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm7.lotus.hpdd.lab.intel.com new efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:11:30,628: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-04 16:11:30,639: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 01ad70ae-b9e7-4589-95f4-11a37772ea83
[2017-05-04 16:11:30,639: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 01ad70ae-b9e7-4589-95f4-11a37772ea83
[2017-05-04 16:11:30,718: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:11:30,718: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:11:30,718: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:11:30,718: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:11:30,718: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 01ad70ae-b9e7-4589-95f4-11a37772ea83
[2017-05-04 16:11:30,718: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 01ad70ae-b9e7-4589-95f4-11a37772ea83
[2017-05-04 16:11:30,761: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 01ad70ae-b9e7-4589-95f4-11a37772ea83
[2017-05-04 16:11:30,761: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 01ad70ae-b9e7-4589-95f4-11a37772ea83
[2017-05-04 16:11:30,762: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-05-04 16:11:30,775: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d46a3a48-6ebf-4706-8d37-9b7f0f8b6e02
[2017-05-04 16:11:30,775: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d46a3a48-6ebf-4706-8d37-9b7f0f8b6e02
[2017-05-04 16:11:31,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:11:31,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:11:31,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:11:31,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:11:31,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6147e7fa-a221-4f19-a0f5-d33da0b88bb2
[2017-05-04 16:11:31,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6147e7fa-a221-4f19-a0f5-d33da0b88bb2
[2017-05-04 16:11:31,686: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6147e7fa-a221-4f19-a0f5-d33da0b88bb2
[2017-05-04 16:11:31,686: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6147e7fa-a221-4f19-a0f5-d33da0b88bb2
[2017-05-04 16:11:31,687: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-05-04 16:11:31,703: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 26856841-0e30-4a33-83d2-e6079cce9357
[2017-05-04 16:11:31,703: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 26856841-0e30-4a33-83d2-e6079cce9357
[2017-05-04 16:11:40,088: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 16:11:40,088: INFO/job_scheduler]   Job 42 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-04 16:11:40,088: INFO/job_scheduler]   Job 43 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 16:11:40,089: INFO/job_scheduler]   Job 44 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 16:11:40,089: INFO/job_scheduler]   Job 45 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 16:11:40,089: INFO/job_scheduler]   Job 46 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 16:11:40,089: INFO/job_scheduler]   Job 50 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 16:11:40,089: INFO/job_scheduler]   Job 52 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 16:11:40,089: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-04 16:11:40,584: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 16:11:40,584: INFO/job_scheduler]   Job 20 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-04 16:11:40,584: INFO/job_scheduler]   Job 21 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 16:11:40,584: INFO/job_scheduler]   Job 22 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 16:11:40,584: INFO/job_scheduler]   Job 23 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 16:11:40,584: INFO/job_scheduler]   Job 24 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 16:11:40,585: INFO/job_scheduler]   Job 27 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 16:11:40,587: INFO/job_scheduler]   Job 30 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 16:11:40,587: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-04 16:11:40,977: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 16:11:40,977: INFO/job_scheduler]   Job 9 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-04 16:11:40,977: INFO/job_scheduler]   Job 10 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 16:11:40,977: INFO/job_scheduler]   Job 11 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 16:11:40,977: INFO/job_scheduler]   Job 12 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 16:11:40,977: INFO/job_scheduler]   Job 13 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 16:11:40,978: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 16:11:40,978: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 16:11:40,978: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-04 16:11:41,330: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-04 16:11:41,330: INFO/job_scheduler]   Job 31 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-04 16:11:41,330: INFO/job_scheduler]   Job 32 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 16:11:41,330: INFO/job_scheduler]   Job 33 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 16:11:41,331: INFO/job_scheduler]   Job 34 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 16:11:41,331: INFO/job_scheduler]   Job 35 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 16:11:41,331: INFO/job_scheduler]   Job 38 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 16:11:41,331: INFO/job_scheduler]   Job 41 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-04 16:11:41,331: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-04 16:12:09,688: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:09,688: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:09,688: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:09,688: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:09,689: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 128b69df-3f89-4cdf-9766-68fc288dba70
[2017-05-04 16:12:09,689: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 128b69df-3f89-4cdf-9766-68fc288dba70
[2017-05-04 16:12:09,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 128b69df-3f89-4cdf-9766-68fc288dba70
[2017-05-04 16:12:09,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 128b69df-3f89-4cdf-9766-68fc288dba70
[2017-05-04 16:12:10,022: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com install_packages {'repos': [u'e2fsprogs', u'iml-agent', u'lustre', u'zfs'], 'packages': [u'chroma-agent-management', u'kernel-devel-3.10.0-514.10.2.el7_lustre', u'lustre-osd-zfs', u'lustre-osd-ldiskfs', u'lustre-modules', u'lustre', u'zfs']}
[2017-05-04 16:12:10,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6afb8856-c579-452d-a977-65fe1a1ae976
[2017-05-04 16:12:10,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6afb8856-c579-452d-a977-65fe1a1ae976
[2017-05-04 16:12:10,376: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:10,376: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:10,376: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:10,376: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:10,376: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d46a3a48-6ebf-4706-8d37-9b7f0f8b6e02
[2017-05-04 16:12:10,376: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d46a3a48-6ebf-4706-8d37-9b7f0f8b6e02
[2017-05-04 16:12:10,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d46a3a48-6ebf-4706-8d37-9b7f0f8b6e02
[2017-05-04 16:12:10,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d46a3a48-6ebf-4706-8d37-9b7f0f8b6e02
[2017-05-04 16:12:10,796: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com install_packages {'repos': [u'e2fsprogs', u'iml-agent', u'lustre', u'zfs'], 'packages': [u'chroma-agent-management', u'kernel-devel-3.10.0-514.10.2.el7_lustre', u'lustre-osd-zfs', u'lustre-osd-ldiskfs', u'lustre-modules', u'lustre', u'zfs']}
[2017-05-04 16:12:10,810: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ae75604e-5be8-47dc-983d-bb2ea53b7de0
[2017-05-04 16:12:10,810: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ae75604e-5be8-47dc-983d-bb2ea53b7de0
[2017-05-04 16:12:27,913: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:27,913: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:27,913: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:27,913: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:27,913: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 746d7e07-f7cd-4b3a-8029-513d17ff5c83
[2017-05-04 16:12:27,913: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 746d7e07-f7cd-4b3a-8029-513d17ff5c83
[2017-05-04 16:12:27,957: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 746d7e07-f7cd-4b3a-8029-513d17ff5c83
[2017-05-04 16:12:27,957: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 746d7e07-f7cd-4b3a-8029-513d17ff5c83
[2017-05-04 16:12:28,292: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:28,292: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:28,293: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:28,293: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:28,293: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 26856841-0e30-4a33-83d2-e6079cce9357
[2017-05-04 16:12:28,293: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 26856841-0e30-4a33-83d2-e6079cce9357
[2017-05-04 16:12:28,301: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 26856841-0e30-4a33-83d2-e6079cce9357
[2017-05-04 16:12:28,301: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 26856841-0e30-4a33-83d2-e6079cce9357
[2017-05-04 16:12:28,437: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com install_packages {'repos': [u'e2fsprogs', u'iml-agent', u'lustre', u'zfs'], 'packages': [u'chroma-agent-management', u'kernel-devel-3.10.0-514.10.2.el7_lustre', u'lustre-osd-zfs', u'lustre-osd-ldiskfs', u'lustre-modules', u'lustre', u'zfs']}
[2017-05-04 16:12:28,454: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc de1ec68f-6ecf-47ec-9cad-e0b92108b042
[2017-05-04 16:12:28,454: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc de1ec68f-6ecf-47ec-9cad-e0b92108b042
[2017-05-04 16:12:28,480: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:28,480: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:28,481: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:28,481: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:28,481: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6afb8856-c579-452d-a977-65fe1a1ae976
[2017-05-04 16:12:28,481: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6afb8856-c579-452d-a977-65fe1a1ae976
[2017-05-04 16:12:28,483: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6afb8856-c579-452d-a977-65fe1a1ae976
[2017-05-04 16:12:28,483: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6afb8856-c579-452d-a977-65fe1a1ae976
[2017-05-04 16:12:29,004: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com kernel_status {}
[2017-05-04 16:12:29,017: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8ce81c37-e8ba-461c-9f10-618411c42764
[2017-05-04 16:12:29,017: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8ce81c37-e8ba-461c-9f10-618411c42764
[2017-05-04 16:12:29,073: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com install_packages {'repos': [u'e2fsprogs', u'iml-agent', u'lustre', u'zfs'], 'packages': [u'chroma-agent-management', u'kernel-devel-3.10.0-514.10.2.el7_lustre', u'lustre-osd-zfs', u'lustre-osd-ldiskfs', u'lustre-modules', u'lustre', u'zfs']}
[2017-05-04 16:12:29,092: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 30c0593c-3272-4a95-bec1-6003dbb5ccbc
[2017-05-04 16:12:29,092: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 30c0593c-3272-4a95-bec1-6003dbb5ccbc
[2017-05-04 16:12:31,137: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:31,137: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:31,137: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:31,137: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:31,137: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8ce81c37-e8ba-461c-9f10-618411c42764
[2017-05-04 16:12:31,137: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8ce81c37-e8ba-461c-9f10-618411c42764
[2017-05-04 16:12:31,185: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8ce81c37-e8ba-461c-9f10-618411c42764
[2017-05-04 16:12:31,185: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8ce81c37-e8ba-461c-9f10-618411c42764
[2017-05-04 16:12:31,185: INFO/job_scheduler] Job 31 finished 4 steps successfully
[2017-05-04 16:12:31,363: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 139657774322576
[2017-05-04 16:12:31,368: INFO/job] Job 31: StateChangeJob complete, setting state packages_installed on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 16:12:31,368: INFO/job] Job 31: not_deleted=True
[2017-05-04 16:12:31,368: INFO/job_scheduler] Job 31: complete_job: Updating cache
[2017-05-04 16:12:31,368: INFO/job_scheduler] Job 31: locked_item 139657908293712 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-04 16:12:31,371: INFO/job_scheduler] Job 31 completing (errored=False, cancelled=False)
[2017-05-04 16:12:31,408: INFO/job_scheduler] run_next: 4 runnable jobs of (40 pending, 3 tasked)
[2017-05-04 16:12:31,427: INFO/job_scheduler] Job 32: RunJobThread.run
[2017-05-04 16:12:31,428: INFO/job_scheduler] Job 33: RunJobThread.run
[2017-05-04 16:12:31,428: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-05-04 16:12:31,439: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-05-04 16:12:31,449: INFO/job_scheduler] Job 34: RunJobThread.run
[2017-05-04 16:12:31,449: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-05-04 16:12:31,451: INFO/job_scheduler] Job 35: open DB connection during completion
[2017-05-04 16:12:31,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bd971ac4-8963-47c1-b627-b58535d449ad
[2017-05-04 16:12:31,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bd971ac4-8963-47c1-b627-b58535d449ad
[2017-05-04 16:12:31,480: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 139657908284816
[2017-05-04 16:12:31,490: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-04 16:12:31,499: INFO/job] Job 35: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-04 16:12:31,499: INFO/job] Job 35: not_deleted=True
[2017-05-04 16:12:31,499: INFO/job_scheduler] Job 35: complete_job: Updating cache
[2017-05-04 16:12:31,499: INFO/job_scheduler] Job 35: locked_item 139657908284816 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-04 16:12:31,510: INFO/job_scheduler] Job 35 completing (errored=False, cancelled=False)
[2017-05-04 16:12:31,517: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 951e8393-6611-4c12-badd-77bffc06eedb
[2017-05-04 16:12:31,517: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 951e8393-6611-4c12-badd-77bffc06eedb
[2017-05-04 16:12:31,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0e39836b-8aae-4b6a-bbb7-f28e1f60c7fd
[2017-05-04 16:12:31,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0e39836b-8aae-4b6a-bbb7-f28e1f60c7fd
[2017-05-04 16:12:31,571: INFO/job_scheduler] run_next: 1 runnable jobs of (36 pending, 6 tasked)
[2017-05-04 16:12:31,586: INFO/job_scheduler] Job 36: RunJobThread.run
[2017-05-04 16:12:31,587: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com load_lnet {}
[2017-05-04 16:12:31,617: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 27341c6f-f370-4c20-bca3-7220ce52c529
[2017-05-04 16:12:31,617: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 27341c6f-f370-4c20-bca3-7220ce52c529
[2017-05-04 16:12:31,668: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:31,668: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:31,668: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:31,668: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:31,668: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ae75604e-5be8-47dc-983d-bb2ea53b7de0
[2017-05-04 16:12:31,668: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ae75604e-5be8-47dc-983d-bb2ea53b7de0
[2017-05-04 16:12:31,707: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ae75604e-5be8-47dc-983d-bb2ea53b7de0
[2017-05-04 16:12:31,707: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ae75604e-5be8-47dc-983d-bb2ea53b7de0
[2017-05-04 16:12:32,422: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com kernel_status {}
[2017-05-04 16:12:32,445: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b8e4d4ab-bfd1-4074-99e7-8b4532abe83b
[2017-05-04 16:12:32,445: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b8e4d4ab-bfd1-4074-99e7-8b4532abe83b
[2017-05-04 16:12:32,679: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:32,679: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:32,679: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:32,679: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:32,679: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 27341c6f-f370-4c20-bca3-7220ce52c529
[2017-05-04 16:12:32,679: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 27341c6f-f370-4c20-bca3-7220ce52c529
[2017-05-04 16:12:32,682: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 27341c6f-f370-4c20-bca3-7220ce52c529
[2017-05-04 16:12:32,682: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 27341c6f-f370-4c20-bca3-7220ce52c529
[2017-05-04 16:12:32,683: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-04 16:12:32,699: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 70e2c9fc-e1c2-48ca-93a7-7bb891c263a2
[2017-05-04 16:12:32,699: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 70e2c9fc-e1c2-48ca-93a7-7bb891c263a2
[2017-05-04 16:12:32,772: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:32,772: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:32,772: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:32,772: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:32,772: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 70e2c9fc-e1c2-48ca-93a7-7bb891c263a2
[2017-05-04 16:12:32,772: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 70e2c9fc-e1c2-48ca-93a7-7bb891c263a2
[2017-05-04 16:12:32,817: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 70e2c9fc-e1c2-48ca-93a7-7bb891c263a2
[2017-05-04 16:12:32,817: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 70e2c9fc-e1c2-48ca-93a7-7bb891c263a2
[2017-05-04 16:12:33,392: INFO/job_scheduler] Job 36 finished 2 steps successfully
[2017-05-04 16:12:33,460: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 139657908284816
[2017-05-04 16:12:33,468: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-04 16:12:33,471: INFO/job] Job 36: StateChangeJob complete, setting state lnet_down on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-04 16:12:33,471: INFO/job] Job 36: not_deleted=True
[2017-05-04 16:12:33,472: INFO/job_scheduler] Job 36: complete_job: Updating cache
[2017-05-04 16:12:33,472: INFO/job_scheduler] Job 36: locked_item 139657908284816 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-04 16:12:33,490: INFO/job_scheduler] Job 36 completing (errored=False, cancelled=False)
[2017-05-04 16:12:33,595: INFO/job_scheduler] run_next: 1 runnable jobs of (35 pending, 6 tasked)
[2017-05-04 16:12:33,600: INFO/job_scheduler] Job 39: RunJobThread.run
[2017-05-04 16:12:33,604: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_lnet {}
[2017-05-04 16:12:33,633: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4ed18bde-c860-40c1-b669-489e1c75a3c1
[2017-05-04 16:12:33,633: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4ed18bde-c860-40c1-b669-489e1c75a3c1
[2017-05-04 16:12:34,519: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:34,519: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:34,519: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:34,519: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:34,519: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b8e4d4ab-bfd1-4074-99e7-8b4532abe83b
[2017-05-04 16:12:34,519: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b8e4d4ab-bfd1-4074-99e7-8b4532abe83b
[2017-05-04 16:12:34,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b8e4d4ab-bfd1-4074-99e7-8b4532abe83b
[2017-05-04 16:12:34,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b8e4d4ab-bfd1-4074-99e7-8b4532abe83b
[2017-05-04 16:12:34,564: INFO/job_scheduler] Job 20 finished 4 steps successfully
[2017-05-04 16:12:34,647: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 139657774272656
[2017-05-04 16:12:34,652: INFO/job] Job 20: StateChangeJob complete, setting state packages_installed on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 16:12:34,652: INFO/job] Job 20: not_deleted=True
[2017-05-04 16:12:34,652: INFO/job_scheduler] Job 20: complete_job: Updating cache
[2017-05-04 16:12:34,652: INFO/job_scheduler] Job 20: locked_item 139657774272400 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-04 16:12:34,655: INFO/job_scheduler] Job 20 completing (errored=False, cancelled=False)
[2017-05-04 16:12:34,675: INFO/job_scheduler] run_next: 4 runnable jobs of (34 pending, 6 tasked)
[2017-05-04 16:12:34,722: INFO/job_scheduler] Job 21: RunJobThread.run
[2017-05-04 16:12:34,723: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-05-04 16:12:34,748: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8eee844b-ec47-4355-bcc0-fbad435c8870
[2017-05-04 16:12:34,748: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8eee844b-ec47-4355-bcc0-fbad435c8870
[2017-05-04 16:12:34,749: INFO/job_scheduler] Job 22: RunJobThread.run
[2017-05-04 16:12:34,754: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-05-04 16:12:34,779: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 50b7bc2a-cbd2-48db-9838-f9ad50fa3de0
[2017-05-04 16:12:34,779: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 50b7bc2a-cbd2-48db-9838-f9ad50fa3de0
[2017-05-04 16:12:34,777: INFO/job_scheduler] Job 23: RunJobThread.run
[2017-05-04 16:12:34,781: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-05-04 16:12:34,777: INFO/job_scheduler] Job 24: open DB connection during completion
[2017-05-04 16:12:34,820: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 48d1b942-a553-4849-af53-6009bded43a9
[2017-05-04 16:12:34,820: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 48d1b942-a553-4849-af53-6009bded43a9
[2017-05-04 16:12:34,856: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 139657774326288
[2017-05-04 16:12:34,868: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-04 16:12:34,883: INFO/job] Job 24: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-04 16:12:34,883: INFO/job] Job 24: not_deleted=True
[2017-05-04 16:12:34,883: INFO/job_scheduler] Job 24: complete_job: Updating cache
[2017-05-04 16:12:34,883: INFO/job_scheduler] Job 24: locked_item 139657774326288 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-04 16:12:34,892: INFO/job_scheduler] Job 24 completing (errored=False, cancelled=False)
[2017-05-04 16:12:34,929: INFO/job_scheduler] run_next: 1 runnable jobs of (30 pending, 9 tasked)
[2017-05-04 16:12:34,952: INFO/job_scheduler] Job 25: RunJobThread.run
[2017-05-04 16:12:34,953: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com load_lnet {}
[2017-05-04 16:12:34,986: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d84db817-e616-4bf1-97a0-579f8deef2bf
[2017-05-04 16:12:34,986: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d84db817-e616-4bf1-97a0-579f8deef2bf
[2017-05-04 16:12:35,080: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:35,080: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:35,080: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:35,080: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:35,080: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d84db817-e616-4bf1-97a0-579f8deef2bf
[2017-05-04 16:12:35,080: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d84db817-e616-4bf1-97a0-579f8deef2bf
[2017-05-04 16:12:35,112: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d84db817-e616-4bf1-97a0-579f8deef2bf
[2017-05-04 16:12:35,112: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d84db817-e616-4bf1-97a0-579f8deef2bf
[2017-05-04 16:12:35,113: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-04 16:12:35,127: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 726fe5cc-630b-477a-b128-cdb20f4efe45
[2017-05-04 16:12:35,127: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 726fe5cc-630b-477a-b128-cdb20f4efe45
[2017-05-04 16:12:35,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:35,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:35,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:35,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:35,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 726fe5cc-630b-477a-b128-cdb20f4efe45
[2017-05-04 16:12:35,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 726fe5cc-630b-477a-b128-cdb20f4efe45
[2017-05-04 16:12:35,249: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 726fe5cc-630b-477a-b128-cdb20f4efe45
[2017-05-04 16:12:35,249: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 726fe5cc-630b-477a-b128-cdb20f4efe45
[2017-05-04 16:12:35,755: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:35,755: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:35,755: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:35,755: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:35,756: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4ed18bde-c860-40c1-b669-489e1c75a3c1
[2017-05-04 16:12:35,756: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4ed18bde-c860-40c1-b669-489e1c75a3c1
[2017-05-04 16:12:35,756: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4ed18bde-c860-40c1-b669-489e1c75a3c1
[2017-05-04 16:12:35,756: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4ed18bde-c860-40c1-b669-489e1c75a3c1
[2017-05-04 16:12:35,757: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-04 16:12:35,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f680cf46-a394-4ec8-9e35-e67126f29d49
[2017-05-04 16:12:35,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f680cf46-a394-4ec8-9e35-e67126f29d49
[2017-05-04 16:12:35,878: INFO/job_scheduler] Job 25 finished 2 steps successfully
[2017-05-04 16:12:35,928: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 139657774326288
[2017-05-04 16:12:35,934: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-04 16:12:35,937: INFO/job] Job 25: StateChangeJob complete, setting state lnet_down on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-04 16:12:35,937: INFO/job] Job 25: not_deleted=True
[2017-05-04 16:12:35,937: INFO/job_scheduler] Job 25: complete_job: Updating cache
[2017-05-04 16:12:35,937: INFO/job_scheduler] Job 25: locked_item 139657774326288 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-04 16:12:35,941: INFO/job_scheduler] Job 25 completing (errored=False, cancelled=False)
[2017-05-04 16:12:35,942: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:35,942: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:35,942: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:35,942: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:35,942: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f680cf46-a394-4ec8-9e35-e67126f29d49
[2017-05-04 16:12:35,942: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f680cf46-a394-4ec8-9e35-e67126f29d49
[2017-05-04 16:12:35,975: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f680cf46-a394-4ec8-9e35-e67126f29d49
[2017-05-04 16:12:35,975: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f680cf46-a394-4ec8-9e35-e67126f29d49
[2017-05-04 16:12:35,986: INFO/job_scheduler] run_next: 1 runnable jobs of (29 pending, 9 tasked)
[2017-05-04 16:12:35,998: INFO/job_scheduler] Job 28: RunJobThread.run
[2017-05-04 16:12:35,998: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_lnet {}
[2017-05-04 16:12:36,035: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b2e16a46-9ce1-419e-b230-4f939cc84c80
[2017-05-04 16:12:36,035: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b2e16a46-9ce1-419e-b230-4f939cc84c80
[2017-05-04 16:12:36,081: INFO/job_scheduler] Job 39 finished 2 steps successfully
[2017-05-04 16:12:36,112: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 139657908284816
[2017-05-04 16:12:36,152: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm8.lotus.hpdd.lab.intel.com at severity 30
[2017-05-04 16:12:36,197: INFO/job] Job 39: StateChangeJob complete, setting state lnet_up on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-04 16:12:36,197: INFO/job] Job 39: not_deleted=True
[2017-05-04 16:12:36,197: INFO/job_scheduler] Job 39: complete_job: Updating cache
[2017-05-04 16:12:36,197: INFO/job_scheduler] Job 39: locked_item 139657908284816 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-04 16:12:36,206: INFO/job_scheduler] Job 39 completing (errored=False, cancelled=False)
[2017-05-04 16:12:36,241: INFO/job_scheduler] run_next: 0 runnable jobs of (28 pending, 9 tasked)
[2017-05-04 16:12:37,160: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:37,160: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:37,170: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:37,170: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:37,170: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b2e16a46-9ce1-419e-b230-4f939cc84c80
[2017-05-04 16:12:37,170: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b2e16a46-9ce1-419e-b230-4f939cc84c80
[2017-05-04 16:12:37,201: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b2e16a46-9ce1-419e-b230-4f939cc84c80
[2017-05-04 16:12:37,201: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b2e16a46-9ce1-419e-b230-4f939cc84c80
[2017-05-04 16:12:37,202: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-04 16:12:37,214: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 97da7ccb-107d-42bc-bd25-c7d347354e0b
[2017-05-04 16:12:37,214: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 97da7ccb-107d-42bc-bd25-c7d347354e0b
[2017-05-04 16:12:37,332: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:37,332: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:37,332: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:37,332: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:37,332: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 97da7ccb-107d-42bc-bd25-c7d347354e0b
[2017-05-04 16:12:37,332: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 97da7ccb-107d-42bc-bd25-c7d347354e0b
[2017-05-04 16:12:37,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 97da7ccb-107d-42bc-bd25-c7d347354e0b
[2017-05-04 16:12:37,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 97da7ccb-107d-42bc-bd25-c7d347354e0b
[2017-05-04 16:12:37,417: INFO/job_scheduler] Job 28 finished 2 steps successfully
[2017-05-04 16:12:37,470: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 139657774326288
[2017-05-04 16:12:37,487: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 30
[2017-05-04 16:12:37,494: INFO/job] Job 28: StateChangeJob complete, setting state lnet_up on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-04 16:12:37,494: INFO/job] Job 28: not_deleted=True
[2017-05-04 16:12:37,494: INFO/job_scheduler] Job 28: complete_job: Updating cache
[2017-05-04 16:12:37,494: INFO/job_scheduler] Job 28: locked_item 139657774326288 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-04 16:12:37,504: INFO/job_scheduler] Job 28 completing (errored=False, cancelled=False)
[2017-05-04 16:12:37,547: INFO/job_scheduler] run_next: 0 runnable jobs of (28 pending, 8 tasked)
[2017-05-04 16:12:37,660: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:37,660: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:37,660: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:37,660: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:37,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 951e8393-6611-4c12-badd-77bffc06eedb
[2017-05-04 16:12:37,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 951e8393-6611-4c12-badd-77bffc06eedb
[2017-05-04 16:12:37,701: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 951e8393-6611-4c12-badd-77bffc06eedb
[2017-05-04 16:12:37,701: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 951e8393-6611-4c12-badd-77bffc06eedb
[2017-05-04 16:12:37,701: INFO/job_scheduler] Job 32 finished 1 steps successfully
[2017-05-04 16:12:37,894: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 139657908284368
[2017-05-04 16:12:37,899: INFO/job] Job 32: StateChangeJob complete, setting state configured on lotus-32vm8.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-04 16:12:37,899: INFO/job] Job 32: not_deleted=True
[2017-05-04 16:12:37,899: INFO/job_scheduler] Job 32: complete_job: Updating cache
[2017-05-04 16:12:37,899: INFO/job_scheduler] Job 32: locked_item 139657908284368 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-04 16:12:37,903: INFO/job_scheduler] Job 32 completing (errored=False, cancelled=False)
[2017-05-04 16:12:37,933: INFO/job_scheduler] run_next: 0 runnable jobs of (28 pending, 7 tasked)
[2017-05-04 16:12:38,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:38,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:38,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:38,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:38,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bd971ac4-8963-47c1-b627-b58535d449ad
[2017-05-04 16:12:38,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bd971ac4-8963-47c1-b627-b58535d449ad
[2017-05-04 16:12:38,643: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bd971ac4-8963-47c1-b627-b58535d449ad
[2017-05-04 16:12:38,643: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bd971ac4-8963-47c1-b627-b58535d449ad
[2017-05-04 16:12:38,644: INFO/job_scheduler] Job 33 finished 1 steps successfully
[2017-05-04 16:12:38,811: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 139657908267024
[2017-05-04 16:12:38,816: INFO/job] Job 33: StateChangeJob complete, setting state configured on lotus-32vm8.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-04 16:12:38,816: INFO/job] Job 33: not_deleted=True
[2017-05-04 16:12:38,816: INFO/job_scheduler] Job 33: complete_job: Updating cache
[2017-05-04 16:12:38,817: INFO/job_scheduler] Job 33: locked_item 139657908267024 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-04 16:12:38,819: INFO/job_scheduler] Job 33 completing (errored=False, cancelled=False)
[2017-05-04 16:12:38,860: INFO/job_scheduler] run_next: 0 runnable jobs of (28 pending, 6 tasked)
[2017-05-04 16:12:40,889: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:40,889: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:40,890: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:40,890: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:40,890: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 50b7bc2a-cbd2-48db-9838-f9ad50fa3de0
[2017-05-04 16:12:40,890: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 50b7bc2a-cbd2-48db-9838-f9ad50fa3de0
[2017-05-04 16:12:40,902: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 50b7bc2a-cbd2-48db-9838-f9ad50fa3de0
[2017-05-04 16:12:40,902: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 50b7bc2a-cbd2-48db-9838-f9ad50fa3de0
[2017-05-04 16:12:40,902: INFO/job_scheduler] Job 22 finished 1 steps successfully
[2017-05-04 16:12:40,986: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 139657908296976
[2017-05-04 16:12:40,990: INFO/job] Job 22: StateChangeJob complete, setting state configured on lotus-32vm7.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-04 16:12:40,990: INFO/job] Job 22: not_deleted=True
[2017-05-04 16:12:40,990: INFO/job_scheduler] Job 22: complete_job: Updating cache
[2017-05-04 16:12:40,990: INFO/job_scheduler] Job 22: locked_item 139657908296976 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-04 16:12:40,993: INFO/job_scheduler] Job 22 completing (errored=False, cancelled=False)
[2017-05-04 16:12:41,019: INFO/job_scheduler] run_next: 0 runnable jobs of (28 pending, 5 tasked)
[2017-05-04 16:12:41,915: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:41,915: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:41,915: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:41,915: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:41,916: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8eee844b-ec47-4355-bcc0-fbad435c8870
[2017-05-04 16:12:41,916: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8eee844b-ec47-4355-bcc0-fbad435c8870
[2017-05-04 16:12:41,966: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8eee844b-ec47-4355-bcc0-fbad435c8870
[2017-05-04 16:12:41,966: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8eee844b-ec47-4355-bcc0-fbad435c8870
[2017-05-04 16:12:41,967: INFO/job_scheduler] Job 21 finished 1 steps successfully
[2017-05-04 16:12:42,052: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 139657908294544
[2017-05-04 16:12:42,056: INFO/job] Job 21: StateChangeJob complete, setting state configured on lotus-32vm7.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-04 16:12:42,056: INFO/job] Job 21: not_deleted=True
[2017-05-04 16:12:42,056: INFO/job_scheduler] Job 21: complete_job: Updating cache
[2017-05-04 16:12:42,056: INFO/job_scheduler] Job 21: locked_item 139657908294544 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-04 16:12:42,059: INFO/job_scheduler] Job 21 completing (errored=False, cancelled=False)
[2017-05-04 16:12:42,074: INFO/job_scheduler] run_next: 0 runnable jobs of (28 pending, 4 tasked)
[2017-05-04 16:12:46,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:46,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:46,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:46,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:46,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc de1ec68f-6ecf-47ec-9cad-e0b92108b042
[2017-05-04 16:12:46,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc de1ec68f-6ecf-47ec-9cad-e0b92108b042
[2017-05-04 16:12:46,895: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc de1ec68f-6ecf-47ec-9cad-e0b92108b042
[2017-05-04 16:12:46,895: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc de1ec68f-6ecf-47ec-9cad-e0b92108b042
[2017-05-04 16:12:47,203: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com kernel_status {}
[2017-05-04 16:12:47,220: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 800ba507-3771-4394-9daa-ecf9e20dc044
[2017-05-04 16:12:47,220: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 800ba507-3771-4394-9daa-ecf9e20dc044
[2017-05-04 16:12:47,783: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:47,783: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:47,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:47,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:47,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 48d1b942-a553-4849-af53-6009bded43a9
[2017-05-04 16:12:47,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 48d1b942-a553-4849-af53-6009bded43a9
[2017-05-04 16:12:47,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 48d1b942-a553-4849-af53-6009bded43a9
[2017-05-04 16:12:47,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 48d1b942-a553-4849-af53-6009bded43a9
[2017-05-04 16:12:47,799: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.226', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-05-04 16:12:47,818: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fcbd59fc-7cfa-47ad-a6b1-7e76941be293
[2017-05-04 16:12:47,818: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fcbd59fc-7cfa-47ad-a6b1-7e76941be293
[2017-05-04 16:12:47,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:47,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:47,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:47,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:12:47,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fcbd59fc-7cfa-47ad-a6b1-7e76941be293
[2017-05-04 16:12:47,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fcbd59fc-7cfa-47ad-a6b1-7e76941be293
[2017-05-04 16:12:47,944: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fcbd59fc-7cfa-47ad-a6b1-7e76941be293
[2017-05-04 16:12:47,944: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fcbd59fc-7cfa-47ad-a6b1-7e76941be293
[2017-05-04 16:12:47,956: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 52649, 'pcs_password': 'pRfZVreYze8ZLIxOSaFQ'}
[2017-05-04 16:12:47,968: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 68c2d040-76bd-4a59-8765-df9fa70fd4ff
[2017-05-04 16:12:47,968: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 68c2d040-76bd-4a59-8765-df9fa70fd4ff
[2017-05-04 16:12:48,592: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:48,592: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:48,593: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:48,593: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:48,593: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 30c0593c-3272-4a95-bec1-6003dbb5ccbc
[2017-05-04 16:12:48,593: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 30c0593c-3272-4a95-bec1-6003dbb5ccbc
[2017-05-04 16:12:48,632: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 30c0593c-3272-4a95-bec1-6003dbb5ccbc
[2017-05-04 16:12:48,632: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 30c0593c-3272-4a95-bec1-6003dbb5ccbc
[2017-05-04 16:12:48,971: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com kernel_status {}
[2017-05-04 16:12:48,984: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8ba12763-962d-44dc-8699-823079dc012f
[2017-05-04 16:12:48,984: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8ba12763-962d-44dc-8699-823079dc012f
[2017-05-04 16:12:49,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:49,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:49,323: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:49,323: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:49,323: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 800ba507-3771-4394-9daa-ecf9e20dc044
[2017-05-04 16:12:49,323: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 800ba507-3771-4394-9daa-ecf9e20dc044
[2017-05-04 16:12:49,336: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 800ba507-3771-4394-9daa-ecf9e20dc044
[2017-05-04 16:12:49,336: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 800ba507-3771-4394-9daa-ecf9e20dc044
[2017-05-04 16:12:49,336: INFO/job_scheduler] Job 9 finished 4 steps successfully
[2017-05-04 16:12:49,429: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 139657774294224
[2017-05-04 16:12:49,434: INFO/job] Job 9: StateChangeJob complete, setting state packages_installed on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 16:12:49,434: INFO/job] Job 9: not_deleted=True
[2017-05-04 16:12:49,434: INFO/job_scheduler] Job 9: complete_job: Updating cache
[2017-05-04 16:12:49,434: INFO/job_scheduler] Job 9: locked_item 139657908295312 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-04 16:12:49,437: INFO/job_scheduler] Job 9 completing (errored=False, cancelled=False)
[2017-05-04 16:12:49,461: INFO/job_scheduler] run_next: 4 runnable jobs of (28 pending, 3 tasked)
[2017-05-04 16:12:49,479: INFO/job_scheduler] Job 10: RunJobThread.run
[2017-05-04 16:12:49,482: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-05-04 16:12:49,481: INFO/job_scheduler] Job 12: open DB connection during completion
[2017-05-04 16:12:49,481: INFO/job_scheduler] Job 11: RunJobThread.run
[2017-05-04 16:12:49,502: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-05-04 16:12:49,508: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 524898f4-f649-41c7-96cc-24b90d91e915
[2017-05-04 16:12:49,508: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 524898f4-f649-41c7-96cc-24b90d91e915
[2017-05-04 16:12:49,526: INFO/job_scheduler] Job 13: RunJobThread.run
[2017-05-04 16:12:49,526: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-05-04 16:12:49,547: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 139657774326224
[2017-05-04 16:12:49,554: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f85068ee-35fc-4941-98b7-2819a11b494e
[2017-05-04 16:12:49,554: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f85068ee-35fc-4941-98b7-2819a11b494e
[2017-05-04 16:12:49,574: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-04 16:12:49,582: INFO/job] Job 12: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-04 16:12:49,583: INFO/job] Job 12: not_deleted=True
[2017-05-04 16:12:49,583: INFO/job_scheduler] Job 12: complete_job: Updating cache
[2017-05-04 16:12:49,583: INFO/job_scheduler] Job 12: locked_item 139657774326224 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-04 16:12:49,595: INFO/job_scheduler] Job 12 completing (errored=False, cancelled=False)
[2017-05-04 16:12:49,611: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6be24868-5b54-4a1e-9992-f61f8505b697
[2017-05-04 16:12:49,611: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6be24868-5b54-4a1e-9992-f61f8505b697
[2017-05-04 16:12:49,611: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:49,611: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:49,611: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:49,611: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:49,614: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0e39836b-8aae-4b6a-bbb7-f28e1f60c7fd
[2017-05-04 16:12:49,614: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0e39836b-8aae-4b6a-bbb7-f28e1f60c7fd
[2017-05-04 16:12:49,624: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0e39836b-8aae-4b6a-bbb7-f28e1f60c7fd
[2017-05-04 16:12:49,624: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0e39836b-8aae-4b6a-bbb7-f28e1f60c7fd
[2017-05-04 16:12:49,625: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.227', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-05-04 16:12:49,649: INFO/job_scheduler] run_next: 1 runnable jobs of (24 pending, 6 tasked)
[2017-05-04 16:12:49,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 855bbb00-f232-4ffb-9b0b-190b61540a08
[2017-05-04 16:12:49,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 855bbb00-f232-4ffb-9b0b-190b61540a08
[2017-05-04 16:12:49,666: INFO/job_scheduler] Job 14: RunJobThread.run
[2017-05-04 16:12:49,667: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com load_lnet {}
[2017-05-04 16:12:49,704: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6c5968eb-7947-46ac-9520-ec92adfb88ac
[2017-05-04 16:12:49,704: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6c5968eb-7947-46ac-9520-ec92adfb88ac
[2017-05-04 16:12:49,742: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:49,742: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:49,742: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:49,742: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:12:49,743: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 855bbb00-f232-4ffb-9b0b-190b61540a08
[2017-05-04 16:12:49,743: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 855bbb00-f232-4ffb-9b0b-190b61540a08
[2017-05-04 16:12:49,743: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 855bbb00-f232-4ffb-9b0b-190b61540a08
[2017-05-04 16:12:49,743: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 855bbb00-f232-4ffb-9b0b-190b61540a08
[2017-05-04 16:12:49,771: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 53219, 'pcs_password': '7PA5uCkdfIMBtfE6Nn60'}
[2017-05-04 16:12:49,799: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0b49a82a-045e-4907-a87c-92f234469536
[2017-05-04 16:12:49,799: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0b49a82a-045e-4907-a87c-92f234469536
[2017-05-04 16:12:49,800: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:49,800: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:49,800: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:49,800: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:49,800: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6c5968eb-7947-46ac-9520-ec92adfb88ac
[2017-05-04 16:12:49,800: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6c5968eb-7947-46ac-9520-ec92adfb88ac
[2017-05-04 16:12:49,829: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6c5968eb-7947-46ac-9520-ec92adfb88ac
[2017-05-04 16:12:49,829: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6c5968eb-7947-46ac-9520-ec92adfb88ac
[2017-05-04 16:12:49,830: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-04 16:12:49,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e7938949-f1e4-43cc-9406-6f7802b895b0
[2017-05-04 16:12:49,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e7938949-f1e4-43cc-9406-6f7802b895b0
[2017-05-04 16:12:49,960: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:49,960: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:49,961: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:49,961: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:49,961: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e7938949-f1e4-43cc-9406-6f7802b895b0
[2017-05-04 16:12:49,961: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e7938949-f1e4-43cc-9406-6f7802b895b0
[2017-05-04 16:12:49,974: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e7938949-f1e4-43cc-9406-6f7802b895b0
[2017-05-04 16:12:49,974: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e7938949-f1e4-43cc-9406-6f7802b895b0
[2017-05-04 16:12:50,626: INFO/job_scheduler] Job 14 finished 2 steps successfully
[2017-05-04 16:12:50,671: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 139657774326224
[2017-05-04 16:12:50,677: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-04 16:12:50,691: INFO/job] Job 14: StateChangeJob complete, setting state lnet_down on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-04 16:12:50,692: INFO/job] Job 14: not_deleted=True
[2017-05-04 16:12:50,692: INFO/job_scheduler] Job 14: complete_job: Updating cache
[2017-05-04 16:12:50,693: INFO/job_scheduler] Job 14: locked_item 139657774326224 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-04 16:12:50,700: INFO/job_scheduler] Job 14 completing (errored=False, cancelled=False)
[2017-05-04 16:12:50,736: INFO/job_scheduler] run_next: 1 runnable jobs of (23 pending, 6 tasked)
[2017-05-04 16:12:50,748: INFO/job_scheduler] Job 17: RunJobThread.run
[2017-05-04 16:12:50,748: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_lnet {}
[2017-05-04 16:12:50,794: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 213583a2-8027-47ea-b115-4a64784851fd
[2017-05-04 16:12:50,794: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 213583a2-8027-47ea-b115-4a64784851fd
[2017-05-04 16:12:50,917: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:50,917: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:50,918: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:50,918: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:50,918: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 213583a2-8027-47ea-b115-4a64784851fd
[2017-05-04 16:12:50,918: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 213583a2-8027-47ea-b115-4a64784851fd
[2017-05-04 16:12:50,918: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 213583a2-8027-47ea-b115-4a64784851fd
[2017-05-04 16:12:50,918: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 213583a2-8027-47ea-b115-4a64784851fd
[2017-05-04 16:12:50,919: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-04 16:12:50,949: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 52ab43d8-33b0-4115-a862-3a0bbdc158a9
[2017-05-04 16:12:50,949: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 52ab43d8-33b0-4115-a862-3a0bbdc158a9
[2017-05-04 16:12:51,094: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:51,094: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:51,095: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:51,095: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:51,095: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8ba12763-962d-44dc-8699-823079dc012f
[2017-05-04 16:12:51,095: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8ba12763-962d-44dc-8699-823079dc012f
[2017-05-04 16:12:51,103: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8ba12763-962d-44dc-8699-823079dc012f
[2017-05-04 16:12:51,103: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8ba12763-962d-44dc-8699-823079dc012f
[2017-05-04 16:12:51,104: INFO/job_scheduler] Job 42 finished 4 steps successfully
[2017-05-04 16:12:51,272: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 139657774429648
[2017-05-04 16:12:51,277: INFO/job] Job 42: StateChangeJob complete, setting state packages_installed on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 16:12:51,277: INFO/job] Job 42: not_deleted=True
[2017-05-04 16:12:51,277: INFO/job_scheduler] Job 42: complete_job: Updating cache
[2017-05-04 16:12:51,277: INFO/job_scheduler] Job 42: locked_item 139657774441552 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-04 16:12:51,281: INFO/job_scheduler] Job 42 completing (errored=False, cancelled=False)
[2017-05-04 16:12:51,303: INFO/job_scheduler] run_next: 4 runnable jobs of (22 pending, 6 tasked)
[2017-05-04 16:12:51,332: INFO/job_scheduler] Job 43: RunJobThread.run
[2017-05-04 16:12:51,333: INFO/job_scheduler] Job 44: open DB connection during completion
[2017-05-04 16:12:51,335: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-05-04 16:12:51,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a8e2ee5a-d40f-4e0d-aa78-156137d3ff5d
[2017-05-04 16:12:51,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a8e2ee5a-d40f-4e0d-aa78-156137d3ff5d
[2017-05-04 16:12:51,351: INFO/job_scheduler] Job 45: RunJobThread.run
[2017-05-04 16:12:51,364: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-05-04 16:12:51,353: INFO/job_scheduler] Job 46: RunJobThread.run
[2017-05-04 16:12:51,365: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-05-04 16:12:51,398: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 139657774444048
[2017-05-04 16:12:51,401: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a3f36124-ddef-490a-bd24-e80366ce384f
[2017-05-04 16:12:51,401: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a3f36124-ddef-490a-bd24-e80366ce384f
[2017-05-04 16:12:51,420: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 28cfc0ca-cc35-410c-b763-5293e24a8412
[2017-05-04 16:12:51,420: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 28cfc0ca-cc35-410c-b763-5293e24a8412
[2017-05-04 16:12:51,427: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-04 16:12:51,439: INFO/job] Job 44: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-04 16:12:51,439: INFO/job] Job 44: not_deleted=True
[2017-05-04 16:12:51,439: INFO/job_scheduler] Job 44: complete_job: Updating cache
[2017-05-04 16:12:51,439: INFO/job_scheduler] Job 44: locked_item 139657774444048 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-04 16:12:51,455: INFO/job_scheduler] Job 44 completing (errored=False, cancelled=False)
[2017-05-04 16:12:51,478: INFO/job_scheduler] run_next: 1 runnable jobs of (18 pending, 9 tasked)
[2017-05-04 16:12:51,510: INFO/job_scheduler] Job 48: RunJobThread.run
[2017-05-04 16:12:51,511: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com load_lnet {}
[2017-05-04 16:12:51,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 66aa8b92-757d-44e0-a505-a95fd4e5c86a
[2017-05-04 16:12:51,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 66aa8b92-757d-44e0-a505-a95fd4e5c86a
[2017-05-04 16:12:52,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:52,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:52,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:52,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:52,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 52ab43d8-33b0-4115-a862-3a0bbdc158a9
[2017-05-04 16:12:52,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 52ab43d8-33b0-4115-a862-3a0bbdc158a9
[2017-05-04 16:12:52,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 52ab43d8-33b0-4115-a862-3a0bbdc158a9
[2017-05-04 16:12:52,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 52ab43d8-33b0-4115-a862-3a0bbdc158a9
[2017-05-04 16:12:52,191: INFO/job_scheduler] Job 17 finished 2 steps successfully
[2017-05-04 16:12:52,214: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 139657774326224
[2017-05-04 16:12:52,248: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 30
[2017-05-04 16:12:52,267: INFO/job] Job 17: StateChangeJob complete, setting state lnet_up on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-04 16:12:52,268: INFO/job] Job 17: not_deleted=True
[2017-05-04 16:12:52,268: INFO/job_scheduler] Job 17: complete_job: Updating cache
[2017-05-04 16:12:52,268: INFO/job_scheduler] Job 17: locked_item 139657774326224 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-04 16:12:52,270: INFO/job_scheduler] Job 17 completing (errored=False, cancelled=False)
[2017-05-04 16:12:52,299: INFO/job_scheduler] run_next: 0 runnable jobs of (17 pending, 9 tasked)
[2017-05-04 16:12:52,630: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:52,630: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:52,630: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:52,630: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:52,630: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 66aa8b92-757d-44e0-a505-a95fd4e5c86a
[2017-05-04 16:12:52,630: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 66aa8b92-757d-44e0-a505-a95fd4e5c86a
[2017-05-04 16:12:52,664: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 66aa8b92-757d-44e0-a505-a95fd4e5c86a
[2017-05-04 16:12:52,664: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 66aa8b92-757d-44e0-a505-a95fd4e5c86a
[2017-05-04 16:12:52,665: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-04 16:12:52,693: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 22c18097-79f5-44a9-b171-fb93abd2a223
[2017-05-04 16:12:52,693: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 22c18097-79f5-44a9-b171-fb93abd2a223
[2017-05-04 16:12:52,795: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:52,795: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:52,795: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:52,795: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:52,796: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 22c18097-79f5-44a9-b171-fb93abd2a223
[2017-05-04 16:12:52,796: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 22c18097-79f5-44a9-b171-fb93abd2a223
[2017-05-04 16:12:52,819: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 22c18097-79f5-44a9-b171-fb93abd2a223
[2017-05-04 16:12:52,819: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 22c18097-79f5-44a9-b171-fb93abd2a223
[2017-05-04 16:12:53,449: INFO/job_scheduler] Job 48 finished 2 steps successfully
[2017-05-04 16:12:53,506: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 139657774444048
[2017-05-04 16:12:53,514: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-04 16:12:53,523: INFO/job] Job 48: StateChangeJob complete, setting state lnet_down on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-04 16:12:53,524: INFO/job] Job 48: not_deleted=True
[2017-05-04 16:12:53,524: INFO/job_scheduler] Job 48: complete_job: Updating cache
[2017-05-04 16:12:53,524: INFO/job_scheduler] Job 48: locked_item 139657774444048 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-04 16:12:53,533: INFO/job_scheduler] Job 48 completing (errored=False, cancelled=False)
[2017-05-04 16:12:53,567: INFO/job_scheduler] run_next: 1 runnable jobs of (17 pending, 8 tasked)
[2017-05-04 16:12:53,589: INFO/job_scheduler] Job 49: RunJobThread.run
[2017-05-04 16:12:53,590: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_lnet {}
[2017-05-04 16:12:53,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dfc06464-e2bc-48cf-ac5a-18bbb73e2c50
[2017-05-04 16:12:53,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dfc06464-e2bc-48cf-ac5a-18bbb73e2c50
[2017-05-04 16:12:55,677: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:55,677: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:55,677: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:55,677: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:55,677: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f85068ee-35fc-4941-98b7-2819a11b494e
[2017-05-04 16:12:55,677: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f85068ee-35fc-4941-98b7-2819a11b494e
[2017-05-04 16:12:55,695: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:55,695: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:55,695: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:55,695: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:55,695: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dfc06464-e2bc-48cf-ac5a-18bbb73e2c50
[2017-05-04 16:12:55,695: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dfc06464-e2bc-48cf-ac5a-18bbb73e2c50
[2017-05-04 16:12:55,724: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f85068ee-35fc-4941-98b7-2819a11b494e
[2017-05-04 16:12:55,724: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f85068ee-35fc-4941-98b7-2819a11b494e
[2017-05-04 16:12:55,724: INFO/job_scheduler] Job 11 finished 1 steps successfully
[2017-05-04 16:12:55,728: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dfc06464-e2bc-48cf-ac5a-18bbb73e2c50
[2017-05-04 16:12:55,728: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dfc06464-e2bc-48cf-ac5a-18bbb73e2c50
[2017-05-04 16:12:55,729: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-04 16:12:55,748: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4a44b840-6c4a-4de5-a081-96b622f792de
[2017-05-04 16:12:55,748: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4a44b840-6c4a-4de5-a081-96b622f792de
[2017-05-04 16:12:55,787: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 139657774327824
[2017-05-04 16:12:55,793: INFO/job] Job 11: StateChangeJob complete, setting state configured on lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-04 16:12:55,793: INFO/job] Job 11: not_deleted=True
[2017-05-04 16:12:55,793: INFO/job_scheduler] Job 11: complete_job: Updating cache
[2017-05-04 16:12:55,793: INFO/job_scheduler] Job 11: locked_item 139657774327824 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-04 16:12:55,796: INFO/job_scheduler] Job 11 completing (errored=False, cancelled=False)
[2017-05-04 16:12:55,844: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 8 tasked)
[2017-05-04 16:12:56,698: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:56,698: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:56,698: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:56,698: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:12:56,699: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6be24868-5b54-4a1e-9992-f61f8505b697
[2017-05-04 16:12:56,699: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6be24868-5b54-4a1e-9992-f61f8505b697
[2017-05-04 16:12:56,742: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6be24868-5b54-4a1e-9992-f61f8505b697
[2017-05-04 16:12:56,742: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6be24868-5b54-4a1e-9992-f61f8505b697
[2017-05-04 16:12:56,742: INFO/job_scheduler] Job 13 finished 1 steps successfully
[2017-05-04 16:12:56,796: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 139657774326352
[2017-05-04 16:12:56,799: INFO/job] Job 13: StateChangeJob complete, setting state configured on lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-04 16:12:56,799: INFO/job] Job 13: not_deleted=True
[2017-05-04 16:12:56,799: INFO/job_scheduler] Job 13: complete_job: Updating cache
[2017-05-04 16:12:56,799: INFO/job_scheduler] Job 13: locked_item 139657774326352 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-04 16:12:56,802: INFO/job_scheduler] Job 13 completing (errored=False, cancelled=False)
[2017-05-04 16:12:56,819: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 7 tasked)
[2017-05-04 16:12:56,870: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:56,870: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:56,870: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:56,870: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:56,871: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4a44b840-6c4a-4de5-a081-96b622f792de
[2017-05-04 16:12:56,871: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4a44b840-6c4a-4de5-a081-96b622f792de
[2017-05-04 16:12:56,917: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4a44b840-6c4a-4de5-a081-96b622f792de
[2017-05-04 16:12:56,917: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4a44b840-6c4a-4de5-a081-96b622f792de
[2017-05-04 16:12:57,009: INFO/job_scheduler] Job 49 finished 2 steps successfully
[2017-05-04 16:12:57,029: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 139657774444048
[2017-05-04 16:12:57,065: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm6.lotus.hpdd.lab.intel.com at severity 30
[2017-05-04 16:12:57,072: INFO/job] Job 49: StateChangeJob complete, setting state lnet_up on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-04 16:12:57,072: INFO/job] Job 49: not_deleted=True
[2017-05-04 16:12:57,072: INFO/job_scheduler] Job 49: complete_job: Updating cache
[2017-05-04 16:12:57,073: INFO/job_scheduler] Job 49: locked_item 139657774444048 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-04 16:12:57,075: INFO/job_scheduler] Job 49 completing (errored=False, cancelled=False)
[2017-05-04 16:12:57,095: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 6 tasked)
[2017-05-04 16:12:57,459: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:57,459: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:57,459: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:57,459: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:57,460: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a8e2ee5a-d40f-4e0d-aa78-156137d3ff5d
[2017-05-04 16:12:57,460: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a8e2ee5a-d40f-4e0d-aa78-156137d3ff5d
[2017-05-04 16:12:57,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a8e2ee5a-d40f-4e0d-aa78-156137d3ff5d
[2017-05-04 16:12:57,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a8e2ee5a-d40f-4e0d-aa78-156137d3ff5d
[2017-05-04 16:12:57,480: INFO/job_scheduler] Job 43 finished 1 steps successfully
[2017-05-04 16:12:57,539: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 139657774315856
[2017-05-04 16:12:57,542: INFO/job] Job 43: StateChangeJob complete, setting state configured on lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-04 16:12:57,542: INFO/job] Job 43: not_deleted=True
[2017-05-04 16:12:57,542: INFO/job_scheduler] Job 43: complete_job: Updating cache
[2017-05-04 16:12:57,542: INFO/job_scheduler] Job 43: locked_item 139657774315856 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-04 16:12:57,545: INFO/job_scheduler] Job 43 completing (errored=False, cancelled=False)
[2017-05-04 16:12:57,569: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 5 tasked)
[2017-05-04 16:12:58,498: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:58,498: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:58,499: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:58,499: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:12:58,499: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 28cfc0ca-cc35-410c-b763-5293e24a8412
[2017-05-04 16:12:58,499: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 28cfc0ca-cc35-410c-b763-5293e24a8412
[2017-05-04 16:12:58,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 28cfc0ca-cc35-410c-b763-5293e24a8412
[2017-05-04 16:12:58,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 28cfc0ca-cc35-410c-b763-5293e24a8412
[2017-05-04 16:12:58,539: INFO/job_scheduler] Job 46 finished 1 steps successfully
[2017-05-04 16:12:58,604: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 139657774330832
[2017-05-04 16:12:58,608: INFO/job] Job 46: StateChangeJob complete, setting state configured on lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-04 16:12:58,608: INFO/job] Job 46: not_deleted=True
[2017-05-04 16:12:58,608: INFO/job_scheduler] Job 46: complete_job: Updating cache
[2017-05-04 16:12:58,608: INFO/job_scheduler] Job 46: locked_item 139657774330832 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-04 16:12:58,610: INFO/job_scheduler] Job 46 completing (errored=False, cancelled=False)
[2017-05-04 16:12:58,627: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 4 tasked)
[2017-05-04 16:13:00,131: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:13:00,131: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:13:00,132: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:13:00,132: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:13:00,132: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 68c2d040-76bd-4a59-8765-df9fa70fd4ff
[2017-05-04 16:13:00,132: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 68c2d040-76bd-4a59-8765-df9fa70fd4ff
[2017-05-04 16:13:00,140: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 68c2d040-76bd-4a59-8765-df9fa70fd4ff
[2017-05-04 16:13:00,140: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 68c2d040-76bd-4a59-8765-df9fa70fd4ff
[2017-05-04 16:13:00,141: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': True, 'mcast_port': 52649, 'new_node_fqdn': u'lotus-32vm7.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'pRfZVreYze8ZLIxOSaFQ'}
[2017-05-04 16:13:00,157: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9b37e907-9a9a-4e9f-bbb2-cc76e067ea07
[2017-05-04 16:13:00,157: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9b37e907-9a9a-4e9f-bbb2-cc76e067ea07
[2017-05-04 16:13:01,985: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:13:01,985: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:13:01,985: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:13:01,985: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:13:01,986: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0b49a82a-045e-4907-a87c-92f234469536
[2017-05-04 16:13:01,986: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0b49a82a-045e-4907-a87c-92f234469536
[2017-05-04 16:13:02,028: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0b49a82a-045e-4907-a87c-92f234469536
[2017-05-04 16:13:02,028: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0b49a82a-045e-4907-a87c-92f234469536
[2017-05-04 16:13:02,029: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': True, 'mcast_port': 53219, 'new_node_fqdn': u'lotus-32vm8.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': '7PA5uCkdfIMBtfE6Nn60'}
[2017-05-04 16:13:02,042: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d268772b-51a0-446a-bc32-6514ddd20c9a
[2017-05-04 16:13:02,042: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d268772b-51a0-446a-bc32-6514ddd20c9a
[2017-05-04 16:13:04,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:13:04,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:13:04,720: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:13:04,720: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:13:04,720: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a3f36124-ddef-490a-bd24-e80366ce384f
[2017-05-04 16:13:04,720: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a3f36124-ddef-490a-bd24-e80366ce384f
[2017-05-04 16:13:04,726: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a3f36124-ddef-490a-bd24-e80366ce384f
[2017-05-04 16:13:04,726: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a3f36124-ddef-490a-bd24-e80366ce384f
[2017-05-04 16:13:04,727: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.225', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-05-04 16:13:04,741: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b4eee6a6-ed97-45b5-8dca-283b261518b1
[2017-05-04 16:13:04,741: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b4eee6a6-ed97-45b5-8dca-283b261518b1
[2017-05-04 16:13:04,842: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:13:04,842: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:13:04,842: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:13:04,842: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:13:04,843: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b4eee6a6-ed97-45b5-8dca-283b261518b1
[2017-05-04 16:13:04,843: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b4eee6a6-ed97-45b5-8dca-283b261518b1
[2017-05-04 16:13:04,857: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b4eee6a6-ed97-45b5-8dca-283b261518b1
[2017-05-04 16:13:04,857: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b4eee6a6-ed97-45b5-8dca-283b261518b1
[2017-05-04 16:13:04,873: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 50049, 'pcs_password': 'W1E4zRxfWbSlXb0ByTzK'}
[2017-05-04 16:13:04,887: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0626f3c2-a640-40c9-a3d6-173787854ade
[2017-05-04 16:13:04,887: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0626f3c2-a640-40c9-a3d6-173787854ade
[2017-05-04 16:13:17,030: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:13:17,030: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:13:17,030: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:13:17,030: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:13:17,030: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0626f3c2-a640-40c9-a3d6-173787854ade
[2017-05-04 16:13:17,030: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0626f3c2-a640-40c9-a3d6-173787854ade
[2017-05-04 16:13:17,056: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0626f3c2-a640-40c9-a3d6-173787854ade
[2017-05-04 16:13:17,056: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0626f3c2-a640-40c9-a3d6-173787854ade
[2017-05-04 16:13:17,057: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': True, 'mcast_port': 50049, 'new_node_fqdn': u'lotus-32vm6.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'W1E4zRxfWbSlXb0ByTzK'}
[2017-05-04 16:13:17,070: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 29ec85bb-b064-4682-82ef-d458ad405869
[2017-05-04 16:13:17,070: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 29ec85bb-b064-4682-82ef-d458ad405869
[2017-05-04 16:13:19,812: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:13:19,812: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:13:19,813: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:13:19,813: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:13:19,813: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 524898f4-f649-41c7-96cc-24b90d91e915
[2017-05-04 16:13:19,813: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 524898f4-f649-41c7-96cc-24b90d91e915
[2017-05-04 16:13:19,840: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 524898f4-f649-41c7-96cc-24b90d91e915
[2017-05-04 16:13:19,840: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 524898f4-f649-41c7-96cc-24b90d91e915
[2017-05-04 16:13:19,841: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.224', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-05-04 16:13:19,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1a1162ea-ea77-4007-83a1-51b5bb0d4680
[2017-05-04 16:13:19,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1a1162ea-ea77-4007-83a1-51b5bb0d4680
[2017-05-04 16:13:19,916: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:13:19,916: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:13:19,917: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:13:19,917: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:13:19,917: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1a1162ea-ea77-4007-83a1-51b5bb0d4680
[2017-05-04 16:13:19,917: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1a1162ea-ea77-4007-83a1-51b5bb0d4680
[2017-05-04 16:13:19,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1a1162ea-ea77-4007-83a1-51b5bb0d4680
[2017-05-04 16:13:19,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1a1162ea-ea77-4007-83a1-51b5bb0d4680
[2017-05-04 16:13:19,936: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 47491, 'pcs_password': 'FGzSdZBpkRGsIVZ4RtNG'}
[2017-05-04 16:13:19,949: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b189bc6d-5620-4838-961e-e9032e2d1268
[2017-05-04 16:13:19,949: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b189bc6d-5620-4838-961e-e9032e2d1268
[2017-05-04 16:13:24,187: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:13:24,187: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:13:24,187: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:13:24,187: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:13:24,188: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 29ec85bb-b064-4682-82ef-d458ad405869
[2017-05-04 16:13:24,188: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 29ec85bb-b064-4682-82ef-d458ad405869
[2017-05-04 16:13:24,188: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 29ec85bb-b064-4682-82ef-d458ad405869
[2017-05-04 16:13:24,188: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 29ec85bb-b064-4682-82ef-d458ad405869
[2017-05-04 16:13:24,188: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-04 16:13:24.188907+00:00:
[2017-05-04 16:13:24,188: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-04 16:13:24.188907+00:00:
[2017-05-04 16:13:24,189: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->50049
[2017-05-04 16:13:24,189: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->50049
[2017-05-04 16:13:24,192: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-04 16:13:24,192: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-04 16:13:24,221: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 16:13:24,221: INFO/job_scheduler]   Job 45 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-05-04 16:13:24,221: INFO/job_scheduler]   Job 47 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-05-04 16:13:24,221: INFO/job_scheduler]   Job 50 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:13:24,222: INFO/job_scheduler]   Job 51 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:13:24,222: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 4 tasked)
[2017-05-04 16:13:24,224: INFO/job_scheduler] Job 45 finished 1 steps successfully
[2017-05-04 16:13:24,361: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 139657774245712
[2017-05-04 16:13:24,370: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-04 16:13:24,383: INFO/job] Job 45: StateChangeJob complete, setting state stopped on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:13:24,383: INFO/job] Job 45: not_deleted=True
[2017-05-04 16:13:24,383: INFO/job_scheduler] Job 45: complete_job: Updating cache
[2017-05-04 16:13:24,383: INFO/job_scheduler] Job 45: locked_item 139657774245712 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-04 16:13:24,387: INFO/job_scheduler] Job 45 completing (errored=False, cancelled=False)
[2017-05-04 16:13:24,416: INFO/job_scheduler] run_next: 1 runnable jobs of (16 pending, 3 tasked)
[2017-05-04 16:13:24,420: INFO/job_scheduler] Job 47: RunJobThread.run
[2017-05-04 16:13:24,420: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-05-04 16:13:24,432: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 35d8bdba-1409-409d-b95c-7945d728f424
[2017-05-04 16:13:24,432: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 35d8bdba-1409-409d-b95c-7945d728f424
[2017-05-04 16:13:32,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:13:32,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:13:32,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:13:32,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:13:32,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b189bc6d-5620-4838-961e-e9032e2d1268
[2017-05-04 16:13:32,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b189bc6d-5620-4838-961e-e9032e2d1268
[2017-05-04 16:13:32,117: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b189bc6d-5620-4838-961e-e9032e2d1268
[2017-05-04 16:13:32,117: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b189bc6d-5620-4838-961e-e9032e2d1268
[2017-05-04 16:13:32,118: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': True, 'mcast_port': 47491, 'new_node_fqdn': u'lotus-32vm5.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'FGzSdZBpkRGsIVZ4RtNG'}
[2017-05-04 16:13:32,137: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 51fcc824-4913-4b2a-8694-05119c5648b6
[2017-05-04 16:13:32,137: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 51fcc824-4913-4b2a-8694-05119c5648b6
[2017-05-04 16:13:32,540: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:13:32,540: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:13:32,540: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:13:32,540: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:13:32,540: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 35d8bdba-1409-409d-b95c-7945d728f424
[2017-05-04 16:13:32,540: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 35d8bdba-1409-409d-b95c-7945d728f424
[2017-05-04 16:13:32,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 35d8bdba-1409-409d-b95c-7945d728f424
[2017-05-04 16:13:32,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 35d8bdba-1409-409d-b95c-7945d728f424
[2017-05-04 16:13:32,549: INFO/job_scheduler] Job 47 finished 1 steps successfully
[2017-05-04 16:13:32,601: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 139657774245712
[2017-05-04 16:13:32,646: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm6.lotus.hpdd.lab.intel.com at severity 30
[2017-05-04 16:13:32,661: INFO/job] Job 47: StateChangeJob complete, setting state started on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:13:32,661: INFO/job] Job 47: not_deleted=True
[2017-05-04 16:13:32,661: INFO/job_scheduler] Job 47: complete_job: Updating cache
[2017-05-04 16:13:32,662: INFO/job_scheduler] Job 47: locked_item 139657774245712 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-04 16:13:32,667: INFO/job_scheduler] Job 47 completing (errored=False, cancelled=False)
[2017-05-04 16:13:32,711: INFO/job_scheduler] run_next: 1 runnable jobs of (15 pending, 3 tasked)
[2017-05-04 16:13:32,765: INFO/job_scheduler] Job 50: RunJobThread.run
[2017-05-04 16:13:32,766: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-04 16:13:32,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7f86af1e-c699-455d-ab06-2d85b02f7c9c
[2017-05-04 16:13:32,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7f86af1e-c699-455d-ab06-2d85b02f7c9c
[2017-05-04 16:13:34,775: INFO/job_scheduler] run_next: 0 runnable jobs of (14 pending, 4 tasked)
[2017-05-04 16:13:34,794: INFO/job_scheduler] run_next: 0 runnable jobs of (14 pending, 4 tasked)
[2017-05-04 16:13:38,200: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:13:38,200: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:13:38,200: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:13:38,200: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:13:38,201: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 51fcc824-4913-4b2a-8694-05119c5648b6
[2017-05-04 16:13:38,201: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 51fcc824-4913-4b2a-8694-05119c5648b6
[2017-05-04 16:13:38,202: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 51fcc824-4913-4b2a-8694-05119c5648b6
[2017-05-04 16:13:38,202: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 51fcc824-4913-4b2a-8694-05119c5648b6
[2017-05-04 16:13:38,202: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-04 16:13:38.202575+00:00:
[2017-05-04 16:13:38,202: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-04 16:13:38.202575+00:00:
[2017-05-04 16:13:38,202: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->47491
[2017-05-04 16:13:38,202: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->47491
[2017-05-04 16:13:38,204: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-04 16:13:38,204: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-04 16:13:38,216: INFO/job_scheduler] Job 10 finished 1 steps successfully
[2017-05-04 16:13:38,217: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 16:13:38,218: INFO/job_scheduler]   Job 10 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-05-04 16:13:38,218: INFO/job_scheduler]   Job 15 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-05-04 16:13:38,218: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:13:38,218: INFO/job_scheduler]   Job 18 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:13:38,219: INFO/job_scheduler] run_next: 0 runnable jobs of (14 pending, 4 tasked)
[2017-05-04 16:13:38,275: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 139657908303440
[2017-05-04 16:13:38,281: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-04 16:13:38,298: INFO/job] Job 10: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:13:38,298: INFO/job] Job 10: not_deleted=True
[2017-05-04 16:13:38,298: INFO/job_scheduler] Job 10: complete_job: Updating cache
[2017-05-04 16:13:38,298: INFO/job_scheduler] Job 10: locked_item 139657908303440 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-04 16:13:38,302: INFO/job_scheduler] Job 10 completing (errored=False, cancelled=False)
[2017-05-04 16:13:38,329: INFO/job_scheduler] run_next: 1 runnable jobs of (14 pending, 3 tasked)
[2017-05-04 16:13:38,333: INFO/job_scheduler] Job 15: RunJobThread.run
[2017-05-04 16:13:38,334: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-05-04 16:13:38,362: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7a13ab44-b053-4652-b0ed-4480505aef4b
[2017-05-04 16:13:38,362: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7a13ab44-b053-4652-b0ed-4480505aef4b
[2017-05-04 16:13:39,879: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:13:39,879: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:13:39,879: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:13:39,879: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:13:39,880: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7f86af1e-c699-455d-ab06-2d85b02f7c9c
[2017-05-04 16:13:39,880: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7f86af1e-c699-455d-ab06-2d85b02f7c9c
[2017-05-04 16:13:39,904: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7f86af1e-c699-455d-ab06-2d85b02f7c9c
[2017-05-04 16:13:39,904: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7f86af1e-c699-455d-ab06-2d85b02f7c9c
[2017-05-04 16:13:39,905: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-05-04 16:13:39,918: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2835444e-ae8f-44a0-8aa6-cbb9002d356e
[2017-05-04 16:13:39,918: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2835444e-ae8f-44a0-8aa6-cbb9002d356e
[2017-05-04 16:13:44,217: INFO/job_scheduler] run_next: 0 runnable jobs of (13 pending, 4 tasked)
[2017-05-04 16:13:44,232: INFO/job_scheduler] run_next: 0 runnable jobs of (13 pending, 4 tasked)
[2017-05-04 16:13:44,801: INFO/job_scheduler] run_next: 0 runnable jobs of (13 pending, 4 tasked)
[2017-05-04 16:13:44,815: INFO/job_scheduler] run_next: 0 runnable jobs of (13 pending, 4 tasked)
[2017-05-04 16:13:46,462: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:13:46,462: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:13:46,463: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:13:46,463: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:13:46,463: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7a13ab44-b053-4652-b0ed-4480505aef4b
[2017-05-04 16:13:46,463: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7a13ab44-b053-4652-b0ed-4480505aef4b
[2017-05-04 16:13:46,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7a13ab44-b053-4652-b0ed-4480505aef4b
[2017-05-04 16:13:46,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7a13ab44-b053-4652-b0ed-4480505aef4b
[2017-05-04 16:13:46,479: INFO/job_scheduler] Job 15 finished 1 steps successfully
[2017-05-04 16:13:46,523: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 139657908303440
[2017-05-04 16:13:46,548: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 30
[2017-05-04 16:13:46,568: INFO/job] Job 15: StateChangeJob complete, setting state started on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:13:46,570: INFO/job] Job 15: not_deleted=True
[2017-05-04 16:13:46,570: INFO/job_scheduler] Job 15: complete_job: Updating cache
[2017-05-04 16:13:46,570: INFO/job_scheduler] Job 15: locked_item 139657908303440 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-04 16:13:46,581: INFO/job_scheduler] Job 15 completing (errored=False, cancelled=False)
[2017-05-04 16:13:46,614: INFO/job_scheduler] run_next: 1 runnable jobs of (13 pending, 3 tasked)
[2017-05-04 16:13:46,654: INFO/job_scheduler] Job 16: RunJobThread.run
[2017-05-04 16:13:46,655: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-04 16:13:46,687: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d6fbfe32-89f3-47b0-bf88-56e82b0518bd
[2017-05-04 16:13:46,687: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d6fbfe32-89f3-47b0-bf88-56e82b0518bd
[2017-05-04 16:13:53,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:13:53,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:13:53,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:13:53,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:13:53,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d6fbfe32-89f3-47b0-bf88-56e82b0518bd
[2017-05-04 16:13:53,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d6fbfe32-89f3-47b0-bf88-56e82b0518bd
[2017-05-04 16:13:53,808: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d6fbfe32-89f3-47b0-bf88-56e82b0518bd
[2017-05-04 16:13:53,808: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d6fbfe32-89f3-47b0-bf88-56e82b0518bd
[2017-05-04 16:13:53,809: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-05-04 16:13:53,822: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 34964967-7b1e-44a7-b013-56ec46360ffe
[2017-05-04 16:13:53,822: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 34964967-7b1e-44a7-b013-56ec46360ffe
[2017-05-04 16:13:54,316: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-04 16:13:54,335: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-04 16:14:04,292: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-04 16:14:04,306: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-04 16:14:04,880: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-04 16:14:04,895: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-04 16:14:24,345: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-04 16:14:24,361: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-04 16:14:34,976: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 16:14:34,976: INFO/job_scheduler]   Job 50 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:14:34,976: INFO/job_scheduler]   Job 51 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:14:34,977: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-04 16:14:44,507: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 16:14:44,508: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:14:44,508: INFO/job_scheduler]   Job 18 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:14:44,508: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-04 16:14:44,975: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 16:14:44,975: INFO/job_scheduler]   Job 50 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:14:44,975: INFO/job_scheduler]   Job 51 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:14:44,975: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-04 16:14:54,462: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 16:14:54,462: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:14:54,462: INFO/job_scheduler]   Job 18 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:14:54,463: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-04 16:14:54,997: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 16:14:54,997: INFO/job_scheduler]   Job 50 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:14:54,997: INFO/job_scheduler]   Job 51 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:14:54,998: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-04 16:15:01,364: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:15:01,364: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:15:01,365: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:15:01,365: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:15:01,365: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2835444e-ae8f-44a0-8aa6-cbb9002d356e
[2017-05-04 16:15:01,365: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2835444e-ae8f-44a0-8aa6-cbb9002d356e
[2017-05-04 16:15:01,415: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2835444e-ae8f-44a0-8aa6-cbb9002d356e
[2017-05-04 16:15:01,415: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2835444e-ae8f-44a0-8aa6-cbb9002d356e
[2017-05-04 16:15:01,417: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-04 16:15:01,436: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ec0eb485-cc74-4ec1-999a-44610841d66a
[2017-05-04 16:15:01,436: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ec0eb485-cc74-4ec1-999a-44610841d66a
[2017-05-04 16:15:04,526: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 16:15:04,526: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:15:04,526: INFO/job_scheduler]   Job 18 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:15:04,526: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-04 16:15:04,999: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-04 16:15:05,015: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-04 16:15:08,532: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:15:08,532: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:15:08,533: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:15:08,533: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:15:08,533: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ec0eb485-cc74-4ec1-999a-44610841d66a
[2017-05-04 16:15:08,533: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ec0eb485-cc74-4ec1-999a-44610841d66a
[2017-05-04 16:15:08,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ec0eb485-cc74-4ec1-999a-44610841d66a
[2017-05-04 16:15:08,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ec0eb485-cc74-4ec1-999a-44610841d66a
[2017-05-04 16:15:08,554: INFO/job_scheduler] Job 50 finished 3 steps successfully
[2017-05-04 16:15:08,598: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 139657774442960
[2017-05-04 16:15:08,606: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-04 16:15:08,611: INFO/job] Job 50: StateChangeJob complete, setting state stopped on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-04 16:15:08,611: INFO/job] Job 50: not_deleted=True
[2017-05-04 16:15:08,611: INFO/job_scheduler] Job 50: complete_job: Updating cache
[2017-05-04 16:15:08,611: INFO/job_scheduler] Job 50: locked_item 139657774442960 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-04 16:15:08,618: INFO/job_scheduler] Job 50 completing (errored=False, cancelled=False)
[2017-05-04 16:15:08,733: INFO/job_scheduler] run_next: 1 runnable jobs of (12 pending, 3 tasked)
[2017-05-04 16:15:08,772: INFO/job_scheduler] Job 51: RunJobThread.run
[2017-05-04 16:15:08,773: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-04 16:15:08,797: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d42dc8d0-2c38-4a68-b2ca-7649eb109496
[2017-05-04 16:15:08,797: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d42dc8d0-2c38-4a68-b2ca-7649eb109496
[2017-05-04 16:15:14,529: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 16:15:14,529: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:15:14,529: INFO/job_scheduler]   Job 18 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:15:14,529: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 4 tasked)
[2017-05-04 16:15:15,065: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 4 tasked)
[2017-05-04 16:15:15,078: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 4 tasked)
[2017-05-04 16:15:15,275: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:15:15,275: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:15:15,275: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:15:15,275: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:15:15,276: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 34964967-7b1e-44a7-b013-56ec46360ffe
[2017-05-04 16:15:15,276: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 34964967-7b1e-44a7-b013-56ec46360ffe
[2017-05-04 16:15:15,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 34964967-7b1e-44a7-b013-56ec46360ffe
[2017-05-04 16:15:15,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 34964967-7b1e-44a7-b013-56ec46360ffe
[2017-05-04 16:15:15,325: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-04 16:15:15,338: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c0461797-e1ec-473b-b5be-4409e1f559f5
[2017-05-04 16:15:15,338: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c0461797-e1ec-473b-b5be-4409e1f559f5
[2017-05-04 16:15:15,921: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:15:15,921: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:15:15,921: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:15:15,921: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:15:15,921: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d42dc8d0-2c38-4a68-b2ca-7649eb109496
[2017-05-04 16:15:15,921: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d42dc8d0-2c38-4a68-b2ca-7649eb109496
[2017-05-04 16:15:15,964: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d42dc8d0-2c38-4a68-b2ca-7649eb109496
[2017-05-04 16:15:15,964: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d42dc8d0-2c38-4a68-b2ca-7649eb109496
[2017-05-04 16:15:15,964: INFO/job_scheduler] Job 51 finished 1 steps successfully
[2017-05-04 16:15:16,026: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 139657774442960
[2017-05-04 16:15:16,056: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm6.lotus.hpdd.lab.intel.com at severity 30
[2017-05-04 16:15:16,066: INFO/job] Job 51: StateChangeJob complete, setting state started on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-04 16:15:16,066: INFO/job] Job 51: not_deleted=True
[2017-05-04 16:15:16,066: INFO/job_scheduler] Job 51: complete_job: Updating cache
[2017-05-04 16:15:16,066: INFO/job_scheduler] Job 51: locked_item 139657774442960 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-04 16:15:16,069: INFO/job_scheduler] Job 51 completing (errored=False, cancelled=False)
[2017-05-04 16:15:16,097: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-4
[2017-05-04 16:15:16,097: INFO/job_scheduler] _notify: Updating .mcast_port of item 4 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 50049
[2017-05-04 16:15:16,113: INFO/job_scheduler] _notify: Set mcast_port=50049 on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-4) and saved
[2017-05-04 16:15:16,117: INFO/job_scheduler] _notify: Updating .network_interfaces of item 4 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-05-04 16:15:16,143: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-4) and saved
[2017-05-04 16:15:16,153: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 4 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-04 16:15:16,169: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-4) and saved
[2017-05-04 16:15:16,178: INFO/job_scheduler] run_next: 1 runnable jobs of (11 pending, 3 tasked)
[2017-05-04 16:15:16,217: INFO/job_scheduler] Job 52: RunJobThread.run
[2017-05-04 16:15:16,219: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-05-04 16:15:16,233: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 57077d4f-1144-4771-84c1-f2c51da53a82
[2017-05-04 16:15:16,233: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 57077d4f-1144-4771-84c1-f2c51da53a82
[2017-05-04 16:15:22,404: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:15:22,404: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:15:22,405: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:15:22,405: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:15:22,405: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c0461797-e1ec-473b-b5be-4409e1f559f5
[2017-05-04 16:15:22,405: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c0461797-e1ec-473b-b5be-4409e1f559f5
[2017-05-04 16:15:22,407: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c0461797-e1ec-473b-b5be-4409e1f559f5
[2017-05-04 16:15:22,407: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c0461797-e1ec-473b-b5be-4409e1f559f5
[2017-05-04 16:15:22,407: INFO/job_scheduler] Job 16 finished 3 steps successfully
[2017-05-04 16:15:22,449: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 139657774326608
[2017-05-04 16:15:22,457: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-04 16:15:22,468: INFO/job] Job 16: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-04 16:15:22,468: INFO/job] Job 16: not_deleted=True
[2017-05-04 16:15:22,468: INFO/job_scheduler] Job 16: complete_job: Updating cache
[2017-05-04 16:15:22,468: INFO/job_scheduler] Job 16: locked_item 139657774326608 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-04 16:15:22,471: INFO/job_scheduler] Job 16 completing (errored=False, cancelled=False)
[2017-05-04 16:15:22,585: INFO/job_scheduler] run_next: 1 runnable jobs of (10 pending, 3 tasked)
[2017-05-04 16:15:22,636: INFO/job_scheduler] Job 18: RunJobThread.run
[2017-05-04 16:15:22,637: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-04 16:15:22,675: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 914157c6-2550-4856-99b3-a081fdc9a7ec
[2017-05-04 16:15:22,675: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 914157c6-2550-4856-99b3-a081fdc9a7ec
[2017-05-04 16:15:25,051: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 4 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from True to False
[2017-05-04 16:15:25,057: INFO/job_scheduler] _notify: Set corosync_reported_up=False on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-4) and saved
[2017-05-04 16:15:25,066: INFO/job_scheduler] run_next: 0 runnable jobs of (9 pending, 4 tasked)
[2017-05-04 16:15:29,822: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:15:29,822: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:15:29,822: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:15:29,822: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:15:29,823: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 914157c6-2550-4856-99b3-a081fdc9a7ec
[2017-05-04 16:15:29,823: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 914157c6-2550-4856-99b3-a081fdc9a7ec
[2017-05-04 16:15:29,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 914157c6-2550-4856-99b3-a081fdc9a7ec
[2017-05-04 16:15:29,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 914157c6-2550-4856-99b3-a081fdc9a7ec
[2017-05-04 16:15:29,841: INFO/job_scheduler] Job 18 finished 1 steps successfully
[2017-05-04 16:15:29,918: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 139657774326608
[2017-05-04 16:15:29,957: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 30
[2017-05-04 16:15:29,964: INFO/job] Job 18: StateChangeJob complete, setting state started on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-04 16:15:29,964: INFO/job] Job 18: not_deleted=True
[2017-05-04 16:15:29,964: INFO/job_scheduler] Job 18: complete_job: Updating cache
[2017-05-04 16:15:29,964: INFO/job_scheduler] Job 18: locked_item 139657774326608 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-04 16:15:29,967: INFO/job_scheduler] Job 18 completing (errored=False, cancelled=False)
[2017-05-04 16:15:29,979: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-1
[2017-05-04 16:15:29,979: INFO/job_scheduler] _notify: Updating .mcast_port of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 47491
[2017-05-04 16:15:29,983: INFO/job_scheduler] _notify: Set mcast_port=47491 on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-05-04 16:15:29,985: INFO/job_scheduler] _notify: Updating .network_interfaces of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-05-04 16:15:30,002: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-05-04 16:15:30,008: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-04 16:15:30,015: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-05-04 16:15:30,025: INFO/job_scheduler] run_next: 1 runnable jobs of (9 pending, 3 tasked)
[2017-05-04 16:15:30,059: INFO/job_scheduler] Job 19: RunJobThread.run
[2017-05-04 16:15:30,062: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-05-04 16:15:30,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0dc0238a-6da2-40cb-90a0-dd78e424de6d
[2017-05-04 16:15:30,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0dc0238a-6da2-40cb-90a0-dd78e424de6d
[2017-05-04 16:15:34,642: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from True to False
[2017-05-04 16:15:34,650: INFO/job_scheduler] _notify: Set corosync_reported_up=False on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-05-04 16:15:34,656: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 16:15:35,125: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 4 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-04 16:15:35,130: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-4) and saved
[2017-05-04 16:15:35,140: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-04 16:15:41,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:15:41,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:15:41,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:15:41,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:15:41,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 57077d4f-1144-4771-84c1-f2c51da53a82
[2017-05-04 16:15:41,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 57077d4f-1144-4771-84c1-f2c51da53a82
[2017-05-04 16:15:41,560: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 57077d4f-1144-4771-84c1-f2c51da53a82
[2017-05-04 16:15:41,560: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 57077d4f-1144-4771-84c1-f2c51da53a82
[2017-05-04 16:15:41,562: INFO/job_scheduler] Job 52 finished 1 steps successfully
[2017-05-04 16:15:41,899: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 139657774441552
[2017-05-04 16:15:41,904: INFO/job] Job 52: StateChangeJob complete, setting state managed on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-04 16:15:41,905: INFO/job] Job 52: not_deleted=True
[2017-05-04 16:15:41,905: INFO/job_scheduler] Job 52: complete_job: Updating cache
[2017-05-04 16:15:41,905: INFO/job_scheduler] Job 52: locked_item 139657774441552 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-04 16:15:41,908: INFO/job_scheduler] Job 52 completing (errored=False, cancelled=False)
[2017-05-04 16:15:41,960: INFO/job_scheduler] Replaying 3 buffered notifications for ManagedHost-4
[2017-05-04 16:15:41,981: INFO/job_scheduler] _notify: Updating .immutable_state of item 4 (lotus-32vm6.lotus.hpdd.lab.intel.com) from True to False
[2017-05-04 16:15:42,003: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-4) and saved
[2017-05-04 16:15:42,060: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 3 tasked)
[2017-05-04 16:15:54,400: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:15:54,400: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:15:54,400: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:15:54,400: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:15:54,400: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0dc0238a-6da2-40cb-90a0-dd78e424de6d
[2017-05-04 16:15:54,400: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0dc0238a-6da2-40cb-90a0-dd78e424de6d
[2017-05-04 16:15:54,411: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0dc0238a-6da2-40cb-90a0-dd78e424de6d
[2017-05-04 16:15:54,411: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0dc0238a-6da2-40cb-90a0-dd78e424de6d
[2017-05-04 16:15:54,413: INFO/job_scheduler] Job 19 finished 1 steps successfully
[2017-05-04 16:15:54,680: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-04 16:15:54,696: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-05-04 16:15:54,710: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 3 tasked)
[2017-05-04 16:15:54,878: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 139657908295312
[2017-05-04 16:15:54,883: INFO/job] Job 19: StateChangeJob complete, setting state managed on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-04 16:15:54,884: INFO/job] Job 19: not_deleted=True
[2017-05-04 16:15:54,884: INFO/job_scheduler] Job 19: complete_job: Updating cache
[2017-05-04 16:15:54,884: INFO/job_scheduler] Job 19: locked_item 139657908295312 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-04 16:15:54,886: INFO/job_scheduler] Job 19 completing (errored=False, cancelled=False)
[2017-05-04 16:15:54,946: INFO/job_scheduler] Replaying 3 buffered notifications for ManagedHost-1
[2017-05-04 16:15:54,952: INFO/job_scheduler] _notify: Updating .immutable_state of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com) from True to False
[2017-05-04 16:15:54,957: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-1) and saved
[2017-05-04 16:15:54,978: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 2 tasked)
[2017-05-04 16:21:35,767: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 2 tasked)
[2017-05-04 16:21:35,782: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 2 tasked)
[2017-05-04 16:21:36,138: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 2 tasked)
[2017-05-04 16:21:36,159: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 2 tasked)
[2017-05-04 16:28:09,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:28:09,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:28:09,089: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:28:09,089: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:28:09,089: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9b37e907-9a9a-4e9f-bbb2-cc76e067ea07
[2017-05-04 16:28:09,089: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9b37e907-9a9a-4e9f-bbb2-cc76e067ea07
[2017-05-04 16:28:09,104: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9b37e907-9a9a-4e9f-bbb2-cc76e067ea07
[2017-05-04 16:28:09,104: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9b37e907-9a9a-4e9f-bbb2-cc76e067ea07
[2017-05-04 16:28:09,105: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-04 16:28:09.105218+00:00:
[2017-05-04 16:28:09,105: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-04 16:28:09.105218+00:00:
[2017-05-04 16:28:09,105: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->52649
[2017-05-04 16:28:09,105: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->52649
[2017-05-04 16:28:09,107: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-04 16:28:09,107: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-04 16:28:09,121: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 16:28:09,121: INFO/job_scheduler]   Job 23 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-05-04 16:28:09,121: INFO/job_scheduler]   Job 26 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-05-04 16:28:09,121: INFO/job_scheduler]   Job 27 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:28:09,122: INFO/job_scheduler]   Job 29 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:28:09,122: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 2 tasked)
[2017-05-04 16:28:09,123: INFO/job_scheduler] Job 23 finished 1 steps successfully
[2017-05-04 16:28:09,184: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 139657774287248
[2017-05-04 16:28:09,191: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-04 16:28:09,201: INFO/job] Job 23: StateChangeJob complete, setting state stopped on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:28:09,202: INFO/job] Job 23: not_deleted=True
[2017-05-04 16:28:09,202: INFO/job_scheduler] Job 23: complete_job: Updating cache
[2017-05-04 16:28:09,204: INFO/job_scheduler] Job 23: locked_item 139657774287248 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-04 16:28:09,215: INFO/job_scheduler] Job 23 completing (errored=False, cancelled=False)
[2017-05-04 16:28:09,239: INFO/job_scheduler] run_next: 1 runnable jobs of (8 pending, 1 tasked)
[2017-05-04 16:28:09,243: INFO/job_scheduler] Job 26: RunJobThread.run
[2017-05-04 16:28:09,244: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-05-04 16:28:09,258: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0dc43ad9-6eb6-40eb-9c1a-d542d450585b
[2017-05-04 16:28:09,258: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0dc43ad9-6eb6-40eb-9c1a-d542d450585b
[2017-05-04 16:28:09,884: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 2 tasked)
[2017-05-04 16:28:09,898: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 2 tasked)
[2017-05-04 16:28:16,934: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 2 tasked)
[2017-05-04 16:28:16,951: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 2 tasked)
[2017-05-04 16:28:17,379: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:28:17,379: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:28:17,379: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:28:17,379: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:28:17,379: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0dc43ad9-6eb6-40eb-9c1a-d542d450585b
[2017-05-04 16:28:17,379: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0dc43ad9-6eb6-40eb-9c1a-d542d450585b
[2017-05-04 16:28:17,428: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0dc43ad9-6eb6-40eb-9c1a-d542d450585b
[2017-05-04 16:28:17,428: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0dc43ad9-6eb6-40eb-9c1a-d542d450585b
[2017-05-04 16:28:17,429: INFO/job_scheduler] Job 26 finished 1 steps successfully
[2017-05-04 16:28:17,482: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 139657774287248
[2017-05-04 16:28:17,520: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 30
[2017-05-04 16:28:17,528: INFO/job] Job 26: StateChangeJob complete, setting state started on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:28:17,528: INFO/job] Job 26: not_deleted=True
[2017-05-04 16:28:17,528: INFO/job_scheduler] Job 26: complete_job: Updating cache
[2017-05-04 16:28:17,529: INFO/job_scheduler] Job 26: locked_item 139657774287248 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-04 16:28:17,535: INFO/job_scheduler] Job 26 completing (errored=False, cancelled=False)
[2017-05-04 16:28:17,554: INFO/job_scheduler] run_next: 1 runnable jobs of (7 pending, 1 tasked)
[2017-05-04 16:28:17,584: INFO/job_scheduler] Job 27: RunJobThread.run
[2017-05-04 16:28:17,585: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-04 16:28:17,603: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 22c21980-717e-429b-9b1b-d09c0b6f242d
[2017-05-04 16:28:17,603: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 22c21980-717e-429b-9b1b-d09c0b6f242d
[2017-05-04 16:28:18,011: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:28:18,011: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:28:18,011: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:28:18,011: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:28:18,011: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d268772b-51a0-446a-bc32-6514ddd20c9a
[2017-05-04 16:28:18,011: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d268772b-51a0-446a-bc32-6514ddd20c9a
[2017-05-04 16:28:18,044: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d268772b-51a0-446a-bc32-6514ddd20c9a
[2017-05-04 16:28:18,044: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d268772b-51a0-446a-bc32-6514ddd20c9a
[2017-05-04 16:28:18,045: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-04 16:28:18.045467+00:00:
[2017-05-04 16:28:18,045: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-04 16:28:18.045467+00:00:
[2017-05-04 16:28:18,045: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->53219
[2017-05-04 16:28:18,045: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->53219
[2017-05-04 16:28:18,047: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-04 16:28:18,047: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-04 16:28:18,061: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 16:28:18,061: INFO/job_scheduler]   Job 34 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-05-04 16:28:18,061: INFO/job_scheduler]   Job 37 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-05-04 16:28:18,061: INFO/job_scheduler]   Job 38 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:28:18,062: INFO/job_scheduler]   Job 40 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:28:18,062: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 2 tasked)
[2017-05-04 16:28:18,064: INFO/job_scheduler] Job 34 finished 1 steps successfully
[2017-05-04 16:28:18,116: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 139657774313680
[2017-05-04 16:28:18,123: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-04 16:28:18,139: INFO/job] Job 34: StateChangeJob complete, setting state stopped on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:28:18,140: INFO/job] Job 34: not_deleted=True
[2017-05-04 16:28:18,141: INFO/job_scheduler] Job 34: complete_job: Updating cache
[2017-05-04 16:28:18,143: INFO/job_scheduler] Job 34: locked_item 139657774313680 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-04 16:28:18,147: INFO/job_scheduler] Job 34 completing (errored=False, cancelled=False)
[2017-05-04 16:28:18,165: INFO/job_scheduler] run_next: 1 runnable jobs of (6 pending, 1 tasked)
[2017-05-04 16:28:18,171: INFO/job_scheduler] Job 37: RunJobThread.run
[2017-05-04 16:28:18,171: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-05-04 16:28:18,209: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 20213445-9f27-4574-8099-1a143f7a20b8
[2017-05-04 16:28:18,209: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 20213445-9f27-4574-8099-1a143f7a20b8
[2017-05-04 16:28:24,653: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:28:24,653: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:28:24,653: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:28:24,653: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:28:24,653: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 22c21980-717e-429b-9b1b-d09c0b6f242d
[2017-05-04 16:28:24,653: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 22c21980-717e-429b-9b1b-d09c0b6f242d
[2017-05-04 16:28:24,668: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 22c21980-717e-429b-9b1b-d09c0b6f242d
[2017-05-04 16:28:24,668: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 22c21980-717e-429b-9b1b-d09c0b6f242d
[2017-05-04 16:28:24,669: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-05-04 16:28:24,682: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3fee0d14-65a4-4b2c-8f67-e4a36727c2b5
[2017-05-04 16:28:24,682: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3fee0d14-65a4-4b2c-8f67-e4a36727c2b5
[2017-05-04 16:28:25,163: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 2 tasked)
[2017-05-04 16:28:25,177: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 2 tasked)
[2017-05-04 16:28:26,329: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:28:26,329: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:28:26,330: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:28:26,330: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:28:26,330: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 20213445-9f27-4574-8099-1a143f7a20b8
[2017-05-04 16:28:26,330: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 20213445-9f27-4574-8099-1a143f7a20b8
[2017-05-04 16:28:26,375: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 20213445-9f27-4574-8099-1a143f7a20b8
[2017-05-04 16:28:26,375: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 20213445-9f27-4574-8099-1a143f7a20b8
[2017-05-04 16:28:26,375: INFO/job_scheduler] Job 37 finished 1 steps successfully
[2017-05-04 16:28:26,424: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 139657774313680
[2017-05-04 16:28:26,442: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm8.lotus.hpdd.lab.intel.com at severity 30
[2017-05-04 16:28:26,461: INFO/job] Job 37: StateChangeJob complete, setting state started on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:28:26,463: INFO/job] Job 37: not_deleted=True
[2017-05-04 16:28:26,465: INFO/job_scheduler] Job 37: complete_job: Updating cache
[2017-05-04 16:28:26,465: INFO/job_scheduler] Job 37: locked_item 139657774313680 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-04 16:28:26,476: INFO/job_scheduler] Job 37 completing (errored=False, cancelled=False)
[2017-05-04 16:28:26,495: INFO/job_scheduler] run_next: 1 runnable jobs of (5 pending, 1 tasked)
[2017-05-04 16:28:26,524: INFO/job_scheduler] Job 38: RunJobThread.run
[2017-05-04 16:28:26,525: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-04 16:28:26,541: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7e454e41-f5a0-4a96-aa32-fc40480dd662
[2017-05-04 16:28:26,541: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7e454e41-f5a0-4a96-aa32-fc40480dd662
[2017-05-04 16:28:33,647: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:28:33,647: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:28:33,648: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:28:33,648: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:28:33,648: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7e454e41-f5a0-4a96-aa32-fc40480dd662
[2017-05-04 16:28:33,648: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7e454e41-f5a0-4a96-aa32-fc40480dd662
[2017-05-04 16:28:33,658: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7e454e41-f5a0-4a96-aa32-fc40480dd662
[2017-05-04 16:28:33,658: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7e454e41-f5a0-4a96-aa32-fc40480dd662
[2017-05-04 16:28:33,659: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-05-04 16:28:33,675: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 768abffb-632f-428f-9c30-da28aea1d1ca
[2017-05-04 16:28:33,675: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 768abffb-632f-428f-9c30-da28aea1d1ca
[2017-05-04 16:28:36,206: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-05-04 16:28:36,223: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-05-04 16:28:46,223: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-05-04 16:28:46,237: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-05-04 16:28:58,013: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-05-04 16:28:58,029: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-05-04 16:29:06,304: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-05-04 16:29:06,318: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-05-04 16:29:18,091: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 16:29:18,091: INFO/job_scheduler]   Job 27 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:29:18,091: INFO/job_scheduler]   Job 29 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:29:18,092: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-05-04 16:29:26,355: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 16:29:26,356: INFO/job_scheduler]   Job 38 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:29:26,356: INFO/job_scheduler]   Job 40 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:29:26,356: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-05-04 16:29:28,116: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 16:29:28,116: INFO/job_scheduler]   Job 27 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:29:28,116: INFO/job_scheduler]   Job 29 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:29:28,116: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-05-04 16:29:36,393: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 16:29:36,393: INFO/job_scheduler]   Job 38 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:29:36,393: INFO/job_scheduler]   Job 40 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:29:36,394: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-05-04 16:29:38,272: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 16:29:38,273: INFO/job_scheduler]   Job 27 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:29:38,273: INFO/job_scheduler]   Job 29 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:29:38,273: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-05-04 16:29:46,040: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:29:46,040: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:29:46,040: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:29:46,040: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:29:46,040: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3fee0d14-65a4-4b2c-8f67-e4a36727c2b5
[2017-05-04 16:29:46,040: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3fee0d14-65a4-4b2c-8f67-e4a36727c2b5
[2017-05-04 16:29:46,065: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3fee0d14-65a4-4b2c-8f67-e4a36727c2b5
[2017-05-04 16:29:46,065: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3fee0d14-65a4-4b2c-8f67-e4a36727c2b5
[2017-05-04 16:29:46,067: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-04 16:29:46,080: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 13be5ebe-14f1-42a8-8a44-a4ca6311ce5b
[2017-05-04 16:29:46,080: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 13be5ebe-14f1-42a8-8a44-a4ca6311ce5b
[2017-05-04 16:29:46,405: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 16:29:46,405: INFO/job_scheduler]   Job 38 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:29:46,405: INFO/job_scheduler]   Job 40 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:29:46,405: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-05-04 16:29:48,257: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-05-04 16:29:48,271: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-05-04 16:29:48,297: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 16:29:48,297: INFO/job_scheduler]   Job 27 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:29:48,297: INFO/job_scheduler]   Job 29 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:29:48,298: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-05-04 16:29:53,134: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:29:53,134: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:29:53,134: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:29:53,134: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:29:53,134: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 13be5ebe-14f1-42a8-8a44-a4ca6311ce5b
[2017-05-04 16:29:53,134: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 13be5ebe-14f1-42a8-8a44-a4ca6311ce5b
[2017-05-04 16:29:53,149: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 13be5ebe-14f1-42a8-8a44-a4ca6311ce5b
[2017-05-04 16:29:53,149: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 13be5ebe-14f1-42a8-8a44-a4ca6311ce5b
[2017-05-04 16:29:53,149: INFO/job_scheduler] Job 27 finished 3 steps successfully
[2017-05-04 16:29:53,205: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 139657774328912
[2017-05-04 16:29:53,212: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-04 16:29:53,214: INFO/job] Job 27: StateChangeJob complete, setting state stopped on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-04 16:29:53,215: INFO/job] Job 27: not_deleted=True
[2017-05-04 16:29:53,215: INFO/job_scheduler] Job 27: complete_job: Updating cache
[2017-05-04 16:29:53,216: INFO/job_scheduler] Job 27: locked_item 139657774328912 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-04 16:29:53,228: INFO/job_scheduler] Job 27 completing (errored=False, cancelled=False)
[2017-05-04 16:29:53,271: INFO/job_scheduler] run_next: 1 runnable jobs of (4 pending, 1 tasked)
[2017-05-04 16:29:53,282: INFO/job_scheduler] Job 29: RunJobThread.run
[2017-05-04 16:29:53,282: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-04 16:29:53,325: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0c8eb6df-d696-44cf-92bb-60d6b2e24969
[2017-05-04 16:29:53,325: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0c8eb6df-d696-44cf-92bb-60d6b2e24969
[2017-05-04 16:29:55,089: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:29:55,089: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:29:55,089: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:29:55,089: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:29:55,089: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 768abffb-632f-428f-9c30-da28aea1d1ca
[2017-05-04 16:29:55,089: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 768abffb-632f-428f-9c30-da28aea1d1ca
[2017-05-04 16:29:55,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 768abffb-632f-428f-9c30-da28aea1d1ca
[2017-05-04 16:29:55,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 768abffb-632f-428f-9c30-da28aea1d1ca
[2017-05-04 16:29:55,117: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-04 16:29:55,130: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 39d262ae-00f6-4d44-9ef7-73c3fe8bf303
[2017-05-04 16:29:55,130: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 39d262ae-00f6-4d44-9ef7-73c3fe8bf303
[2017-05-04 16:29:56,409: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 2 tasked)
[2017-05-04 16:29:56,426: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 2 tasked)
[2017-05-04 16:29:56,453: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-04 16:29:56,453: INFO/job_scheduler]   Job 38 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:29:56,453: INFO/job_scheduler]   Job 40 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-04 16:29:56,453: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 2 tasked)
[2017-05-04 16:29:58,314: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 2 tasked)
[2017-05-04 16:29:58,331: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 2 tasked)
[2017-05-04 16:30:00,452: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:30:00,452: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:30:00,453: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:30:00,453: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:30:00,453: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0c8eb6df-d696-44cf-92bb-60d6b2e24969
[2017-05-04 16:30:00,453: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0c8eb6df-d696-44cf-92bb-60d6b2e24969
[2017-05-04 16:30:00,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0c8eb6df-d696-44cf-92bb-60d6b2e24969
[2017-05-04 16:30:00,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0c8eb6df-d696-44cf-92bb-60d6b2e24969
[2017-05-04 16:30:00,495: INFO/job_scheduler] Job 29 finished 1 steps successfully
[2017-05-04 16:30:00,536: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 139657774328912
[2017-05-04 16:30:00,572: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 30
[2017-05-04 16:30:00,581: INFO/job] Job 29: StateChangeJob complete, setting state started on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-04 16:30:00,581: INFO/job] Job 29: not_deleted=True
[2017-05-04 16:30:00,581: INFO/job_scheduler] Job 29: complete_job: Updating cache
[2017-05-04 16:30:00,581: INFO/job_scheduler] Job 29: locked_item 139657774328912 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-04 16:30:00,584: INFO/job_scheduler] Job 29 completing (errored=False, cancelled=False)
[2017-05-04 16:30:00,597: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-2
[2017-05-04 16:30:00,598: INFO/job_scheduler] _notify: Updating .mcast_port of item 2 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 52649
[2017-05-04 16:30:00,617: INFO/job_scheduler] _notify: Set mcast_port=52649 on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-2) and saved
[2017-05-04 16:30:00,618: INFO/job_scheduler] _notify: Updating .network_interfaces of item 2 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-05-04 16:30:00,628: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-2) and saved
[2017-05-04 16:30:00,634: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 2 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-04 16:30:00,638: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-2) and saved
[2017-05-04 16:30:00,649: INFO/job_scheduler] run_next: 1 runnable jobs of (3 pending, 1 tasked)
[2017-05-04 16:30:00,722: INFO/job_scheduler] Job 30: RunJobThread.run
[2017-05-04 16:30:00,724: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-05-04 16:30:00,738: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8f95d92f-882e-4b74-9227-0cbcd00d2116
[2017-05-04 16:30:00,738: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8f95d92f-882e-4b74-9227-0cbcd00d2116
[2017-05-04 16:30:02,240: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:30:02,240: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:30:02,240: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:30:02,240: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:30:02,241: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 39d262ae-00f6-4d44-9ef7-73c3fe8bf303
[2017-05-04 16:30:02,241: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 39d262ae-00f6-4d44-9ef7-73c3fe8bf303
[2017-05-04 16:30:02,246: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 39d262ae-00f6-4d44-9ef7-73c3fe8bf303
[2017-05-04 16:30:02,246: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 39d262ae-00f6-4d44-9ef7-73c3fe8bf303
[2017-05-04 16:30:02,246: INFO/job_scheduler] Job 38 finished 3 steps successfully
[2017-05-04 16:30:02,311: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 139657908284432
[2017-05-04 16:30:02,317: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-04 16:30:02,323: INFO/job] Job 38: StateChangeJob complete, setting state stopped on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-04 16:30:02,323: INFO/job] Job 38: not_deleted=True
[2017-05-04 16:30:02,323: INFO/job_scheduler] Job 38: complete_job: Updating cache
[2017-05-04 16:30:02,323: INFO/job_scheduler] Job 38: locked_item 139657908284432 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-04 16:30:02,329: INFO/job_scheduler] Job 38 completing (errored=False, cancelled=False)
[2017-05-04 16:30:02,358: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 1 tasked)
[2017-05-04 16:30:02,403: INFO/job_scheduler] Job 40: RunJobThread.run
[2017-05-04 16:30:02,404: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-04 16:30:02,428: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fe9fae61-2dd2-4d31-a069-a30c2fa45985
[2017-05-04 16:30:02,428: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fe9fae61-2dd2-4d31-a069-a30c2fa45985
[2017-05-04 16:30:06,463: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 16:30:06,478: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 16:30:08,313: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 2 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from True to False
[2017-05-04 16:30:08,318: INFO/job_scheduler] _notify: Set corosync_reported_up=False on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-2) and saved
[2017-05-04 16:30:08,324: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-04 16:30:09,503: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:30:09,503: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:30:09,504: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:30:09,504: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:30:09,504: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fe9fae61-2dd2-4d31-a069-a30c2fa45985
[2017-05-04 16:30:09,504: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fe9fae61-2dd2-4d31-a069-a30c2fa45985
[2017-05-04 16:30:09,542: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fe9fae61-2dd2-4d31-a069-a30c2fa45985
[2017-05-04 16:30:09,542: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fe9fae61-2dd2-4d31-a069-a30c2fa45985
[2017-05-04 16:30:09,543: INFO/job_scheduler] Job 40 finished 1 steps successfully
[2017-05-04 16:30:09,618: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 139657908284432
[2017-05-04 16:30:09,648: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm8.lotus.hpdd.lab.intel.com at severity 30
[2017-05-04 16:30:09,658: INFO/job] Job 40: StateChangeJob complete, setting state started on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-04 16:30:09,658: INFO/job] Job 40: not_deleted=True
[2017-05-04 16:30:09,659: INFO/job_scheduler] Job 40: complete_job: Updating cache
[2017-05-04 16:30:09,659: INFO/job_scheduler] Job 40: locked_item 139657908284432 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-04 16:30:09,661: INFO/job_scheduler] Job 40 completing (errored=False, cancelled=False)
[2017-05-04 16:30:09,686: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-3
[2017-05-04 16:30:09,687: INFO/job_scheduler] _notify: Updating .mcast_port of item 3 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 53219
[2017-05-04 16:30:09,691: INFO/job_scheduler] _notify: Set mcast_port=53219 on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-3) and saved
[2017-05-04 16:30:09,692: INFO/job_scheduler] _notify: Updating .network_interfaces of item 3 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-05-04 16:30:09,721: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-3) and saved
[2017-05-04 16:30:09,737: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 3 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-04 16:30:09,752: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-3) and saved
[2017-05-04 16:30:09,771: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-05-04 16:30:09,815: INFO/job_scheduler] Job 41: RunJobThread.run
[2017-05-04 16:30:09,817: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-05-04 16:30:09,832: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9c9d8069-ff89-4b7a-9394-3c9f9f0ea396
[2017-05-04 16:30:09,832: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9c9d8069-ff89-4b7a-9394-3c9f9f0ea396
[2017-05-04 16:30:16,473: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 3 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from True to False
[2017-05-04 16:30:16,480: INFO/job_scheduler] _notify: Set corosync_reported_up=False on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-3) and saved
[2017-05-04 16:30:16,488: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-05-04 16:30:27,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:30:27,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:30:27,086: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:30:27,086: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:30:27,086: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8f95d92f-882e-4b74-9227-0cbcd00d2116
[2017-05-04 16:30:27,086: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8f95d92f-882e-4b74-9227-0cbcd00d2116
[2017-05-04 16:30:27,109: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8f95d92f-882e-4b74-9227-0cbcd00d2116
[2017-05-04 16:30:27,109: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8f95d92f-882e-4b74-9227-0cbcd00d2116
[2017-05-04 16:30:27,111: INFO/job_scheduler] Job 30 finished 1 steps successfully
[2017-05-04 16:30:27,408: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 139657774272400
[2017-05-04 16:30:27,413: INFO/job] Job 30: StateChangeJob complete, setting state managed on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-04 16:30:27,414: INFO/job] Job 30: not_deleted=True
[2017-05-04 16:30:27,414: INFO/job_scheduler] Job 30: complete_job: Updating cache
[2017-05-04 16:30:27,414: INFO/job_scheduler] Job 30: locked_item 139657774272400 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-04 16:30:27,417: INFO/job_scheduler] Job 30 completing (errored=False, cancelled=False)
[2017-05-04 16:30:27,453: INFO/job_scheduler] Replaying 3 buffered notifications for ManagedHost-2
[2017-05-04 16:30:27,485: INFO/job_scheduler] _notify: Updating .immutable_state of item 2 (lotus-32vm7.lotus.hpdd.lab.intel.com) from True to False
[2017-05-04 16:30:27,500: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-2) and saved
[2017-05-04 16:30:27,548: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-04 16:30:28,432: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 2 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-04 16:30:28,437: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-2) and saved
[2017-05-04 16:30:28,444: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-04 16:30:34,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:30:34,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:30:34,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:30:34,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:30:34,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9c9d8069-ff89-4b7a-9394-3c9f9f0ea396
[2017-05-04 16:30:34,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9c9d8069-ff89-4b7a-9394-3c9f9f0ea396
[2017-05-04 16:30:34,200: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9c9d8069-ff89-4b7a-9394-3c9f9f0ea396
[2017-05-04 16:30:34,200: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9c9d8069-ff89-4b7a-9394-3c9f9f0ea396
[2017-05-04 16:30:34,202: INFO/job_scheduler] Job 41 finished 1 steps successfully
[2017-05-04 16:30:34,637: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 139657908293712
[2017-05-04 16:30:34,642: INFO/job] Job 41: StateChangeJob complete, setting state managed on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-04 16:30:34,643: INFO/job] Job 41: not_deleted=True
[2017-05-04 16:30:34,643: INFO/job_scheduler] Job 41: complete_job: Updating cache
[2017-05-04 16:30:34,643: INFO/job_scheduler] Job 41: locked_item 139657908293712 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-04 16:30:34,645: INFO/job_scheduler] Job 41 completing (errored=False, cancelled=False)
[2017-05-04 16:30:34,693: INFO/job_scheduler] Replaying 3 buffered notifications for ManagedHost-3
[2017-05-04 16:30:34,717: INFO/job_scheduler] _notify: Updating .immutable_state of item 3 (lotus-32vm8.lotus.hpdd.lab.intel.com) from True to False
[2017-05-04 16:30:34,723: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-3) and saved
[2017-05-04 16:30:34,803: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 16:30:36,578: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 3 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-04 16:30:36,583: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-3) and saved
[2017-05-04 16:30:36,588: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 16:31:46,784: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 16:31:46,802: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 16:31:47,714: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 16:31:47,755: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 16:31:47,829: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 16:31:47,845: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 16:31:48,609: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 16:31:48,738: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 16:32:03,352: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 16:32:03,354: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-04 16:32:03,385: INFO/command_plan] add_jobs: created Job 53 (Trigger plugin poll for linux plugins)
[2017-05-04 16:32:03,451: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-04 16:32:03,455: INFO/job_scheduler] Job 53: RunJobThread.run
[2017-05-04 16:32:03,456: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-04 16:32:03,466: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-04 16:32:03,466: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-04 16:32:03,473: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8cdfd77d-672f-4544-a03a-5e606458a7d6
[2017-05-04 16:32:03,473: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8cdfd77d-672f-4544-a03a-5e606458a7d6
[2017-05-04 16:32:03,487: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 392eb202-b9ce-401a-a1f9-538b22e5a155
[2017-05-04 16:32:03,487: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 392eb202-b9ce-401a-a1f9-538b22e5a155
[2017-05-04 16:32:03,508: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6deac2fb-e389-405f-82a2-7d1dd8f55535
[2017-05-04 16:32:03,508: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6deac2fb-e389-405f-82a2-7d1dd8f55535
[2017-05-04 16:32:03,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:32:03,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:32:03,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:32:03,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:32:03,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 392eb202-b9ce-401a-a1f9-538b22e5a155
[2017-05-04 16:32:03,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 392eb202-b9ce-401a-a1f9-538b22e5a155
[2017-05-04 16:32:03,603: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:32:03,603: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:32:03,603: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:32:03,603: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:32:03,603: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8cdfd77d-672f-4544-a03a-5e606458a7d6
[2017-05-04 16:32:03,603: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8cdfd77d-672f-4544-a03a-5e606458a7d6
[2017-05-04 16:32:03,604: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8cdfd77d-672f-4544-a03a-5e606458a7d6
[2017-05-04 16:32:03,604: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8cdfd77d-672f-4544-a03a-5e606458a7d6
[2017-05-04 16:32:03,605: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-04 16:32:03,606: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:32:03,606: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:32:03,607: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:32:03,607: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:32:03,607: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6deac2fb-e389-405f-82a2-7d1dd8f55535
[2017-05-04 16:32:03,607: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6deac2fb-e389-405f-82a2-7d1dd8f55535
[2017-05-04 16:32:03,629: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 392eb202-b9ce-401a-a1f9-538b22e5a155
[2017-05-04 16:32:03,629: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 392eb202-b9ce-401a-a1f9-538b22e5a155
[2017-05-04 16:32:03,629: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-04 16:32:03,646: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6deac2fb-e389-405f-82a2-7d1dd8f55535
[2017-05-04 16:32:03,646: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6deac2fb-e389-405f-82a2-7d1dd8f55535
[2017-05-04 16:32:03,646: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-04 16:32:03,646: INFO/job_scheduler] Job 53 finished 1 steps successfully
[2017-05-04 16:32:03,731: INFO/job_scheduler] Job 53: complete_job: Updating cache
[2017-05-04 16:32:03,731: INFO/job_scheduler] Job 53 completing (errored=False, cancelled=False)
[2017-05-04 16:32:03,791: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 16:32:43,362: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 16:32:43,363: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-04 16:32:43,384: INFO/command_plan] add_jobs: created Job 54 (Trigger plugin poll for linux plugins)
[2017-05-04 16:32:43,415: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-04 16:32:43,431: INFO/job_scheduler] Job 54: RunJobThread.run
[2017-05-04 16:32:43,444: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-04 16:32:43,468: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a3877f67-2bbb-45ae-af2f-f4613b2fd4bf
[2017-05-04 16:32:43,468: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a3877f67-2bbb-45ae-af2f-f4613b2fd4bf
[2017-05-04 16:32:43,469: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-04 16:32:43,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 23abb2be-a405-46d6-9cf3-af137e05c276
[2017-05-04 16:32:43,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 23abb2be-a405-46d6-9cf3-af137e05c276
[2017-05-04 16:32:43,528: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-04 16:32:43,562: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c4b5b6be-2168-4e4c-9378-2428122f92c3
[2017-05-04 16:32:43,562: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c4b5b6be-2168-4e4c-9378-2428122f92c3
[2017-05-04 16:32:43,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:32:43,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:32:43,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:32:43,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:32:43,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 23abb2be-a405-46d6-9cf3-af137e05c276
[2017-05-04 16:32:43,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 23abb2be-a405-46d6-9cf3-af137e05c276
[2017-05-04 16:32:43,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:32:43,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:32:43,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:32:43,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:32:43,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a3877f67-2bbb-45ae-af2f-f4613b2fd4bf
[2017-05-04 16:32:43,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a3877f67-2bbb-45ae-af2f-f4613b2fd4bf
[2017-05-04 16:32:43,620: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 23abb2be-a405-46d6-9cf3-af137e05c276
[2017-05-04 16:32:43,620: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 23abb2be-a405-46d6-9cf3-af137e05c276
[2017-05-04 16:32:43,621: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-04 16:32:43,647: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a3877f67-2bbb-45ae-af2f-f4613b2fd4bf
[2017-05-04 16:32:43,647: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a3877f67-2bbb-45ae-af2f-f4613b2fd4bf
[2017-05-04 16:32:43,648: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-04 16:32:43,682: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:32:43,682: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:32:43,682: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:32:43,682: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:32:43,682: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c4b5b6be-2168-4e4c-9378-2428122f92c3
[2017-05-04 16:32:43,682: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c4b5b6be-2168-4e4c-9378-2428122f92c3
[2017-05-04 16:32:43,687: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c4b5b6be-2168-4e4c-9378-2428122f92c3
[2017-05-04 16:32:43,687: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c4b5b6be-2168-4e4c-9378-2428122f92c3
[2017-05-04 16:32:43,687: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-04 16:32:43,688: INFO/job_scheduler] Job 54 finished 1 steps successfully
[2017-05-04 16:32:43,725: INFO/job_scheduler] Job 54: complete_job: Updating cache
[2017-05-04 16:32:43,725: INFO/job_scheduler] Job 54 completing (errored=False, cancelled=False)
[2017-05-04 16:32:43,803: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 16:32:52,594: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 16:32:52,594: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-04 16:32:52,610: INFO/command_plan] add_jobs: created Job 55 (Trigger plugin poll for linux plugins)
[2017-05-04 16:32:52,672: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-04 16:32:52,711: INFO/job_scheduler] Job 55: RunJobThread.run
[2017-05-04 16:32:52,728: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-04 16:32:52,763: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc da65160a-987c-4538-aee9-693504a30530
[2017-05-04 16:32:52,763: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc da65160a-987c-4538-aee9-693504a30530
[2017-05-04 16:32:52,765: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-04 16:32:52,784: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-04 16:32:52,791: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 05feedae-42e6-47f1-a255-4b3c87a02c8e
[2017-05-04 16:32:52,791: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 05feedae-42e6-47f1-a255-4b3c87a02c8e
[2017-05-04 16:32:52,803: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0903ac3a-1efd-485c-930d-7ad3d1515a0d
[2017-05-04 16:32:52,803: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0903ac3a-1efd-485c-930d-7ad3d1515a0d
[2017-05-04 16:32:52,834: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:32:52,834: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:32:52,834: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:32:52,834: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:32:52,835: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc da65160a-987c-4538-aee9-693504a30530
[2017-05-04 16:32:52,835: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc da65160a-987c-4538-aee9-693504a30530
[2017-05-04 16:32:52,839: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:32:52,839: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:32:52,839: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:32:52,839: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:32:52,839: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 05feedae-42e6-47f1-a255-4b3c87a02c8e
[2017-05-04 16:32:52,839: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 05feedae-42e6-47f1-a255-4b3c87a02c8e
[2017-05-04 16:32:52,865: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 05feedae-42e6-47f1-a255-4b3c87a02c8e
[2017-05-04 16:32:52,865: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 05feedae-42e6-47f1-a255-4b3c87a02c8e
[2017-05-04 16:32:52,865: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-04 16:32:52,884: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc da65160a-987c-4538-aee9-693504a30530
[2017-05-04 16:32:52,884: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc da65160a-987c-4538-aee9-693504a30530
[2017-05-04 16:32:52,884: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-04 16:32:52,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:32:52,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:32:52,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:32:52,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:32:52,909: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0903ac3a-1efd-485c-930d-7ad3d1515a0d
[2017-05-04 16:32:52,909: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0903ac3a-1efd-485c-930d-7ad3d1515a0d
[2017-05-04 16:32:52,921: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0903ac3a-1efd-485c-930d-7ad3d1515a0d
[2017-05-04 16:32:52,921: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0903ac3a-1efd-485c-930d-7ad3d1515a0d
[2017-05-04 16:32:52,921: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-04 16:32:52,925: INFO/job_scheduler] Job 55 finished 1 steps successfully
[2017-05-04 16:32:52,961: INFO/job_scheduler] Job 55: complete_job: Updating cache
[2017-05-04 16:32:52,961: INFO/job_scheduler] Job 55 completing (errored=False, cancelled=False)
[2017-05-04 16:32:53,009: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 16:32:56,143: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 16:32:56,144: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-04 16:32:56,159: INFO/command_plan] add_jobs: created Job 56 (Trigger plugin poll for linux plugins)
[2017-05-04 16:32:56,175: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-04 16:32:56,178: INFO/job_scheduler] Job 56: RunJobThread.run
[2017-05-04 16:32:56,179: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-04 16:32:56,188: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-04 16:32:56,188: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-04 16:32:56,202: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d3ca2060-bbb5-4243-b38f-3b00e56e84c4
[2017-05-04 16:32:56,202: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d3ca2060-bbb5-4243-b38f-3b00e56e84c4
[2017-05-04 16:32:56,218: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2848b77b-af5c-4b37-b8db-e271f2f70e1d
[2017-05-04 16:32:56,218: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2848b77b-af5c-4b37-b8db-e271f2f70e1d
[2017-05-04 16:32:56,239: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c5959ede-7921-4cf8-8767-c2c68b8c733b
[2017-05-04 16:32:56,239: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c5959ede-7921-4cf8-8767-c2c68b8c733b
[2017-05-04 16:32:56,310: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:32:56,310: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:32:56,311: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:32:56,311: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f9720a2c-cad9-4e47-92f4-4b0c987e9822
[2017-05-04 16:32:56,311: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d3ca2060-bbb5-4243-b38f-3b00e56e84c4
[2017-05-04 16:32:56,311: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d3ca2060-bbb5-4243-b38f-3b00e56e84c4
[2017-05-04 16:32:56,329: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d3ca2060-bbb5-4243-b38f-3b00e56e84c4
[2017-05-04 16:32:56,329: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d3ca2060-bbb5-4243-b38f-3b00e56e84c4
[2017-05-04 16:32:56,329: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-04 16:32:56,344: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:32:56,344: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:32:56,344: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:32:56,344: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:32:56,345: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c5959ede-7921-4cf8-8767-c2c68b8c733b
[2017-05-04 16:32:56,345: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c5959ede-7921-4cf8-8767-c2c68b8c733b
[2017-05-04 16:32:56,348: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:32:56,348: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:32:56,348: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:32:56,348: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:32:56,349: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2848b77b-af5c-4b37-b8db-e271f2f70e1d
[2017-05-04 16:32:56,349: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2848b77b-af5c-4b37-b8db-e271f2f70e1d
[2017-05-04 16:32:56,349: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2848b77b-af5c-4b37-b8db-e271f2f70e1d
[2017-05-04 16:32:56,349: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2848b77b-af5c-4b37-b8db-e271f2f70e1d
[2017-05-04 16:32:56,349: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-04 16:32:56,360: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c5959ede-7921-4cf8-8767-c2c68b8c733b
[2017-05-04 16:32:56,360: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c5959ede-7921-4cf8-8767-c2c68b8c733b
[2017-05-04 16:32:56,361: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-04 16:32:56,361: INFO/job_scheduler] Job 56 finished 1 steps successfully
[2017-05-04 16:32:56,402: INFO/job_scheduler] Job 56: complete_job: Updating cache
[2017-05-04 16:32:56,402: INFO/job_scheduler] Job 56 completing (errored=False, cancelled=False)
[2017-05-04 16:32:56,436: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 16:32:58,868: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-04 16:32:58,868: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-04 16:32:58,878: INFO/command_plan] add_jobs: created Job 57 (Trigger plugin poll for linux plugins)
[2017-05-04 16:32:58,904: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-04 16:32:58,930: INFO/job_scheduler] Job 57: RunJobThread.run
[2017-05-04 16:32:58,948: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-04 16:32:58,972: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 10e1d336-7051-4eed-8ed4-45fe6d7826ac
[2017-05-04 16:32:58,972: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 10e1d336-7051-4eed-8ed4-45fe6d7826ac
[2017-05-04 16:32:58,972: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-04 16:32:59,015: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b4268df1-71fe-4c29-a3a0-5474ad931d8c
[2017-05-04 16:32:59,015: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b4268df1-71fe-4c29-a3a0-5474ad931d8c
[2017-05-04 16:32:59,016: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-04 16:32:59,035: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5ae61426-8df0-4684-87b2-dbb4b8cb21d5
[2017-05-04 16:32:59,035: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5ae61426-8df0-4684-87b2-dbb4b8cb21d5
[2017-05-04 16:32:59,069: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:32:59,069: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:32:59,069: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:32:59,069: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/efa5e8b3-9c32-4db1-87e8-2d499ca720b4
[2017-05-04 16:32:59,069: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 10e1d336-7051-4eed-8ed4-45fe6d7826ac
[2017-05-04 16:32:59,069: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 10e1d336-7051-4eed-8ed4-45fe6d7826ac
[2017-05-04 16:32:59,077: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:32:59,077: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:32:59,077: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:32:59,077: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/d549f7cd-83d8-4fa4-a09a-d5117397458d
[2017-05-04 16:32:59,078: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b4268df1-71fe-4c29-a3a0-5474ad931d8c
[2017-05-04 16:32:59,078: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b4268df1-71fe-4c29-a3a0-5474ad931d8c
[2017-05-04 16:32:59,091: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:32:59,091: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:32:59,092: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:32:59,092: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2ea5e8da-7ee1-42ca-8669-9b72cff41d30
[2017-05-04 16:32:59,092: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5ae61426-8df0-4684-87b2-dbb4b8cb21d5
[2017-05-04 16:32:59,092: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5ae61426-8df0-4684-87b2-dbb4b8cb21d5
[2017-05-04 16:32:59,093: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b4268df1-71fe-4c29-a3a0-5474ad931d8c
[2017-05-04 16:32:59,093: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b4268df1-71fe-4c29-a3a0-5474ad931d8c
[2017-05-04 16:32:59,094: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-04 16:32:59,097: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 10e1d336-7051-4eed-8ed4-45fe6d7826ac
[2017-05-04 16:32:59,097: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 10e1d336-7051-4eed-8ed4-45fe6d7826ac
[2017-05-04 16:32:59,097: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-04 16:32:59,106: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5ae61426-8df0-4684-87b2-dbb4b8cb21d5
[2017-05-04 16:32:59,106: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5ae61426-8df0-4684-87b2-dbb4b8cb21d5
[2017-05-04 16:32:59,106: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-04 16:32:59,107: INFO/job_scheduler] Job 57 finished 1 steps successfully
[2017-05-04 16:32:59,134: INFO/job_scheduler] Job 57: complete_job: Updating cache
[2017-05-04 16:32:59,134: INFO/job_scheduler] Job 57 completing (errored=False, cancelled=False)
[2017-05-04 16:32:59,209: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-04 16:34:05,412: INFO/chroma_service] Stopping job_scheduler
[2017-05-04 16:34:05,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-05-04 16:34:05,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-05-04 16:34:05,412: INFO/queue] Stopping ServiceQueue agent_action_runner_rx
[2017-05-04 16:34:06,172: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-05-04 16:34:06,172: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-05-04 16:34:06,172: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-05-04 16:34:06,172: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-05-04 16:34:06,173: INFO/job_scheduler] Stopping...
[2017-05-04 16:34:06,173: INFO/queue] Stopping ServiceQueue job_scheduler_notifications
[2017-05-04 16:34:06,173: INFO/job_scheduler] Joining...
[2017-05-04 16:34:06,923: INFO/job_scheduler] Complete.
[2017-05-04 16:34:06,923: INFO/chroma_service] Joining job_scheduler
[2017-05-04 16:34:06,923: INFO/job_scheduler] Cancelling outstanding jobs...
[2017-05-04 16:34:06,926: ERROR/chroma_service] Rogue thread still running, exiting hard
[2017-05-04 16:34:06,926: ERROR/chroma_service] ['Thread-11980', 'Thread-11029', 'Thread-11996', 'Thread-11997', 'Thread-11026', 'Thread-11993', 'MainThread', 'Thread-11021']
[2017-05-04 16:35:56,849: INFO/queue] Purged 0 messages from 'job_scheduler_notifications' queue
[2017-05-04 16:35:56,946: INFO/queue] Purged 1 messages from 'agent_action_runner_rx' queue
[2017-05-10 10:11:04,214: INFO/chroma_service] Stopping job_scheduler
[2017-05-10 10:11:04,214: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-05-10 10:11:04,214: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-05-10 10:11:04,215: INFO/queue] Stopping ServiceQueue agent_action_runner_rx
[2017-05-10 10:11:04,217: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-05-10 10:11:04,217: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-05-10 10:11:04,217: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-05-10 10:11:04,217: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-05-10 10:11:04,218: INFO/job_scheduler] Stopping...
[2017-05-10 10:11:04,218: INFO/queue] Stopping ServiceQueue job_scheduler_notifications
[2017-05-10 10:11:04,218: INFO/job_scheduler] Joining...
[2017-05-10 10:11:05,219: INFO/job_scheduler] Complete.
[2017-05-10 10:11:05,219: INFO/chroma_service] Joining job_scheduler
[2017-05-10 10:11:05,219: INFO/job_scheduler] Cancelling outstanding jobs...
[2017-05-10 10:11:05,222: ERROR/chroma_service] Rogue thread still running, exiting hard
[2017-05-10 10:11:05,223: ERROR/chroma_service] ['Thread-18842', 'Thread-19089', 'Thread-18844', 'Thread-18938', 'Thread-18937', 'Thread-18845', 'Thread-19093', 'Thread-19096', 'Thread-19097', 'Thread-18846', 'Thread-19120', 'Thread-18843', 'MainThread', 'Thread-19116', 'Thread-19098']
[2017-05-10 10:14:19,615: INFO/queue] Purged 0 messages from 'job_scheduler_notifications' queue
[2017-05-10 10:14:19,639: INFO/queue] Purged 0 messages from 'agent_action_runner_rx' queue
[2017-05-10 10:15:13,381: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:15:13,381: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 10:15:13,396: INFO/command_plan] add_jobs: created Job 1 (Test for host connectivity)
[2017-05-10 10:15:13,423: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 10:15:13,431: INFO/job_scheduler] Job 1: RunJobThread.run
[2017-05-10 10:15:13,514: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:15:13,514: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 10:15:13,530: INFO/command_plan] add_jobs: created Job 2 (Test for host connectivity)
[2017-05-10 10:15:13,569: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 10:15:13,571: INFO/job_scheduler] Job 2: RunJobThread.run
[2017-05-10 10:15:13,617: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:15:13,618: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 10:15:13,635: INFO/command_plan] add_jobs: created Job 3 (Test for host connectivity)
[2017-05-10 10:15:13,673: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 10:15:13,680: INFO/job_scheduler] Job 3: RunJobThread.run
[2017-05-10 10:15:13,706: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:15:13,708: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 10:15:13,743: INFO/command_plan] add_jobs: created Job 4 (Test for host connectivity)
[2017-05-10 10:15:13,795: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 10:15:13,798: INFO/job_scheduler] Job 4: RunJobThread.run
[2017-05-10 10:15:20,222: INFO/job_scheduler] Job 1 finished 1 steps successfully
[2017-05-10 10:15:20,223: INFO/job_scheduler] Job 4 finished 1 steps successfully
[2017-05-10 10:15:20,253: INFO/job_scheduler] Job 3 finished 1 steps successfully
[2017-05-10 10:15:20,300: INFO/job_scheduler] Job 1: complete_job: Updating cache
[2017-05-10 10:15:20,300: INFO/job_scheduler] Job 1 completing (errored=False, cancelled=False)
[2017-05-10 10:15:20,342: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 10:15:20,343: INFO/job_scheduler] Job 4: complete_job: Updating cache
[2017-05-10 10:15:20,343: INFO/job_scheduler] Job 4 completing (errored=False, cancelled=False)
[2017-05-10 10:15:20,375: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-05-10 10:15:20,383: INFO/job_scheduler] Job 3: complete_job: Updating cache
[2017-05-10 10:15:20,384: INFO/job_scheduler] Job 3 completing (errored=False, cancelled=False)
[2017-05-10 10:15:20,417: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 10:15:20,520: INFO/job_scheduler] Job 2 finished 1 steps successfully
[2017-05-10 10:15:20,542: INFO/job_scheduler] Job 2: complete_job: Updating cache
[2017-05-10 10:15:20,542: INFO/job_scheduler] Job 2 completing (errored=False, cancelled=False)
[2017-05-10 10:15:20,567: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:15:24,304: INFO/job_scheduler] Getting FQDN for 'lotus-32vm5': lotus-32vm5
lotus-32vm5.lotus.hpdd.lab.intel.com

[2017-05-10 10:15:24,350: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:15:24,354: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-1 transitioning from undeployed to unconfigured
[2017-05-10 10:15:24,397: INFO/command_plan] Created command 5 (Deploying agent to host lotus-32vm5.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-05-10 10:15:24,434: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 10:15:24,544: INFO/job_scheduler] Job 5: RunJobThread.run
[2017-05-10 10:15:25,022: INFO/job_scheduler] Getting FQDN for 'lotus-32vm7.lotus.hpdd.lab.intel.com': lotus-32vm7
lotus-32vm7.lotus.hpdd.lab.intel.com

[2017-05-10 10:15:25,057: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:15:25,059: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-2 transitioning from undeployed to unconfigured
[2017-05-10 10:15:25,077: INFO/command_plan] Created command 6 (Deploying agent to host lotus-32vm7.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-05-10 10:15:25,131: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 10:15:25,168: INFO/job_scheduler] Job 6: RunJobThread.run
[2017-05-10 10:15:25,596: INFO/job_scheduler] Getting FQDN for 'lotus-32vm8.lotus.hpdd.lab.intel.com': lotus-32vm8
lotus-32vm8.lotus.hpdd.lab.intel.com

[2017-05-10 10:15:25,626: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:15:25,631: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-3 transitioning from undeployed to unconfigured
[2017-05-10 10:15:25,650: INFO/command_plan] Created command 7 (Deploying agent to host lotus-32vm8.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-05-10 10:15:25,698: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 10:15:25,968: INFO/job_scheduler] Job 7: RunJobThread.run
[2017-05-10 10:15:26,468: INFO/job_scheduler] Getting FQDN for 'lotus-32vm6.lotus.hpdd.lab.intel.com': lotus-32vm6
lotus-32vm6.lotus.hpdd.lab.intel.com

[2017-05-10 10:15:26,508: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:15:26,511: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-4 transitioning from undeployed to unconfigured
[2017-05-10 10:15:26,532: INFO/command_plan] Created command 8 (Deploying agent to host lotus-32vm6.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-05-10 10:15:26,574: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 10:15:26,709: INFO/job_scheduler] Job 8: RunJobThread.run
[2017-05-10 10:15:26,754: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 10:15:26,783: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 10:15:26,784: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 10:15:27,569: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 10:15:33,399: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:15:33,399: INFO/job_scheduler]   Job 5 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-10 10:15:33,399: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 10:15:33,512: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-10 10:15:33,512: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-10 10:15:34,513: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 359 seconds remain
[2017-05-10 10:15:34,513: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 359 seconds remain
[2017-05-10 10:15:35,507: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:15:35,507: INFO/job_scheduler]   Job 7 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-10 10:15:35,507: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 10:15:35,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 358 seconds remain
[2017-05-10 10:15:35,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 358 seconds remain
[2017-05-10 10:15:35,610: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-10 10:15:35,610: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-10 10:15:36,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 357 seconds remain
[2017-05-10 10:15:36,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 357 seconds remain
[2017-05-10 10:15:36,611: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 359 seconds remain
[2017-05-10 10:15:36,611: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 359 seconds remain
[2017-05-10 10:15:36,630: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:15:36,630: INFO/job_scheduler]   Job 6 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-10 10:15:36,630: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 10:15:36,721: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-10 10:15:36,721: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-10 10:15:37,242: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:15:37,242: INFO/job_scheduler]   Job 8 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-10 10:15:37,242: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 10:15:37,340: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-10 10:15:37,340: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-10 10:15:37,515: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 356 seconds remain
[2017-05-10 10:15:37,515: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 356 seconds remain
[2017-05-10 10:15:37,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 358 seconds remain
[2017-05-10 10:15:37,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 358 seconds remain
[2017-05-10 10:15:37,723: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 359 seconds remain
[2017-05-10 10:15:37,723: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 359 seconds remain
[2017-05-10 10:15:38,340: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 359 seconds remain
[2017-05-10 10:15:38,340: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 359 seconds remain
[2017-05-10 10:15:38,516: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 355 seconds remain
[2017-05-10 10:15:38,516: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 355 seconds remain
[2017-05-10 10:15:38,613: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 357 seconds remain
[2017-05-10 10:15:38,613: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 357 seconds remain
[2017-05-10 10:15:38,724: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 358 seconds remain
[2017-05-10 10:15:38,724: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 358 seconds remain
[2017-05-10 10:15:39,342: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 358 seconds remain
[2017-05-10 10:15:39,342: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 358 seconds remain
[2017-05-10 10:15:39,518: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 354 seconds remain
[2017-05-10 10:15:39,518: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 354 seconds remain
[2017-05-10 10:15:39,615: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 356 seconds remain
[2017-05-10 10:15:39,615: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 356 seconds remain
[2017-05-10 10:15:39,725: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 357 seconds remain
[2017-05-10 10:15:39,725: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 357 seconds remain
[2017-05-10 10:15:40,343: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 357 seconds remain
[2017-05-10 10:15:40,343: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 357 seconds remain
[2017-05-10 10:15:40,519: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 353 seconds remain
[2017-05-10 10:15:40,519: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 353 seconds remain
[2017-05-10 10:15:40,616: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 355 seconds remain
[2017-05-10 10:15:40,616: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 355 seconds remain
[2017-05-10 10:15:40,726: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 356 seconds remain
[2017-05-10 10:15:40,726: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 356 seconds remain
[2017-05-10 10:15:41,345: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 356 seconds remain
[2017-05-10 10:15:41,345: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 356 seconds remain
[2017-05-10 10:15:41,521: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 352 seconds remain
[2017-05-10 10:15:41,521: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 352 seconds remain
[2017-05-10 10:15:41,618: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 354 seconds remain
[2017-05-10 10:15:41,618: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 354 seconds remain
[2017-05-10 10:15:41,728: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 355 seconds remain
[2017-05-10 10:15:41,728: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 355 seconds remain
[2017-05-10 10:15:42,345: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 355 seconds remain
[2017-05-10 10:15:42,345: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 355 seconds remain
[2017-05-10 10:15:42,522: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 351 seconds remain
[2017-05-10 10:15:42,522: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 351 seconds remain
[2017-05-10 10:15:42,619: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 353 seconds remain
[2017-05-10 10:15:42,619: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 353 seconds remain
[2017-05-10 10:15:42,729: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 354 seconds remain
[2017-05-10 10:15:42,729: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 354 seconds remain
[2017-05-10 10:15:43,347: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 354 seconds remain
[2017-05-10 10:15:43,347: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 354 seconds remain
[2017-05-10 10:15:43,524: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 350 seconds remain
[2017-05-10 10:15:43,524: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 350 seconds remain
[2017-05-10 10:15:43,620: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 352 seconds remain
[2017-05-10 10:15:43,620: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 352 seconds remain
[2017-05-10 10:15:43,731: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 353 seconds remain
[2017-05-10 10:15:43,731: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 353 seconds remain
[2017-05-10 10:15:44,223: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:15:44,223: INFO/job_scheduler]   Job 5 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-10 10:15:44,224: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 10:15:44,349: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 353 seconds remain
[2017-05-10 10:15:44,349: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 353 seconds remain
[2017-05-10 10:15:44,400: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:15:44,401: INFO/job_scheduler]   Job 5 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-10 10:15:44,401: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 10:15:44,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 349 seconds remain
[2017-05-10 10:15:44,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 349 seconds remain
[2017-05-10 10:15:44,621: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 351 seconds remain
[2017-05-10 10:15:44,621: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 351 seconds remain
[2017-05-10 10:15:44,707: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:15:44,707: INFO/job_scheduler]   Job 7 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-10 10:15:44,707: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 10:15:44,732: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 352 seconds remain
[2017-05-10 10:15:44,732: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 352 seconds remain
[2017-05-10 10:15:44,826: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:15:44,827: INFO/job_scheduler]   Job 7 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-10 10:15:44,827: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 10:15:45,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 352 seconds remain
[2017-05-10 10:15:45,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 352 seconds remain
[2017-05-10 10:15:45,390: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:15:45,390: INFO/job_scheduler]   Job 6 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-10 10:15:45,390: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 10:15:45,496: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:15:45,497: INFO/job_scheduler]   Job 6 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-10 10:15:45,497: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 10:15:45,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 348 seconds remain
[2017-05-10 10:15:45,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 348 seconds remain
[2017-05-10 10:15:45,623: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 350 seconds remain
[2017-05-10 10:15:45,623: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 350 seconds remain
[2017-05-10 10:15:45,734: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 351 seconds remain
[2017-05-10 10:15:45,734: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 351 seconds remain
[2017-05-10 10:15:46,351: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 351 seconds remain
[2017-05-10 10:15:46,351: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 351 seconds remain
[2017-05-10 10:15:46,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 347 seconds remain
[2017-05-10 10:15:46,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 347 seconds remain
[2017-05-10 10:15:46,624: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 349 seconds remain
[2017-05-10 10:15:46,624: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 349 seconds remain
[2017-05-10 10:15:46,735: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 350 seconds remain
[2017-05-10 10:15:46,735: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 350 seconds remain
[2017-05-10 10:15:46,800: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:15:46,800: INFO/job_scheduler]   Job 8 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-10 10:15:46,800: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 10:15:46,911: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:15:46,911: INFO/job_scheduler]   Job 8 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-10 10:15:46,912: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 10:15:47,353: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 350 seconds remain
[2017-05-10 10:15:47,353: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 350 seconds remain
[2017-05-10 10:15:47,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 346 seconds remain
[2017-05-10 10:15:47,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 346 seconds remain
[2017-05-10 10:15:47,626: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 348 seconds remain
[2017-05-10 10:15:47,626: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 348 seconds remain
[2017-05-10 10:15:47,737: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 349 seconds remain
[2017-05-10 10:15:47,737: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 349 seconds remain
[2017-05-10 10:15:48,354: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 349 seconds remain
[2017-05-10 10:15:48,354: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 349 seconds remain
[2017-05-10 10:15:48,530: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 345 seconds remain
[2017-05-10 10:15:48,530: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 345 seconds remain
[2017-05-10 10:15:48,627: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 347 seconds remain
[2017-05-10 10:15:48,627: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 347 seconds remain
[2017-05-10 10:15:48,738: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 348 seconds remain
[2017-05-10 10:15:48,738: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 348 seconds remain
[2017-05-10 10:15:49,355: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 348 seconds remain
[2017-05-10 10:15:49,355: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 348 seconds remain
[2017-05-10 10:15:49,531: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 344 seconds remain
[2017-05-10 10:15:49,531: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 344 seconds remain
[2017-05-10 10:15:49,628: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 346 seconds remain
[2017-05-10 10:15:49,628: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 346 seconds remain
[2017-05-10 10:15:49,740: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 347 seconds remain
[2017-05-10 10:15:49,740: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 347 seconds remain
[2017-05-10 10:15:50,356: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 347 seconds remain
[2017-05-10 10:15:50,356: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 347 seconds remain
[2017-05-10 10:15:50,532: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 343 seconds remain
[2017-05-10 10:15:50,532: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 343 seconds remain
[2017-05-10 10:15:50,630: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 345 seconds remain
[2017-05-10 10:15:50,630: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 345 seconds remain
[2017-05-10 10:15:50,741: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 346 seconds remain
[2017-05-10 10:15:50,741: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 346 seconds remain
[2017-05-10 10:15:51,358: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 346 seconds remain
[2017-05-10 10:15:51,358: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 346 seconds remain
[2017-05-10 10:15:51,534: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 342 seconds remain
[2017-05-10 10:15:51,534: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 342 seconds remain
[2017-05-10 10:15:51,631: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 344 seconds remain
[2017-05-10 10:15:51,631: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 344 seconds remain
[2017-05-10 10:15:51,742: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 345 seconds remain
[2017-05-10 10:15:51,742: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 345 seconds remain
[2017-05-10 10:15:52,359: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 345 seconds remain
[2017-05-10 10:15:52,359: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 345 seconds remain
[2017-05-10 10:15:52,535: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 341 seconds remain
[2017-05-10 10:15:52,535: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 341 seconds remain
[2017-05-10 10:15:52,632: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 343 seconds remain
[2017-05-10 10:15:52,632: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 343 seconds remain
[2017-05-10 10:15:52,743: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 344 seconds remain
[2017-05-10 10:15:52,743: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 344 seconds remain
[2017-05-10 10:15:53,361: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 344 seconds remain
[2017-05-10 10:15:53,361: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 344 seconds remain
[2017-05-10 10:15:53,536: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 340 seconds remain
[2017-05-10 10:15:53,536: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 340 seconds remain
[2017-05-10 10:15:53,634: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 342 seconds remain
[2017-05-10 10:15:53,634: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 342 seconds remain
[2017-05-10 10:15:53,744: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 343 seconds remain
[2017-05-10 10:15:53,744: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 343 seconds remain
[2017-05-10 10:15:54,362: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 343 seconds remain
[2017-05-10 10:15:54,362: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 343 seconds remain
[2017-05-10 10:15:54,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 339 seconds remain
[2017-05-10 10:15:54,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 339 seconds remain
[2017-05-10 10:15:54,635: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 341 seconds remain
[2017-05-10 10:15:54,635: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 341 seconds remain
[2017-05-10 10:15:54,746: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 342 seconds remain
[2017-05-10 10:15:54,746: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 342 seconds remain
[2017-05-10 10:15:55,364: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 342 seconds remain
[2017-05-10 10:15:55,364: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 342 seconds remain
[2017-05-10 10:15:55,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 338 seconds remain
[2017-05-10 10:15:55,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 338 seconds remain
[2017-05-10 10:15:55,636: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 340 seconds remain
[2017-05-10 10:15:55,636: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 340 seconds remain
[2017-05-10 10:15:55,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 341 seconds remain
[2017-05-10 10:15:55,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 341 seconds remain
[2017-05-10 10:15:56,372: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 341 seconds remain
[2017-05-10 10:15:56,372: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 341 seconds remain
[2017-05-10 10:15:56,540: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 337 seconds remain
[2017-05-10 10:15:56,540: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 337 seconds remain
[2017-05-10 10:15:56,638: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 339 seconds remain
[2017-05-10 10:15:56,638: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 339 seconds remain
[2017-05-10 10:15:56,749: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 340 seconds remain
[2017-05-10 10:15:56,749: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 340 seconds remain
[2017-05-10 10:15:57,374: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 340 seconds remain
[2017-05-10 10:15:57,374: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 340 seconds remain
[2017-05-10 10:15:57,542: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 336 seconds remain
[2017-05-10 10:15:57,542: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 336 seconds remain
[2017-05-10 10:15:57,638: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 338 seconds remain
[2017-05-10 10:15:57,638: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 338 seconds remain
[2017-05-10 10:15:57,750: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 339 seconds remain
[2017-05-10 10:15:57,750: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 339 seconds remain
[2017-05-10 10:15:58,375: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 339 seconds remain
[2017-05-10 10:15:58,375: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 339 seconds remain
[2017-05-10 10:15:58,543: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 335 seconds remain
[2017-05-10 10:15:58,543: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 335 seconds remain
[2017-05-10 10:15:58,640: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 337 seconds remain
[2017-05-10 10:15:58,640: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 337 seconds remain
[2017-05-10 10:15:58,752: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 338 seconds remain
[2017-05-10 10:15:58,752: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 338 seconds remain
[2017-05-10 10:15:59,377: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 338 seconds remain
[2017-05-10 10:15:59,377: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 338 seconds remain
[2017-05-10 10:15:59,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 334 seconds remain
[2017-05-10 10:15:59,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 334 seconds remain
[2017-05-10 10:15:59,640: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 336 seconds remain
[2017-05-10 10:15:59,640: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 336 seconds remain
[2017-05-10 10:15:59,753: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 337 seconds remain
[2017-05-10 10:15:59,753: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 337 seconds remain
[2017-05-10 10:16:00,378: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 337 seconds remain
[2017-05-10 10:16:00,378: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 337 seconds remain
[2017-05-10 10:16:00,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 333 seconds remain
[2017-05-10 10:16:00,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 333 seconds remain
[2017-05-10 10:16:00,642: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 335 seconds remain
[2017-05-10 10:16:00,642: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 335 seconds remain
[2017-05-10 10:16:00,754: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 336 seconds remain
[2017-05-10 10:16:00,754: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 336 seconds remain
[2017-05-10 10:16:01,388: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 336 seconds remain
[2017-05-10 10:16:01,388: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 336 seconds remain
[2017-05-10 10:16:01,547: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 332 seconds remain
[2017-05-10 10:16:01,547: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 332 seconds remain
[2017-05-10 10:16:01,643: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 334 seconds remain
[2017-05-10 10:16:01,643: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 334 seconds remain
[2017-05-10 10:16:01,756: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 335 seconds remain
[2017-05-10 10:16:01,756: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 335 seconds remain
[2017-05-10 10:16:02,390: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 335 seconds remain
[2017-05-10 10:16:02,390: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 335 seconds remain
[2017-05-10 10:16:02,548: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 331 seconds remain
[2017-05-10 10:16:02,548: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 331 seconds remain
[2017-05-10 10:16:02,644: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 333 seconds remain
[2017-05-10 10:16:02,644: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 333 seconds remain
[2017-05-10 10:16:02,757: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 334 seconds remain
[2017-05-10 10:16:02,757: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 334 seconds remain
[2017-05-10 10:16:03,391: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 334 seconds remain
[2017-05-10 10:16:03,391: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 334 seconds remain
[2017-05-10 10:16:03,550: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 330 seconds remain
[2017-05-10 10:16:03,550: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 330 seconds remain
[2017-05-10 10:16:03,645: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 332 seconds remain
[2017-05-10 10:16:03,645: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 332 seconds remain
[2017-05-10 10:16:03,758: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 333 seconds remain
[2017-05-10 10:16:03,758: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 333 seconds remain
[2017-05-10 10:16:04,393: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 333 seconds remain
[2017-05-10 10:16:04,393: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 333 seconds remain
[2017-05-10 10:16:04,551: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 329 seconds remain
[2017-05-10 10:16:04,551: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 329 seconds remain
[2017-05-10 10:16:04,646: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 331 seconds remain
[2017-05-10 10:16:04,646: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 331 seconds remain
[2017-05-10 10:16:04,760: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 332 seconds remain
[2017-05-10 10:16:04,760: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 332 seconds remain
[2017-05-10 10:16:05,394: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 332 seconds remain
[2017-05-10 10:16:05,394: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 332 seconds remain
[2017-05-10 10:16:05,552: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 328 seconds remain
[2017-05-10 10:16:05,552: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 328 seconds remain
[2017-05-10 10:16:05,648: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 330 seconds remain
[2017-05-10 10:16:05,648: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 330 seconds remain
[2017-05-10 10:16:05,761: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 331 seconds remain
[2017-05-10 10:16:05,761: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 331 seconds remain
[2017-05-10 10:16:06,396: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 331 seconds remain
[2017-05-10 10:16:06,396: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 331 seconds remain
[2017-05-10 10:16:06,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 327 seconds remain
[2017-05-10 10:16:06,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 327 seconds remain
[2017-05-10 10:16:06,649: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 329 seconds remain
[2017-05-10 10:16:06,649: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 329 seconds remain
[2017-05-10 10:16:06,762: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 330 seconds remain
[2017-05-10 10:16:06,762: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 330 seconds remain
[2017-05-10 10:16:07,398: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 330 seconds remain
[2017-05-10 10:16:07,398: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 330 seconds remain
[2017-05-10 10:16:07,555: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 326 seconds remain
[2017-05-10 10:16:07,555: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 326 seconds remain
[2017-05-10 10:16:07,650: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 328 seconds remain
[2017-05-10 10:16:07,650: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 328 seconds remain
[2017-05-10 10:16:07,764: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 329 seconds remain
[2017-05-10 10:16:07,764: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 329 seconds remain
[2017-05-10 10:16:08,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 329 seconds remain
[2017-05-10 10:16:08,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 329 seconds remain
[2017-05-10 10:16:08,557: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 325 seconds remain
[2017-05-10 10:16:08,557: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 325 seconds remain
[2017-05-10 10:16:08,652: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 327 seconds remain
[2017-05-10 10:16:08,652: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 327 seconds remain
[2017-05-10 10:16:08,765: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 328 seconds remain
[2017-05-10 10:16:08,765: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 328 seconds remain
[2017-05-10 10:16:09,400: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 328 seconds remain
[2017-05-10 10:16:09,400: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 328 seconds remain
[2017-05-10 10:16:09,558: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 324 seconds remain
[2017-05-10 10:16:09,558: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 324 seconds remain
[2017-05-10 10:16:09,653: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 326 seconds remain
[2017-05-10 10:16:09,653: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 326 seconds remain
[2017-05-10 10:16:09,766: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 327 seconds remain
[2017-05-10 10:16:09,766: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 327 seconds remain
[2017-05-10 10:16:10,402: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 327 seconds remain
[2017-05-10 10:16:10,402: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 327 seconds remain
[2017-05-10 10:16:10,560: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 323 seconds remain
[2017-05-10 10:16:10,560: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 323 seconds remain
[2017-05-10 10:16:10,655: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 325 seconds remain
[2017-05-10 10:16:10,655: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 325 seconds remain
[2017-05-10 10:16:10,768: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 326 seconds remain
[2017-05-10 10:16:10,768: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 326 seconds remain
[2017-05-10 10:16:11,404: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 326 seconds remain
[2017-05-10 10:16:11,404: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 326 seconds remain
[2017-05-10 10:16:11,561: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 322 seconds remain
[2017-05-10 10:16:11,561: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 322 seconds remain
[2017-05-10 10:16:11,710: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 324 seconds remain
[2017-05-10 10:16:11,710: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 324 seconds remain
[2017-05-10 10:16:11,769: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 325 seconds remain
[2017-05-10 10:16:11,769: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 325 seconds remain
[2017-05-10 10:16:12,405: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 325 seconds remain
[2017-05-10 10:16:12,405: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 325 seconds remain
[2017-05-10 10:16:12,562: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 321 seconds remain
[2017-05-10 10:16:12,562: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 321 seconds remain
[2017-05-10 10:16:12,712: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 323 seconds remain
[2017-05-10 10:16:12,712: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 323 seconds remain
[2017-05-10 10:16:12,771: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 324 seconds remain
[2017-05-10 10:16:12,771: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 324 seconds remain
[2017-05-10 10:16:13,407: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 324 seconds remain
[2017-05-10 10:16:13,407: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 324 seconds remain
[2017-05-10 10:16:13,564: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 320 seconds remain
[2017-05-10 10:16:13,564: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 320 seconds remain
[2017-05-10 10:16:13,713: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 322 seconds remain
[2017-05-10 10:16:13,713: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 322 seconds remain
[2017-05-10 10:16:13,772: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 323 seconds remain
[2017-05-10 10:16:13,772: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 323 seconds remain
[2017-05-10 10:16:14,408: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 323 seconds remain
[2017-05-10 10:16:14,408: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 323 seconds remain
[2017-05-10 10:16:14,564: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 319 seconds remain
[2017-05-10 10:16:14,564: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 319 seconds remain
[2017-05-10 10:16:14,715: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 321 seconds remain
[2017-05-10 10:16:14,715: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 321 seconds remain
[2017-05-10 10:16:14,773: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 322 seconds remain
[2017-05-10 10:16:14,773: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 322 seconds remain
[2017-05-10 10:16:15,409: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 322 seconds remain
[2017-05-10 10:16:15,409: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 322 seconds remain
[2017-05-10 10:16:15,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 318 seconds remain
[2017-05-10 10:16:15,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 318 seconds remain
[2017-05-10 10:16:15,716: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 320 seconds remain
[2017-05-10 10:16:15,716: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 320 seconds remain
[2017-05-10 10:16:15,775: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 321 seconds remain
[2017-05-10 10:16:15,775: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 321 seconds remain
[2017-05-10 10:16:16,411: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 321 seconds remain
[2017-05-10 10:16:16,411: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 321 seconds remain
[2017-05-10 10:16:16,567: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 317 seconds remain
[2017-05-10 10:16:16,567: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 317 seconds remain
[2017-05-10 10:16:16,725: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 319 seconds remain
[2017-05-10 10:16:16,725: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 319 seconds remain
[2017-05-10 10:16:16,776: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 320 seconds remain
[2017-05-10 10:16:16,776: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 320 seconds remain
[2017-05-10 10:16:17,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 320 seconds remain
[2017-05-10 10:16:17,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 320 seconds remain
[2017-05-10 10:16:17,568: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 316 seconds remain
[2017-05-10 10:16:17,568: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 316 seconds remain
[2017-05-10 10:16:17,726: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 318 seconds remain
[2017-05-10 10:16:17,726: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 318 seconds remain
[2017-05-10 10:16:17,777: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 319 seconds remain
[2017-05-10 10:16:17,777: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 319 seconds remain
[2017-05-10 10:16:18,414: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 319 seconds remain
[2017-05-10 10:16:18,414: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 319 seconds remain
[2017-05-10 10:16:18,570: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 315 seconds remain
[2017-05-10 10:16:18,570: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 315 seconds remain
[2017-05-10 10:16:18,728: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 317 seconds remain
[2017-05-10 10:16:18,728: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 317 seconds remain
[2017-05-10 10:16:18,779: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 318 seconds remain
[2017-05-10 10:16:18,779: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 318 seconds remain
[2017-05-10 10:16:19,415: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 318 seconds remain
[2017-05-10 10:16:19,415: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 318 seconds remain
[2017-05-10 10:16:19,571: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 314 seconds remain
[2017-05-10 10:16:19,571: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 314 seconds remain
[2017-05-10 10:16:19,729: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 316 seconds remain
[2017-05-10 10:16:19,729: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 316 seconds remain
[2017-05-10 10:16:19,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 317 seconds remain
[2017-05-10 10:16:19,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 317 seconds remain
[2017-05-10 10:16:20,417: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 317 seconds remain
[2017-05-10 10:16:20,417: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 317 seconds remain
[2017-05-10 10:16:20,572: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 313 seconds remain
[2017-05-10 10:16:20,572: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 313 seconds remain
[2017-05-10 10:16:20,730: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 315 seconds remain
[2017-05-10 10:16:20,730: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 315 seconds remain
[2017-05-10 10:16:20,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 316 seconds remain
[2017-05-10 10:16:20,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 316 seconds remain
[2017-05-10 10:16:21,418: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 316 seconds remain
[2017-05-10 10:16:21,418: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 316 seconds remain
[2017-05-10 10:16:21,573: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 312 seconds remain
[2017-05-10 10:16:21,573: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 312 seconds remain
[2017-05-10 10:16:21,741: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 314 seconds remain
[2017-05-10 10:16:21,741: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 314 seconds remain
[2017-05-10 10:16:21,783: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 315 seconds remain
[2017-05-10 10:16:21,783: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 315 seconds remain
[2017-05-10 10:16:22,420: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 315 seconds remain
[2017-05-10 10:16:22,420: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 315 seconds remain
[2017-05-10 10:16:22,574: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 311 seconds remain
[2017-05-10 10:16:22,574: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 311 seconds remain
[2017-05-10 10:16:22,743: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 313 seconds remain
[2017-05-10 10:16:22,743: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 313 seconds remain
[2017-05-10 10:16:22,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 314 seconds remain
[2017-05-10 10:16:22,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 314 seconds remain
[2017-05-10 10:16:23,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 314 seconds remain
[2017-05-10 10:16:23,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 314 seconds remain
[2017-05-10 10:16:23,575: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 310 seconds remain
[2017-05-10 10:16:23,575: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 310 seconds remain
[2017-05-10 10:16:23,744: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 312 seconds remain
[2017-05-10 10:16:23,744: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 312 seconds remain
[2017-05-10 10:16:23,785: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 313 seconds remain
[2017-05-10 10:16:23,785: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 313 seconds remain
[2017-05-10 10:16:24,422: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 313 seconds remain
[2017-05-10 10:16:24,422: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 313 seconds remain
[2017-05-10 10:16:24,576: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 309 seconds remain
[2017-05-10 10:16:24,576: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 309 seconds remain
[2017-05-10 10:16:24,746: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 311 seconds remain
[2017-05-10 10:16:24,746: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 311 seconds remain
[2017-05-10 10:16:24,787: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 312 seconds remain
[2017-05-10 10:16:24,787: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 312 seconds remain
[2017-05-10 10:16:25,423: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 312 seconds remain
[2017-05-10 10:16:25,423: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 312 seconds remain
[2017-05-10 10:16:25,577: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 308 seconds remain
[2017-05-10 10:16:25,577: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 308 seconds remain
[2017-05-10 10:16:25,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 310 seconds remain
[2017-05-10 10:16:25,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 310 seconds remain
[2017-05-10 10:16:25,788: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 311 seconds remain
[2017-05-10 10:16:25,788: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 311 seconds remain
[2017-05-10 10:16:26,424: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 311 seconds remain
[2017-05-10 10:16:26,424: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 311 seconds remain
[2017-05-10 10:16:26,578: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 307 seconds remain
[2017-05-10 10:16:26,578: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 307 seconds remain
[2017-05-10 10:16:26,748: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 309 seconds remain
[2017-05-10 10:16:26,748: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 309 seconds remain
[2017-05-10 10:16:26,790: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 310 seconds remain
[2017-05-10 10:16:26,790: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 310 seconds remain
[2017-05-10 10:16:27,425: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 310 seconds remain
[2017-05-10 10:16:27,425: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 310 seconds remain
[2017-05-10 10:16:27,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 306 seconds remain
[2017-05-10 10:16:27,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 306 seconds remain
[2017-05-10 10:16:27,751: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 308 seconds remain
[2017-05-10 10:16:27,751: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 308 seconds remain
[2017-05-10 10:16:27,792: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 309 seconds remain
[2017-05-10 10:16:27,792: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 309 seconds remain
[2017-05-10 10:16:28,426: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 309 seconds remain
[2017-05-10 10:16:28,426: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 309 seconds remain
[2017-05-10 10:16:28,582: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 305 seconds remain
[2017-05-10 10:16:28,582: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 305 seconds remain
[2017-05-10 10:16:28,752: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 307 seconds remain
[2017-05-10 10:16:28,752: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 307 seconds remain
[2017-05-10 10:16:28,793: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 308 seconds remain
[2017-05-10 10:16:28,793: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 308 seconds remain
[2017-05-10 10:16:29,429: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 308 seconds remain
[2017-05-10 10:16:29,429: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 308 seconds remain
[2017-05-10 10:16:29,583: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 304 seconds remain
[2017-05-10 10:16:29,583: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 304 seconds remain
[2017-05-10 10:16:29,753: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 306 seconds remain
[2017-05-10 10:16:29,753: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 306 seconds remain
[2017-05-10 10:16:29,795: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 307 seconds remain
[2017-05-10 10:16:29,795: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 307 seconds remain
[2017-05-10 10:16:30,431: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 307 seconds remain
[2017-05-10 10:16:30,431: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 307 seconds remain
[2017-05-10 10:16:30,584: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 303 seconds remain
[2017-05-10 10:16:30,584: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 303 seconds remain
[2017-05-10 10:16:30,755: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 305 seconds remain
[2017-05-10 10:16:30,755: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 305 seconds remain
[2017-05-10 10:16:30,796: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 306 seconds remain
[2017-05-10 10:16:30,796: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 306 seconds remain
[2017-05-10 10:16:31,432: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 306 seconds remain
[2017-05-10 10:16:31,432: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 306 seconds remain
[2017-05-10 10:16:31,586: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 302 seconds remain
[2017-05-10 10:16:31,586: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 302 seconds remain
[2017-05-10 10:16:31,756: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 304 seconds remain
[2017-05-10 10:16:31,756: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 304 seconds remain
[2017-05-10 10:16:31,797: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 305 seconds remain
[2017-05-10 10:16:31,797: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 305 seconds remain
[2017-05-10 10:16:32,434: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 305 seconds remain
[2017-05-10 10:16:32,434: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 305 seconds remain
[2017-05-10 10:16:32,587: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 301 seconds remain
[2017-05-10 10:16:32,587: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 301 seconds remain
[2017-05-10 10:16:32,758: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 303 seconds remain
[2017-05-10 10:16:32,758: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 303 seconds remain
[2017-05-10 10:16:32,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 304 seconds remain
[2017-05-10 10:16:32,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 304 seconds remain
[2017-05-10 10:16:33,434: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 304 seconds remain
[2017-05-10 10:16:33,434: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 304 seconds remain
[2017-05-10 10:16:33,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 300 seconds remain
[2017-05-10 10:16:33,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 300 seconds remain
[2017-05-10 10:16:33,758: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 302 seconds remain
[2017-05-10 10:16:33,758: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 302 seconds remain
[2017-05-10 10:16:33,799: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 303 seconds remain
[2017-05-10 10:16:33,799: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 303 seconds remain
[2017-05-10 10:16:34,436: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 303 seconds remain
[2017-05-10 10:16:34,436: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 303 seconds remain
[2017-05-10 10:16:34,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 299 seconds remain
[2017-05-10 10:16:34,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 299 seconds remain
[2017-05-10 10:16:34,760: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 301 seconds remain
[2017-05-10 10:16:34,760: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 301 seconds remain
[2017-05-10 10:16:34,801: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 302 seconds remain
[2017-05-10 10:16:34,801: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 302 seconds remain
[2017-05-10 10:16:35,437: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 302 seconds remain
[2017-05-10 10:16:35,437: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 302 seconds remain
[2017-05-10 10:16:35,591: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 298 seconds remain
[2017-05-10 10:16:35,591: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 298 seconds remain
[2017-05-10 10:16:35,761: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 300 seconds remain
[2017-05-10 10:16:35,761: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 300 seconds remain
[2017-05-10 10:16:35,802: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 301 seconds remain
[2017-05-10 10:16:35,802: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 301 seconds remain
[2017-05-10 10:16:36,438: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 301 seconds remain
[2017-05-10 10:16:36,438: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 301 seconds remain
[2017-05-10 10:16:36,592: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 297 seconds remain
[2017-05-10 10:16:36,592: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 297 seconds remain
[2017-05-10 10:16:36,762: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 299 seconds remain
[2017-05-10 10:16:36,762: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 299 seconds remain
[2017-05-10 10:16:36,813: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 300 seconds remain
[2017-05-10 10:16:36,813: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 300 seconds remain
[2017-05-10 10:16:37,440: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 300 seconds remain
[2017-05-10 10:16:37,440: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 300 seconds remain
[2017-05-10 10:16:37,593: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 296 seconds remain
[2017-05-10 10:16:37,593: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 296 seconds remain
[2017-05-10 10:16:37,763: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 298 seconds remain
[2017-05-10 10:16:37,763: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 298 seconds remain
[2017-05-10 10:16:37,815: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 299 seconds remain
[2017-05-10 10:16:37,815: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 299 seconds remain
[2017-05-10 10:16:38,443: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 299 seconds remain
[2017-05-10 10:16:38,443: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 299 seconds remain
[2017-05-10 10:16:38,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 295 seconds remain
[2017-05-10 10:16:38,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 295 seconds remain
[2017-05-10 10:16:38,765: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 297 seconds remain
[2017-05-10 10:16:38,765: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 297 seconds remain
[2017-05-10 10:16:38,816: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 298 seconds remain
[2017-05-10 10:16:38,816: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 298 seconds remain
[2017-05-10 10:16:39,445: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 298 seconds remain
[2017-05-10 10:16:39,445: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 298 seconds remain
[2017-05-10 10:16:39,596: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 294 seconds remain
[2017-05-10 10:16:39,596: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 294 seconds remain
[2017-05-10 10:16:39,766: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 296 seconds remain
[2017-05-10 10:16:39,766: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 296 seconds remain
[2017-05-10 10:16:39,817: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 297 seconds remain
[2017-05-10 10:16:39,817: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 297 seconds remain
[2017-05-10 10:16:40,446: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 297 seconds remain
[2017-05-10 10:16:40,446: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 297 seconds remain
[2017-05-10 10:16:40,597: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 293 seconds remain
[2017-05-10 10:16:40,597: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 293 seconds remain
[2017-05-10 10:16:40,767: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 295 seconds remain
[2017-05-10 10:16:40,767: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 295 seconds remain
[2017-05-10 10:16:40,818: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 296 seconds remain
[2017-05-10 10:16:40,818: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 296 seconds remain
[2017-05-10 10:16:41,448: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 296 seconds remain
[2017-05-10 10:16:41,448: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 296 seconds remain
[2017-05-10 10:16:41,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 292 seconds remain
[2017-05-10 10:16:41,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 292 seconds remain
[2017-05-10 10:16:41,769: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 294 seconds remain
[2017-05-10 10:16:41,769: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 294 seconds remain
[2017-05-10 10:16:41,829: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 295 seconds remain
[2017-05-10 10:16:41,829: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 295 seconds remain
[2017-05-10 10:16:42,449: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 295 seconds remain
[2017-05-10 10:16:42,449: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 295 seconds remain
[2017-05-10 10:16:42,600: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 291 seconds remain
[2017-05-10 10:16:42,600: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 291 seconds remain
[2017-05-10 10:16:42,770: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 293 seconds remain
[2017-05-10 10:16:42,770: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 293 seconds remain
[2017-05-10 10:16:42,830: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 294 seconds remain
[2017-05-10 10:16:42,830: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 294 seconds remain
[2017-05-10 10:16:43,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 294 seconds remain
[2017-05-10 10:16:43,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 294 seconds remain
[2017-05-10 10:16:43,602: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 290 seconds remain
[2017-05-10 10:16:43,602: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 290 seconds remain
[2017-05-10 10:16:43,771: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 292 seconds remain
[2017-05-10 10:16:43,771: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 292 seconds remain
[2017-05-10 10:16:43,832: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 293 seconds remain
[2017-05-10 10:16:43,832: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 293 seconds remain
[2017-05-10 10:16:44,453: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 293 seconds remain
[2017-05-10 10:16:44,453: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 293 seconds remain
[2017-05-10 10:16:44,603: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 289 seconds remain
[2017-05-10 10:16:44,603: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 289 seconds remain
[2017-05-10 10:16:44,773: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 291 seconds remain
[2017-05-10 10:16:44,773: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 291 seconds remain
[2017-05-10 10:16:44,833: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 292 seconds remain
[2017-05-10 10:16:44,833: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 292 seconds remain
[2017-05-10 10:16:45,454: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 292 seconds remain
[2017-05-10 10:16:45,454: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 292 seconds remain
[2017-05-10 10:16:45,604: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 288 seconds remain
[2017-05-10 10:16:45,604: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 288 seconds remain
[2017-05-10 10:16:45,774: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 290 seconds remain
[2017-05-10 10:16:45,774: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 290 seconds remain
[2017-05-10 10:16:45,834: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 291 seconds remain
[2017-05-10 10:16:45,834: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 291 seconds remain
[2017-05-10 10:16:46,456: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 291 seconds remain
[2017-05-10 10:16:46,456: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 291 seconds remain
[2017-05-10 10:16:46,606: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 287 seconds remain
[2017-05-10 10:16:46,606: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 287 seconds remain
[2017-05-10 10:16:46,775: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 289 seconds remain
[2017-05-10 10:16:46,775: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 289 seconds remain
[2017-05-10 10:16:46,836: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 290 seconds remain
[2017-05-10 10:16:46,836: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 290 seconds remain
[2017-05-10 10:16:47,456: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 290 seconds remain
[2017-05-10 10:16:47,456: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 290 seconds remain
[2017-05-10 10:16:47,607: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 286 seconds remain
[2017-05-10 10:16:47,607: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 286 seconds remain
[2017-05-10 10:16:47,777: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 288 seconds remain
[2017-05-10 10:16:47,777: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 288 seconds remain
[2017-05-10 10:16:47,837: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 289 seconds remain
[2017-05-10 10:16:47,837: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 289 seconds remain
[2017-05-10 10:16:48,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 289 seconds remain
[2017-05-10 10:16:48,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 289 seconds remain
[2017-05-10 10:16:48,609: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 285 seconds remain
[2017-05-10 10:16:48,609: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 285 seconds remain
[2017-05-10 10:16:48,778: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 287 seconds remain
[2017-05-10 10:16:48,778: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 287 seconds remain
[2017-05-10 10:16:48,839: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 288 seconds remain
[2017-05-10 10:16:48,839: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 288 seconds remain
[2017-05-10 10:16:49,459: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 288 seconds remain
[2017-05-10 10:16:49,459: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 288 seconds remain
[2017-05-10 10:16:49,610: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 284 seconds remain
[2017-05-10 10:16:49,610: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 284 seconds remain
[2017-05-10 10:16:49,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 286 seconds remain
[2017-05-10 10:16:49,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 286 seconds remain
[2017-05-10 10:16:49,840: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 287 seconds remain
[2017-05-10 10:16:49,840: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 287 seconds remain
[2017-05-10 10:16:50,461: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 287 seconds remain
[2017-05-10 10:16:50,461: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 287 seconds remain
[2017-05-10 10:16:50,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 283 seconds remain
[2017-05-10 10:16:50,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 283 seconds remain
[2017-05-10 10:16:50,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 285 seconds remain
[2017-05-10 10:16:50,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 285 seconds remain
[2017-05-10 10:16:50,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 286 seconds remain
[2017-05-10 10:16:50,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 286 seconds remain
[2017-05-10 10:16:51,463: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 286 seconds remain
[2017-05-10 10:16:51,463: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 286 seconds remain
[2017-05-10 10:16:51,613: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 282 seconds remain
[2017-05-10 10:16:51,613: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 282 seconds remain
[2017-05-10 10:16:51,782: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 284 seconds remain
[2017-05-10 10:16:51,782: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 284 seconds remain
[2017-05-10 10:16:51,843: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 285 seconds remain
[2017-05-10 10:16:51,843: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 285 seconds remain
[2017-05-10 10:16:52,464: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 285 seconds remain
[2017-05-10 10:16:52,464: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 285 seconds remain
[2017-05-10 10:16:52,614: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 281 seconds remain
[2017-05-10 10:16:52,614: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 281 seconds remain
[2017-05-10 10:16:52,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 283 seconds remain
[2017-05-10 10:16:52,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 283 seconds remain
[2017-05-10 10:16:52,843: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 284 seconds remain
[2017-05-10 10:16:52,843: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 284 seconds remain
[2017-05-10 10:16:53,466: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 284 seconds remain
[2017-05-10 10:16:53,466: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 284 seconds remain
[2017-05-10 10:16:53,616: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 280 seconds remain
[2017-05-10 10:16:53,616: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 280 seconds remain
[2017-05-10 10:16:53,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 282 seconds remain
[2017-05-10 10:16:53,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 282 seconds remain
[2017-05-10 10:16:53,845: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 283 seconds remain
[2017-05-10 10:16:53,845: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 283 seconds remain
[2017-05-10 10:16:54,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 283 seconds remain
[2017-05-10 10:16:54,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 283 seconds remain
[2017-05-10 10:16:54,617: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 279 seconds remain
[2017-05-10 10:16:54,617: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 279 seconds remain
[2017-05-10 10:16:54,786: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 281 seconds remain
[2017-05-10 10:16:54,786: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 281 seconds remain
[2017-05-10 10:16:54,846: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 282 seconds remain
[2017-05-10 10:16:54,846: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 282 seconds remain
[2017-05-10 10:16:55,469: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 282 seconds remain
[2017-05-10 10:16:55,469: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 282 seconds remain
[2017-05-10 10:16:55,619: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 278 seconds remain
[2017-05-10 10:16:55,619: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 278 seconds remain
[2017-05-10 10:16:55,787: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 280 seconds remain
[2017-05-10 10:16:55,787: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 280 seconds remain
[2017-05-10 10:16:55,847: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 281 seconds remain
[2017-05-10 10:16:55,847: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 281 seconds remain
[2017-05-10 10:16:56,470: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 281 seconds remain
[2017-05-10 10:16:56,470: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 281 seconds remain
[2017-05-10 10:16:56,620: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 277 seconds remain
[2017-05-10 10:16:56,620: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 277 seconds remain
[2017-05-10 10:16:56,788: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 279 seconds remain
[2017-05-10 10:16:56,788: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 279 seconds remain
[2017-05-10 10:16:56,849: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 280 seconds remain
[2017-05-10 10:16:56,849: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 280 seconds remain
[2017-05-10 10:16:57,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 280 seconds remain
[2017-05-10 10:16:57,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 280 seconds remain
[2017-05-10 10:16:57,622: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 276 seconds remain
[2017-05-10 10:16:57,622: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 276 seconds remain
[2017-05-10 10:16:57,790: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 278 seconds remain
[2017-05-10 10:16:57,790: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 278 seconds remain
[2017-05-10 10:16:57,850: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 279 seconds remain
[2017-05-10 10:16:57,850: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 279 seconds remain
[2017-05-10 10:16:58,475: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 279 seconds remain
[2017-05-10 10:16:58,475: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 279 seconds remain
[2017-05-10 10:16:58,623: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 275 seconds remain
[2017-05-10 10:16:58,623: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 275 seconds remain
[2017-05-10 10:16:58,791: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 277 seconds remain
[2017-05-10 10:16:58,791: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 277 seconds remain
[2017-05-10 10:16:58,852: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 278 seconds remain
[2017-05-10 10:16:58,852: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 278 seconds remain
[2017-05-10 10:16:59,477: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 278 seconds remain
[2017-05-10 10:16:59,477: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 278 seconds remain
[2017-05-10 10:16:59,624: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 274 seconds remain
[2017-05-10 10:16:59,624: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 274 seconds remain
[2017-05-10 10:16:59,793: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 276 seconds remain
[2017-05-10 10:16:59,793: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 276 seconds remain
[2017-05-10 10:16:59,853: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 277 seconds remain
[2017-05-10 10:16:59,853: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 277 seconds remain
[2017-05-10 10:17:00,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 277 seconds remain
[2017-05-10 10:17:00,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 277 seconds remain
[2017-05-10 10:17:00,626: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 273 seconds remain
[2017-05-10 10:17:00,626: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 273 seconds remain
[2017-05-10 10:17:00,794: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 275 seconds remain
[2017-05-10 10:17:00,794: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 275 seconds remain
[2017-05-10 10:17:00,855: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 276 seconds remain
[2017-05-10 10:17:00,855: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 276 seconds remain
[2017-05-10 10:17:01,480: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 276 seconds remain
[2017-05-10 10:17:01,480: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 276 seconds remain
[2017-05-10 10:17:01,627: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 272 seconds remain
[2017-05-10 10:17:01,627: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 272 seconds remain
[2017-05-10 10:17:01,795: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 274 seconds remain
[2017-05-10 10:17:01,795: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 274 seconds remain
[2017-05-10 10:17:01,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 275 seconds remain
[2017-05-10 10:17:01,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 275 seconds remain
[2017-05-10 10:17:02,481: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 275 seconds remain
[2017-05-10 10:17:02,481: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 275 seconds remain
[2017-05-10 10:17:02,629: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 271 seconds remain
[2017-05-10 10:17:02,629: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 271 seconds remain
[2017-05-10 10:17:02,797: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 273 seconds remain
[2017-05-10 10:17:02,797: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 273 seconds remain
[2017-05-10 10:17:02,857: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 274 seconds remain
[2017-05-10 10:17:02,857: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 274 seconds remain
[2017-05-10 10:17:03,483: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 274 seconds remain
[2017-05-10 10:17:03,483: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 274 seconds remain
[2017-05-10 10:17:03,630: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 270 seconds remain
[2017-05-10 10:17:03,630: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 270 seconds remain
[2017-05-10 10:17:03,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 272 seconds remain
[2017-05-10 10:17:03,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 272 seconds remain
[2017-05-10 10:17:03,858: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 273 seconds remain
[2017-05-10 10:17:03,858: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 273 seconds remain
[2017-05-10 10:17:04,484: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 273 seconds remain
[2017-05-10 10:17:04,484: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 273 seconds remain
[2017-05-10 10:17:04,632: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 269 seconds remain
[2017-05-10 10:17:04,632: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 269 seconds remain
[2017-05-10 10:17:04,800: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 271 seconds remain
[2017-05-10 10:17:04,800: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 271 seconds remain
[2017-05-10 10:17:04,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 272 seconds remain
[2017-05-10 10:17:04,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 272 seconds remain
[2017-05-10 10:17:05,486: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 272 seconds remain
[2017-05-10 10:17:05,486: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 272 seconds remain
[2017-05-10 10:17:05,633: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 268 seconds remain
[2017-05-10 10:17:05,633: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 268 seconds remain
[2017-05-10 10:17:05,801: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 270 seconds remain
[2017-05-10 10:17:05,801: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 270 seconds remain
[2017-05-10 10:17:05,860: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 271 seconds remain
[2017-05-10 10:17:05,860: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 271 seconds remain
[2017-05-10 10:17:06,487: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 271 seconds remain
[2017-05-10 10:17:06,487: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 271 seconds remain
[2017-05-10 10:17:06,634: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 267 seconds remain
[2017-05-10 10:17:06,634: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 267 seconds remain
[2017-05-10 10:17:06,803: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 269 seconds remain
[2017-05-10 10:17:06,803: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 269 seconds remain
[2017-05-10 10:17:06,862: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 270 seconds remain
[2017-05-10 10:17:06,862: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 270 seconds remain
[2017-05-10 10:17:07,488: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 270 seconds remain
[2017-05-10 10:17:07,488: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 270 seconds remain
[2017-05-10 10:17:07,636: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 266 seconds remain
[2017-05-10 10:17:07,636: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 266 seconds remain
[2017-05-10 10:17:07,804: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 268 seconds remain
[2017-05-10 10:17:07,804: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 268 seconds remain
[2017-05-10 10:17:07,863: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 269 seconds remain
[2017-05-10 10:17:07,863: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 269 seconds remain
[2017-05-10 10:17:08,490: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 269 seconds remain
[2017-05-10 10:17:08,490: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 269 seconds remain
[2017-05-10 10:17:08,637: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 265 seconds remain
[2017-05-10 10:17:08,637: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 265 seconds remain
[2017-05-10 10:17:08,805: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 267 seconds remain
[2017-05-10 10:17:08,805: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 267 seconds remain
[2017-05-10 10:17:08,864: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 268 seconds remain
[2017-05-10 10:17:08,864: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 268 seconds remain
[2017-05-10 10:17:09,490: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 268 seconds remain
[2017-05-10 10:17:09,490: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 268 seconds remain
[2017-05-10 10:17:09,639: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 264 seconds remain
[2017-05-10 10:17:09,639: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 264 seconds remain
[2017-05-10 10:17:09,807: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 266 seconds remain
[2017-05-10 10:17:09,807: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 266 seconds remain
[2017-05-10 10:17:09,866: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 267 seconds remain
[2017-05-10 10:17:09,866: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 267 seconds remain
[2017-05-10 10:17:10,492: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 267 seconds remain
[2017-05-10 10:17:10,492: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 267 seconds remain
[2017-05-10 10:17:10,640: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 263 seconds remain
[2017-05-10 10:17:10,640: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 263 seconds remain
[2017-05-10 10:17:10,808: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 265 seconds remain
[2017-05-10 10:17:10,808: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 265 seconds remain
[2017-05-10 10:17:10,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 266 seconds remain
[2017-05-10 10:17:10,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 266 seconds remain
[2017-05-10 10:17:11,493: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 266 seconds remain
[2017-05-10 10:17:11,493: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 266 seconds remain
[2017-05-10 10:17:11,641: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 262 seconds remain
[2017-05-10 10:17:11,641: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 262 seconds remain
[2017-05-10 10:17:11,809: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 264 seconds remain
[2017-05-10 10:17:11,809: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 264 seconds remain
[2017-05-10 10:17:11,869: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 265 seconds remain
[2017-05-10 10:17:11,869: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 265 seconds remain
[2017-05-10 10:17:12,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 265 seconds remain
[2017-05-10 10:17:12,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 265 seconds remain
[2017-05-10 10:17:12,642: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 261 seconds remain
[2017-05-10 10:17:12,642: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 261 seconds remain
[2017-05-10 10:17:12,811: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 263 seconds remain
[2017-05-10 10:17:12,811: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 263 seconds remain
[2017-05-10 10:17:12,870: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 264 seconds remain
[2017-05-10 10:17:12,870: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 264 seconds remain
[2017-05-10 10:17:13,496: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 264 seconds remain
[2017-05-10 10:17:13,496: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 264 seconds remain
[2017-05-10 10:17:13,644: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 260 seconds remain
[2017-05-10 10:17:13,644: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 260 seconds remain
[2017-05-10 10:17:13,812: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 262 seconds remain
[2017-05-10 10:17:13,812: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 262 seconds remain
[2017-05-10 10:17:13,871: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 263 seconds remain
[2017-05-10 10:17:13,871: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 263 seconds remain
[2017-05-10 10:17:14,497: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 263 seconds remain
[2017-05-10 10:17:14,497: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 263 seconds remain
[2017-05-10 10:17:14,645: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 259 seconds remain
[2017-05-10 10:17:14,645: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 259 seconds remain
[2017-05-10 10:17:14,813: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 261 seconds remain
[2017-05-10 10:17:14,813: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 261 seconds remain
[2017-05-10 10:17:14,872: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 262 seconds remain
[2017-05-10 10:17:14,872: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 262 seconds remain
[2017-05-10 10:17:15,498: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 262 seconds remain
[2017-05-10 10:17:15,498: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 262 seconds remain
[2017-05-10 10:17:15,646: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 258 seconds remain
[2017-05-10 10:17:15,646: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 258 seconds remain
[2017-05-10 10:17:15,815: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 260 seconds remain
[2017-05-10 10:17:15,815: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 260 seconds remain
[2017-05-10 10:17:15,872: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 261 seconds remain
[2017-05-10 10:17:15,872: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 261 seconds remain
[2017-05-10 10:17:16,499: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 261 seconds remain
[2017-05-10 10:17:16,499: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 261 seconds remain
[2017-05-10 10:17:16,648: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 257 seconds remain
[2017-05-10 10:17:16,648: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 257 seconds remain
[2017-05-10 10:17:16,815: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 259 seconds remain
[2017-05-10 10:17:16,815: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 259 seconds remain
[2017-05-10 10:17:16,874: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 260 seconds remain
[2017-05-10 10:17:16,874: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 260 seconds remain
[2017-05-10 10:17:17,501: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 260 seconds remain
[2017-05-10 10:17:17,501: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 260 seconds remain
[2017-05-10 10:17:17,649: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 256 seconds remain
[2017-05-10 10:17:17,649: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 256 seconds remain
[2017-05-10 10:17:17,817: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 258 seconds remain
[2017-05-10 10:17:17,817: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 258 seconds remain
[2017-05-10 10:17:17,875: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 259 seconds remain
[2017-05-10 10:17:17,875: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 259 seconds remain
[2017-05-10 10:17:18,502: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 259 seconds remain
[2017-05-10 10:17:18,502: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 259 seconds remain
[2017-05-10 10:17:18,651: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 255 seconds remain
[2017-05-10 10:17:18,651: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 255 seconds remain
[2017-05-10 10:17:18,818: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 257 seconds remain
[2017-05-10 10:17:18,818: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 257 seconds remain
[2017-05-10 10:17:18,876: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 258 seconds remain
[2017-05-10 10:17:18,876: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 258 seconds remain
[2017-05-10 10:17:19,503: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 258 seconds remain
[2017-05-10 10:17:19,503: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 258 seconds remain
[2017-05-10 10:17:19,652: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 254 seconds remain
[2017-05-10 10:17:19,652: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 254 seconds remain
[2017-05-10 10:17:19,819: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 256 seconds remain
[2017-05-10 10:17:19,819: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 256 seconds remain
[2017-05-10 10:17:19,877: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 257 seconds remain
[2017-05-10 10:17:19,877: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 257 seconds remain
[2017-05-10 10:17:20,504: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 257 seconds remain
[2017-05-10 10:17:20,504: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 257 seconds remain
[2017-05-10 10:17:20,653: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 253 seconds remain
[2017-05-10 10:17:20,653: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 253 seconds remain
[2017-05-10 10:17:20,821: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 255 seconds remain
[2017-05-10 10:17:20,821: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 255 seconds remain
[2017-05-10 10:17:20,879: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 256 seconds remain
[2017-05-10 10:17:20,879: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 256 seconds remain
[2017-05-10 10:17:21,505: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 256 seconds remain
[2017-05-10 10:17:21,505: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 256 seconds remain
[2017-05-10 10:17:21,655: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 252 seconds remain
[2017-05-10 10:17:21,655: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 252 seconds remain
[2017-05-10 10:17:21,822: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 254 seconds remain
[2017-05-10 10:17:21,822: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 254 seconds remain
[2017-05-10 10:17:21,880: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 255 seconds remain
[2017-05-10 10:17:21,880: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 255 seconds remain
[2017-05-10 10:17:22,507: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 255 seconds remain
[2017-05-10 10:17:22,507: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 255 seconds remain
[2017-05-10 10:17:22,656: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 251 seconds remain
[2017-05-10 10:17:22,656: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 251 seconds remain
[2017-05-10 10:17:22,824: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 253 seconds remain
[2017-05-10 10:17:22,824: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 253 seconds remain
[2017-05-10 10:17:22,880: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 254 seconds remain
[2017-05-10 10:17:22,880: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 254 seconds remain
[2017-05-10 10:17:23,508: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 254 seconds remain
[2017-05-10 10:17:23,508: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 254 seconds remain
[2017-05-10 10:17:23,657: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 250 seconds remain
[2017-05-10 10:17:23,657: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 250 seconds remain
[2017-05-10 10:17:23,825: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 252 seconds remain
[2017-05-10 10:17:23,825: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 252 seconds remain
[2017-05-10 10:17:23,882: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 253 seconds remain
[2017-05-10 10:17:23,882: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 253 seconds remain
[2017-05-10 10:17:24,510: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 253 seconds remain
[2017-05-10 10:17:24,510: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 253 seconds remain
[2017-05-10 10:17:24,659: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 249 seconds remain
[2017-05-10 10:17:24,659: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 249 seconds remain
[2017-05-10 10:17:24,826: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 251 seconds remain
[2017-05-10 10:17:24,826: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 251 seconds remain
[2017-05-10 10:17:24,882: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 252 seconds remain
[2017-05-10 10:17:24,882: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 252 seconds remain
[2017-05-10 10:17:25,511: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 252 seconds remain
[2017-05-10 10:17:25,511: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 252 seconds remain
[2017-05-10 10:17:25,660: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 248 seconds remain
[2017-05-10 10:17:25,660: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 248 seconds remain
[2017-05-10 10:17:25,827: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 250 seconds remain
[2017-05-10 10:17:25,827: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 250 seconds remain
[2017-05-10 10:17:25,883: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 251 seconds remain
[2017-05-10 10:17:25,883: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 251 seconds remain
[2017-05-10 10:17:26,513: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 251 seconds remain
[2017-05-10 10:17:26,513: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 251 seconds remain
[2017-05-10 10:17:26,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 247 seconds remain
[2017-05-10 10:17:26,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 247 seconds remain
[2017-05-10 10:17:26,828: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 249 seconds remain
[2017-05-10 10:17:26,828: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 249 seconds remain
[2017-05-10 10:17:26,884: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 250 seconds remain
[2017-05-10 10:17:26,884: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 250 seconds remain
[2017-05-10 10:17:27,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 250 seconds remain
[2017-05-10 10:17:27,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 250 seconds remain
[2017-05-10 10:17:27,663: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 246 seconds remain
[2017-05-10 10:17:27,663: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 246 seconds remain
[2017-05-10 10:17:27,831: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 248 seconds remain
[2017-05-10 10:17:27,831: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 248 seconds remain
[2017-05-10 10:17:27,885: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 249 seconds remain
[2017-05-10 10:17:27,885: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 249 seconds remain
[2017-05-10 10:17:28,516: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 249 seconds remain
[2017-05-10 10:17:28,516: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 249 seconds remain
[2017-05-10 10:17:28,664: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 245 seconds remain
[2017-05-10 10:17:28,664: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 245 seconds remain
[2017-05-10 10:17:28,833: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 247 seconds remain
[2017-05-10 10:17:28,833: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 247 seconds remain
[2017-05-10 10:17:28,886: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 248 seconds remain
[2017-05-10 10:17:28,886: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 248 seconds remain
[2017-05-10 10:17:29,518: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 248 seconds remain
[2017-05-10 10:17:29,518: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 248 seconds remain
[2017-05-10 10:17:29,664: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 244 seconds remain
[2017-05-10 10:17:29,664: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 244 seconds remain
[2017-05-10 10:17:29,834: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 246 seconds remain
[2017-05-10 10:17:29,834: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 246 seconds remain
[2017-05-10 10:17:29,888: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 247 seconds remain
[2017-05-10 10:17:29,888: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 247 seconds remain
[2017-05-10 10:17:30,519: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 247 seconds remain
[2017-05-10 10:17:30,519: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 247 seconds remain
[2017-05-10 10:17:30,666: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 243 seconds remain
[2017-05-10 10:17:30,666: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 243 seconds remain
[2017-05-10 10:17:30,835: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 245 seconds remain
[2017-05-10 10:17:30,835: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 245 seconds remain
[2017-05-10 10:17:30,889: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 246 seconds remain
[2017-05-10 10:17:30,889: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 246 seconds remain
[2017-05-10 10:17:31,520: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 246 seconds remain
[2017-05-10 10:17:31,520: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 246 seconds remain
[2017-05-10 10:17:31,667: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 242 seconds remain
[2017-05-10 10:17:31,667: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 242 seconds remain
[2017-05-10 10:17:31,837: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 244 seconds remain
[2017-05-10 10:17:31,837: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 244 seconds remain
[2017-05-10 10:17:31,890: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 245 seconds remain
[2017-05-10 10:17:31,890: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 245 seconds remain
[2017-05-10 10:17:32,522: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 245 seconds remain
[2017-05-10 10:17:32,522: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 245 seconds remain
[2017-05-10 10:17:32,668: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 241 seconds remain
[2017-05-10 10:17:32,668: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 241 seconds remain
[2017-05-10 10:17:32,838: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 243 seconds remain
[2017-05-10 10:17:32,838: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 243 seconds remain
[2017-05-10 10:17:32,892: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 244 seconds remain
[2017-05-10 10:17:32,892: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 244 seconds remain
[2017-05-10 10:17:33,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 244 seconds remain
[2017-05-10 10:17:33,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 244 seconds remain
[2017-05-10 10:17:33,670: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 240 seconds remain
[2017-05-10 10:17:33,670: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 240 seconds remain
[2017-05-10 10:17:33,839: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 242 seconds remain
[2017-05-10 10:17:33,839: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 242 seconds remain
[2017-05-10 10:17:33,893: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 243 seconds remain
[2017-05-10 10:17:33,893: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 243 seconds remain
[2017-05-10 10:17:34,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 243 seconds remain
[2017-05-10 10:17:34,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 243 seconds remain
[2017-05-10 10:17:34,670: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 239 seconds remain
[2017-05-10 10:17:34,670: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 239 seconds remain
[2017-05-10 10:17:34,840: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 241 seconds remain
[2017-05-10 10:17:34,840: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 241 seconds remain
[2017-05-10 10:17:34,894: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 242 seconds remain
[2017-05-10 10:17:34,894: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 242 seconds remain
[2017-05-10 10:17:35,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 242 seconds remain
[2017-05-10 10:17:35,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 242 seconds remain
[2017-05-10 10:17:35,671: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 238 seconds remain
[2017-05-10 10:17:35,671: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 238 seconds remain
[2017-05-10 10:17:35,842: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 240 seconds remain
[2017-05-10 10:17:35,842: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 240 seconds remain
[2017-05-10 10:17:35,896: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 241 seconds remain
[2017-05-10 10:17:35,896: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 241 seconds remain
[2017-05-10 10:17:36,527: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 241 seconds remain
[2017-05-10 10:17:36,527: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 241 seconds remain
[2017-05-10 10:17:36,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 237 seconds remain
[2017-05-10 10:17:36,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 237 seconds remain
[2017-05-10 10:17:36,843: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 239 seconds remain
[2017-05-10 10:17:36,843: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 239 seconds remain
[2017-05-10 10:17:36,896: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 240 seconds remain
[2017-05-10 10:17:36,896: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 240 seconds remain
[2017-05-10 10:17:37,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 240 seconds remain
[2017-05-10 10:17:37,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 240 seconds remain
[2017-05-10 10:17:37,674: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 236 seconds remain
[2017-05-10 10:17:37,674: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 236 seconds remain
[2017-05-10 10:17:37,844: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 238 seconds remain
[2017-05-10 10:17:37,844: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 238 seconds remain
[2017-05-10 10:17:37,897: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 239 seconds remain
[2017-05-10 10:17:37,897: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 239 seconds remain
[2017-05-10 10:17:38,530: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 239 seconds remain
[2017-05-10 10:17:38,530: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 239 seconds remain
[2017-05-10 10:17:38,675: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 235 seconds remain
[2017-05-10 10:17:38,675: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 235 seconds remain
[2017-05-10 10:17:38,846: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 237 seconds remain
[2017-05-10 10:17:38,846: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 237 seconds remain
[2017-05-10 10:17:38,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 238 seconds remain
[2017-05-10 10:17:38,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 238 seconds remain
[2017-05-10 10:17:39,530: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 238 seconds remain
[2017-05-10 10:17:39,530: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 238 seconds remain
[2017-05-10 10:17:39,677: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 234 seconds remain
[2017-05-10 10:17:39,677: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 234 seconds remain
[2017-05-10 10:17:39,847: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 236 seconds remain
[2017-05-10 10:17:39,847: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 236 seconds remain
[2017-05-10 10:17:39,900: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 237 seconds remain
[2017-05-10 10:17:39,900: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 237 seconds remain
[2017-05-10 10:17:40,531: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 237 seconds remain
[2017-05-10 10:17:40,531: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 237 seconds remain
[2017-05-10 10:17:40,678: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 233 seconds remain
[2017-05-10 10:17:40,678: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 233 seconds remain
[2017-05-10 10:17:40,849: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 235 seconds remain
[2017-05-10 10:17:40,849: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 235 seconds remain
[2017-05-10 10:17:40,901: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 236 seconds remain
[2017-05-10 10:17:40,901: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 236 seconds remain
[2017-05-10 10:17:41,533: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 236 seconds remain
[2017-05-10 10:17:41,533: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 236 seconds remain
[2017-05-10 10:17:41,679: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 232 seconds remain
[2017-05-10 10:17:41,679: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 232 seconds remain
[2017-05-10 10:17:41,850: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 234 seconds remain
[2017-05-10 10:17:41,850: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 234 seconds remain
[2017-05-10 10:17:41,901: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 235 seconds remain
[2017-05-10 10:17:41,901: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 235 seconds remain
[2017-05-10 10:17:42,534: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 235 seconds remain
[2017-05-10 10:17:42,534: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 235 seconds remain
[2017-05-10 10:17:42,681: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 231 seconds remain
[2017-05-10 10:17:42,681: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 231 seconds remain
[2017-05-10 10:17:42,851: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 233 seconds remain
[2017-05-10 10:17:42,851: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 233 seconds remain
[2017-05-10 10:17:42,903: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 234 seconds remain
[2017-05-10 10:17:42,903: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 234 seconds remain
[2017-05-10 10:17:43,536: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 234 seconds remain
[2017-05-10 10:17:43,536: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 234 seconds remain
[2017-05-10 10:17:43,682: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 230 seconds remain
[2017-05-10 10:17:43,682: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 230 seconds remain
[2017-05-10 10:17:43,853: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 232 seconds remain
[2017-05-10 10:17:43,853: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 232 seconds remain
[2017-05-10 10:17:43,904: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 233 seconds remain
[2017-05-10 10:17:43,904: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 233 seconds remain
[2017-05-10 10:17:44,537: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 233 seconds remain
[2017-05-10 10:17:44,537: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 233 seconds remain
[2017-05-10 10:17:44,684: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 229 seconds remain
[2017-05-10 10:17:44,684: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 229 seconds remain
[2017-05-10 10:17:44,854: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 231 seconds remain
[2017-05-10 10:17:44,854: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 231 seconds remain
[2017-05-10 10:17:44,905: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 232 seconds remain
[2017-05-10 10:17:44,905: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 232 seconds remain
[2017-05-10 10:17:45,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 232 seconds remain
[2017-05-10 10:17:45,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 232 seconds remain
[2017-05-10 10:17:45,685: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 228 seconds remain
[2017-05-10 10:17:45,685: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 228 seconds remain
[2017-05-10 10:17:45,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 230 seconds remain
[2017-05-10 10:17:45,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 230 seconds remain
[2017-05-10 10:17:45,907: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 231 seconds remain
[2017-05-10 10:17:45,907: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 231 seconds remain
[2017-05-10 10:17:46,540: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 231 seconds remain
[2017-05-10 10:17:46,540: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 231 seconds remain
[2017-05-10 10:17:46,686: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 227 seconds remain
[2017-05-10 10:17:46,686: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 227 seconds remain
[2017-05-10 10:17:46,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 229 seconds remain
[2017-05-10 10:17:46,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 229 seconds remain
[2017-05-10 10:17:46,907: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 230 seconds remain
[2017-05-10 10:17:46,907: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 230 seconds remain
[2017-05-10 10:17:47,542: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 230 seconds remain
[2017-05-10 10:17:47,542: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 230 seconds remain
[2017-05-10 10:17:47,688: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 226 seconds remain
[2017-05-10 10:17:47,688: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 226 seconds remain
[2017-05-10 10:17:47,858: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 228 seconds remain
[2017-05-10 10:17:47,858: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 228 seconds remain
[2017-05-10 10:17:47,909: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 229 seconds remain
[2017-05-10 10:17:47,909: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 229 seconds remain
[2017-05-10 10:17:48,542: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 229 seconds remain
[2017-05-10 10:17:48,542: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 229 seconds remain
[2017-05-10 10:17:48,689: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 225 seconds remain
[2017-05-10 10:17:48,689: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 225 seconds remain
[2017-05-10 10:17:48,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 227 seconds remain
[2017-05-10 10:17:48,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 227 seconds remain
[2017-05-10 10:17:48,909: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 228 seconds remain
[2017-05-10 10:17:48,909: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 228 seconds remain
[2017-05-10 10:17:49,544: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 228 seconds remain
[2017-05-10 10:17:49,544: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 228 seconds remain
[2017-05-10 10:17:49,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 224 seconds remain
[2017-05-10 10:17:49,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 224 seconds remain
[2017-05-10 10:17:49,861: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 226 seconds remain
[2017-05-10 10:17:49,861: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 226 seconds remain
[2017-05-10 10:17:49,911: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 227 seconds remain
[2017-05-10 10:17:49,911: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 227 seconds remain
[2017-05-10 10:17:50,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 227 seconds remain
[2017-05-10 10:17:50,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 227 seconds remain
[2017-05-10 10:17:50,692: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 223 seconds remain
[2017-05-10 10:17:50,692: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 223 seconds remain
[2017-05-10 10:17:50,862: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 225 seconds remain
[2017-05-10 10:17:50,862: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 225 seconds remain
[2017-05-10 10:17:50,911: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 226 seconds remain
[2017-05-10 10:17:50,911: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 226 seconds remain
[2017-05-10 10:17:51,547: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 226 seconds remain
[2017-05-10 10:17:51,547: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 226 seconds remain
[2017-05-10 10:17:51,693: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 222 seconds remain
[2017-05-10 10:17:51,693: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 222 seconds remain
[2017-05-10 10:17:51,864: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 224 seconds remain
[2017-05-10 10:17:51,864: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 224 seconds remain
[2017-05-10 10:17:51,912: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 225 seconds remain
[2017-05-10 10:17:51,912: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 225 seconds remain
[2017-05-10 10:17:52,548: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 225 seconds remain
[2017-05-10 10:17:52,548: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 225 seconds remain
[2017-05-10 10:17:52,695: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 221 seconds remain
[2017-05-10 10:17:52,695: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 221 seconds remain
[2017-05-10 10:17:52,865: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 223 seconds remain
[2017-05-10 10:17:52,865: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 223 seconds remain
[2017-05-10 10:17:52,914: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 224 seconds remain
[2017-05-10 10:17:52,914: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 224 seconds remain
[2017-05-10 10:17:53,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 224 seconds remain
[2017-05-10 10:17:53,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 224 seconds remain
[2017-05-10 10:17:53,696: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 220 seconds remain
[2017-05-10 10:17:53,696: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 220 seconds remain
[2017-05-10 10:17:53,866: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 222 seconds remain
[2017-05-10 10:17:53,866: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 222 seconds remain
[2017-05-10 10:17:53,915: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 223 seconds remain
[2017-05-10 10:17:53,915: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 223 seconds remain
[2017-05-10 10:17:54,550: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 223 seconds remain
[2017-05-10 10:17:54,550: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 223 seconds remain
[2017-05-10 10:17:54,697: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 219 seconds remain
[2017-05-10 10:17:54,697: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 219 seconds remain
[2017-05-10 10:17:54,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 221 seconds remain
[2017-05-10 10:17:54,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 221 seconds remain
[2017-05-10 10:17:54,916: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 222 seconds remain
[2017-05-10 10:17:54,916: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 222 seconds remain
[2017-05-10 10:17:55,552: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 222 seconds remain
[2017-05-10 10:17:55,552: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 222 seconds remain
[2017-05-10 10:17:55,699: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 218 seconds remain
[2017-05-10 10:17:55,699: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 218 seconds remain
[2017-05-10 10:17:55,869: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 220 seconds remain
[2017-05-10 10:17:55,869: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 220 seconds remain
[2017-05-10 10:17:55,918: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 221 seconds remain
[2017-05-10 10:17:55,918: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 221 seconds remain
[2017-05-10 10:17:56,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 221 seconds remain
[2017-05-10 10:17:56,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 221 seconds remain
[2017-05-10 10:17:56,700: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 217 seconds remain
[2017-05-10 10:17:56,700: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 217 seconds remain
[2017-05-10 10:17:56,871: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 219 seconds remain
[2017-05-10 10:17:56,871: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 219 seconds remain
[2017-05-10 10:17:56,919: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 220 seconds remain
[2017-05-10 10:17:56,919: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 220 seconds remain
[2017-05-10 10:17:57,555: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 220 seconds remain
[2017-05-10 10:17:57,555: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 220 seconds remain
[2017-05-10 10:17:57,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 216 seconds remain
[2017-05-10 10:17:57,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 216 seconds remain
[2017-05-10 10:17:57,873: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 218 seconds remain
[2017-05-10 10:17:57,873: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 218 seconds remain
[2017-05-10 10:17:57,920: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 219 seconds remain
[2017-05-10 10:17:57,920: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 219 seconds remain
[2017-05-10 10:17:58,555: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 219 seconds remain
[2017-05-10 10:17:58,555: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 219 seconds remain
[2017-05-10 10:17:58,703: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 215 seconds remain
[2017-05-10 10:17:58,703: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 215 seconds remain
[2017-05-10 10:17:58,875: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 217 seconds remain
[2017-05-10 10:17:58,875: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 217 seconds remain
[2017-05-10 10:17:58,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 218 seconds remain
[2017-05-10 10:17:58,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 218 seconds remain
[2017-05-10 10:17:59,558: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 218 seconds remain
[2017-05-10 10:17:59,558: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 218 seconds remain
[2017-05-10 10:17:59,704: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 214 seconds remain
[2017-05-10 10:17:59,704: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 214 seconds remain
[2017-05-10 10:17:59,876: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 216 seconds remain
[2017-05-10 10:17:59,876: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 216 seconds remain
[2017-05-10 10:17:59,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 217 seconds remain
[2017-05-10 10:17:59,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 217 seconds remain
[2017-05-10 10:18:00,560: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 217 seconds remain
[2017-05-10 10:18:00,560: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 217 seconds remain
[2017-05-10 10:18:00,705: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 213 seconds remain
[2017-05-10 10:18:00,705: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 213 seconds remain
[2017-05-10 10:18:00,877: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 215 seconds remain
[2017-05-10 10:18:00,877: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 215 seconds remain
[2017-05-10 10:18:00,924: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 216 seconds remain
[2017-05-10 10:18:00,924: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 216 seconds remain
[2017-05-10 10:18:01,561: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 216 seconds remain
[2017-05-10 10:18:01,561: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 216 seconds remain
[2017-05-10 10:18:01,707: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 212 seconds remain
[2017-05-10 10:18:01,707: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 212 seconds remain
[2017-05-10 10:18:01,879: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 214 seconds remain
[2017-05-10 10:18:01,879: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 214 seconds remain
[2017-05-10 10:18:01,925: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 215 seconds remain
[2017-05-10 10:18:01,925: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 215 seconds remain
[2017-05-10 10:18:02,562: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 215 seconds remain
[2017-05-10 10:18:02,562: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 215 seconds remain
[2017-05-10 10:18:02,708: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 211 seconds remain
[2017-05-10 10:18:02,708: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 211 seconds remain
[2017-05-10 10:18:02,880: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 213 seconds remain
[2017-05-10 10:18:02,880: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 213 seconds remain
[2017-05-10 10:18:02,926: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 214 seconds remain
[2017-05-10 10:18:02,926: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 214 seconds remain
[2017-05-10 10:18:03,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 214 seconds remain
[2017-05-10 10:18:03,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 214 seconds remain
[2017-05-10 10:18:03,709: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 210 seconds remain
[2017-05-10 10:18:03,709: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 210 seconds remain
[2017-05-10 10:18:03,881: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 212 seconds remain
[2017-05-10 10:18:03,881: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 212 seconds remain
[2017-05-10 10:18:03,928: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 213 seconds remain
[2017-05-10 10:18:03,928: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 213 seconds remain
[2017-05-10 10:18:04,564: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 213 seconds remain
[2017-05-10 10:18:04,564: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 213 seconds remain
[2017-05-10 10:18:04,711: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 209 seconds remain
[2017-05-10 10:18:04,711: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 209 seconds remain
[2017-05-10 10:18:04,883: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 211 seconds remain
[2017-05-10 10:18:04,883: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 211 seconds remain
[2017-05-10 10:18:04,929: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 212 seconds remain
[2017-05-10 10:18:04,929: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 212 seconds remain
[2017-05-10 10:18:05,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 212 seconds remain
[2017-05-10 10:18:05,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 212 seconds remain
[2017-05-10 10:18:05,712: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 208 seconds remain
[2017-05-10 10:18:05,712: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 208 seconds remain
[2017-05-10 10:18:05,883: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 210 seconds remain
[2017-05-10 10:18:05,883: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 210 seconds remain
[2017-05-10 10:18:05,930: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 211 seconds remain
[2017-05-10 10:18:05,930: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 211 seconds remain
[2017-05-10 10:18:06,568: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 211 seconds remain
[2017-05-10 10:18:06,568: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 211 seconds remain
[2017-05-10 10:18:06,713: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 207 seconds remain
[2017-05-10 10:18:06,713: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 207 seconds remain
[2017-05-10 10:18:06,885: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 209 seconds remain
[2017-05-10 10:18:06,885: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 209 seconds remain
[2017-05-10 10:18:06,932: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 210 seconds remain
[2017-05-10 10:18:06,932: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 210 seconds remain
[2017-05-10 10:18:07,569: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 210 seconds remain
[2017-05-10 10:18:07,569: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 210 seconds remain
[2017-05-10 10:18:07,715: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 206 seconds remain
[2017-05-10 10:18:07,715: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 206 seconds remain
[2017-05-10 10:18:07,886: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 208 seconds remain
[2017-05-10 10:18:07,886: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 208 seconds remain
[2017-05-10 10:18:07,933: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 209 seconds remain
[2017-05-10 10:18:07,933: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 209 seconds remain
[2017-05-10 10:18:08,579: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 209 seconds remain
[2017-05-10 10:18:08,579: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 209 seconds remain
[2017-05-10 10:18:08,716: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 205 seconds remain
[2017-05-10 10:18:08,716: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 205 seconds remain
[2017-05-10 10:18:08,887: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 207 seconds remain
[2017-05-10 10:18:08,887: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 207 seconds remain
[2017-05-10 10:18:08,934: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 208 seconds remain
[2017-05-10 10:18:08,934: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 208 seconds remain
[2017-05-10 10:18:09,581: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 208 seconds remain
[2017-05-10 10:18:09,581: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 208 seconds remain
[2017-05-10 10:18:09,717: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 204 seconds remain
[2017-05-10 10:18:09,717: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 204 seconds remain
[2017-05-10 10:18:09,889: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 206 seconds remain
[2017-05-10 10:18:09,889: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 206 seconds remain
[2017-05-10 10:18:09,936: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 207 seconds remain
[2017-05-10 10:18:09,936: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 207 seconds remain
[2017-05-10 10:18:10,582: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 207 seconds remain
[2017-05-10 10:18:10,582: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 207 seconds remain
[2017-05-10 10:18:10,718: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 203 seconds remain
[2017-05-10 10:18:10,718: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 203 seconds remain
[2017-05-10 10:18:10,890: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 205 seconds remain
[2017-05-10 10:18:10,890: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 205 seconds remain
[2017-05-10 10:18:10,937: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 206 seconds remain
[2017-05-10 10:18:10,937: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 206 seconds remain
[2017-05-10 10:18:11,584: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 206 seconds remain
[2017-05-10 10:18:11,584: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 206 seconds remain
[2017-05-10 10:18:11,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 202 seconds remain
[2017-05-10 10:18:11,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 202 seconds remain
[2017-05-10 10:18:11,891: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 204 seconds remain
[2017-05-10 10:18:11,891: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 204 seconds remain
[2017-05-10 10:18:11,938: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 205 seconds remain
[2017-05-10 10:18:11,938: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 205 seconds remain
[2017-05-10 10:18:12,585: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 205 seconds remain
[2017-05-10 10:18:12,585: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 205 seconds remain
[2017-05-10 10:18:12,720: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 201 seconds remain
[2017-05-10 10:18:12,720: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 201 seconds remain
[2017-05-10 10:18:12,892: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 203 seconds remain
[2017-05-10 10:18:12,892: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 203 seconds remain
[2017-05-10 10:18:12,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 204 seconds remain
[2017-05-10 10:18:12,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 204 seconds remain
[2017-05-10 10:18:13,586: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 204 seconds remain
[2017-05-10 10:18:13,586: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 204 seconds remain
[2017-05-10 10:18:13,721: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 200 seconds remain
[2017-05-10 10:18:13,721: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 200 seconds remain
[2017-05-10 10:18:13,894: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 202 seconds remain
[2017-05-10 10:18:13,894: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 202 seconds remain
[2017-05-10 10:18:13,940: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 203 seconds remain
[2017-05-10 10:18:13,940: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 203 seconds remain
[2017-05-10 10:18:14,588: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 203 seconds remain
[2017-05-10 10:18:14,588: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 203 seconds remain
[2017-05-10 10:18:14,722: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 199 seconds remain
[2017-05-10 10:18:14,722: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 199 seconds remain
[2017-05-10 10:18:14,895: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 201 seconds remain
[2017-05-10 10:18:14,895: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 201 seconds remain
[2017-05-10 10:18:14,942: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 202 seconds remain
[2017-05-10 10:18:14,942: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 202 seconds remain
[2017-05-10 10:18:15,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 202 seconds remain
[2017-05-10 10:18:15,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 202 seconds remain
[2017-05-10 10:18:15,723: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 198 seconds remain
[2017-05-10 10:18:15,723: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 198 seconds remain
[2017-05-10 10:18:15,897: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 200 seconds remain
[2017-05-10 10:18:15,897: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 200 seconds remain
[2017-05-10 10:18:15,943: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 201 seconds remain
[2017-05-10 10:18:15,943: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 201 seconds remain
[2017-05-10 10:18:16,590: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 201 seconds remain
[2017-05-10 10:18:16,590: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 201 seconds remain
[2017-05-10 10:18:16,724: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 197 seconds remain
[2017-05-10 10:18:16,724: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 197 seconds remain
[2017-05-10 10:18:16,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 199 seconds remain
[2017-05-10 10:18:16,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 199 seconds remain
[2017-05-10 10:18:16,944: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 200 seconds remain
[2017-05-10 10:18:16,944: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 200 seconds remain
[2017-05-10 10:18:17,592: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 200 seconds remain
[2017-05-10 10:18:17,592: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 200 seconds remain
[2017-05-10 10:18:17,724: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 196 seconds remain
[2017-05-10 10:18:17,724: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 196 seconds remain
[2017-05-10 10:18:17,900: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 198 seconds remain
[2017-05-10 10:18:17,900: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 198 seconds remain
[2017-05-10 10:18:17,946: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 199 seconds remain
[2017-05-10 10:18:17,946: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 199 seconds remain
[2017-05-10 10:18:18,593: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 199 seconds remain
[2017-05-10 10:18:18,593: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 199 seconds remain
[2017-05-10 10:18:18,726: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 195 seconds remain
[2017-05-10 10:18:18,726: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 195 seconds remain
[2017-05-10 10:18:18,902: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 197 seconds remain
[2017-05-10 10:18:18,902: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 197 seconds remain
[2017-05-10 10:18:18,946: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 198 seconds remain
[2017-05-10 10:18:18,946: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 198 seconds remain
[2017-05-10 10:18:19,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 198 seconds remain
[2017-05-10 10:18:19,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 198 seconds remain
[2017-05-10 10:18:19,743: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 194 seconds remain
[2017-05-10 10:18:19,743: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 194 seconds remain
[2017-05-10 10:18:19,904: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 196 seconds remain
[2017-05-10 10:18:19,904: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 196 seconds remain
[2017-05-10 10:18:19,948: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 197 seconds remain
[2017-05-10 10:18:19,948: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 197 seconds remain
[2017-05-10 10:18:20,596: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 197 seconds remain
[2017-05-10 10:18:20,596: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 197 seconds remain
[2017-05-10 10:18:20,744: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 193 seconds remain
[2017-05-10 10:18:20,744: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 193 seconds remain
[2017-05-10 10:18:20,905: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 195 seconds remain
[2017-05-10 10:18:20,905: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 195 seconds remain
[2017-05-10 10:18:20,949: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 196 seconds remain
[2017-05-10 10:18:20,949: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 196 seconds remain
[2017-05-10 10:18:21,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 196 seconds remain
[2017-05-10 10:18:21,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 196 seconds remain
[2017-05-10 10:18:21,746: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 192 seconds remain
[2017-05-10 10:18:21,746: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 192 seconds remain
[2017-05-10 10:18:21,907: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 194 seconds remain
[2017-05-10 10:18:21,907: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 194 seconds remain
[2017-05-10 10:18:21,950: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 195 seconds remain
[2017-05-10 10:18:21,950: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 195 seconds remain
[2017-05-10 10:18:22,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 195 seconds remain
[2017-05-10 10:18:22,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 195 seconds remain
[2017-05-10 10:18:22,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 191 seconds remain
[2017-05-10 10:18:22,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 191 seconds remain
[2017-05-10 10:18:22,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 193 seconds remain
[2017-05-10 10:18:22,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 193 seconds remain
[2017-05-10 10:18:22,952: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 194 seconds remain
[2017-05-10 10:18:22,952: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 194 seconds remain
[2017-05-10 10:18:23,601: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 194 seconds remain
[2017-05-10 10:18:23,601: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 194 seconds remain
[2017-05-10 10:18:23,749: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 190 seconds remain
[2017-05-10 10:18:23,749: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 190 seconds remain
[2017-05-10 10:18:23,909: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 192 seconds remain
[2017-05-10 10:18:23,909: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 192 seconds remain
[2017-05-10 10:18:23,953: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 193 seconds remain
[2017-05-10 10:18:23,953: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 193 seconds remain
[2017-05-10 10:18:24,602: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 193 seconds remain
[2017-05-10 10:18:24,602: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 193 seconds remain
[2017-05-10 10:18:24,750: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 189 seconds remain
[2017-05-10 10:18:24,750: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 189 seconds remain
[2017-05-10 10:18:24,911: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 191 seconds remain
[2017-05-10 10:18:24,911: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 191 seconds remain
[2017-05-10 10:18:24,954: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 192 seconds remain
[2017-05-10 10:18:24,954: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 192 seconds remain
[2017-05-10 10:18:25,604: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 192 seconds remain
[2017-05-10 10:18:25,604: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 192 seconds remain
[2017-05-10 10:18:25,751: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 188 seconds remain
[2017-05-10 10:18:25,751: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 188 seconds remain
[2017-05-10 10:18:25,912: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 190 seconds remain
[2017-05-10 10:18:25,912: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 190 seconds remain
[2017-05-10 10:18:25,956: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 191 seconds remain
[2017-05-10 10:18:25,956: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 191 seconds remain
[2017-05-10 10:18:26,606: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 191 seconds remain
[2017-05-10 10:18:26,606: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 191 seconds remain
[2017-05-10 10:18:26,752: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 187 seconds remain
[2017-05-10 10:18:26,752: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 187 seconds remain
[2017-05-10 10:18:26,914: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 189 seconds remain
[2017-05-10 10:18:26,914: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 189 seconds remain
[2017-05-10 10:18:26,957: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 190 seconds remain
[2017-05-10 10:18:26,957: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 190 seconds remain
[2017-05-10 10:18:27,607: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 190 seconds remain
[2017-05-10 10:18:27,607: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 190 seconds remain
[2017-05-10 10:18:27,754: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 186 seconds remain
[2017-05-10 10:18:27,754: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 186 seconds remain
[2017-05-10 10:18:27,914: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 188 seconds remain
[2017-05-10 10:18:27,914: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 188 seconds remain
[2017-05-10 10:18:27,958: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 189 seconds remain
[2017-05-10 10:18:27,958: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 189 seconds remain
[2017-05-10 10:18:28,608: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 189 seconds remain
[2017-05-10 10:18:28,608: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 189 seconds remain
[2017-05-10 10:18:28,754: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 185 seconds remain
[2017-05-10 10:18:28,754: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 185 seconds remain
[2017-05-10 10:18:28,916: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 187 seconds remain
[2017-05-10 10:18:28,916: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 187 seconds remain
[2017-05-10 10:18:28,959: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 188 seconds remain
[2017-05-10 10:18:28,959: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 188 seconds remain
[2017-05-10 10:18:29,610: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 188 seconds remain
[2017-05-10 10:18:29,610: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 188 seconds remain
[2017-05-10 10:18:29,755: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 184 seconds remain
[2017-05-10 10:18:29,755: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 184 seconds remain
[2017-05-10 10:18:29,917: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 186 seconds remain
[2017-05-10 10:18:29,917: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 186 seconds remain
[2017-05-10 10:18:29,961: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 187 seconds remain
[2017-05-10 10:18:29,961: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 187 seconds remain
[2017-05-10 10:18:30,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 187 seconds remain
[2017-05-10 10:18:30,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 187 seconds remain
[2017-05-10 10:18:30,757: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 183 seconds remain
[2017-05-10 10:18:30,757: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 183 seconds remain
[2017-05-10 10:18:30,918: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 185 seconds remain
[2017-05-10 10:18:30,918: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 185 seconds remain
[2017-05-10 10:18:30,962: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 186 seconds remain
[2017-05-10 10:18:30,962: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 186 seconds remain
[2017-05-10 10:18:31,613: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 186 seconds remain
[2017-05-10 10:18:31,613: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 186 seconds remain
[2017-05-10 10:18:31,758: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 182 seconds remain
[2017-05-10 10:18:31,758: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 182 seconds remain
[2017-05-10 10:18:31,920: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 184 seconds remain
[2017-05-10 10:18:31,920: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 184 seconds remain
[2017-05-10 10:18:31,963: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 185 seconds remain
[2017-05-10 10:18:31,963: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 185 seconds remain
[2017-05-10 10:18:32,614: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 185 seconds remain
[2017-05-10 10:18:32,614: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 185 seconds remain
[2017-05-10 10:18:32,759: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 181 seconds remain
[2017-05-10 10:18:32,759: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 181 seconds remain
[2017-05-10 10:18:32,921: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 183 seconds remain
[2017-05-10 10:18:32,921: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 183 seconds remain
[2017-05-10 10:18:32,964: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 184 seconds remain
[2017-05-10 10:18:32,964: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 184 seconds remain
[2017-05-10 10:18:33,615: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 184 seconds remain
[2017-05-10 10:18:33,615: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 184 seconds remain
[2017-05-10 10:18:33,761: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 180 seconds remain
[2017-05-10 10:18:33,761: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 180 seconds remain
[2017-05-10 10:18:33,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 182 seconds remain
[2017-05-10 10:18:33,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 182 seconds remain
[2017-05-10 10:18:33,965: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 183 seconds remain
[2017-05-10 10:18:33,965: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 183 seconds remain
[2017-05-10 10:18:34,617: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 183 seconds remain
[2017-05-10 10:18:34,617: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 183 seconds remain
[2017-05-10 10:18:34,762: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 179 seconds remain
[2017-05-10 10:18:34,762: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 179 seconds remain
[2017-05-10 10:18:34,924: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 181 seconds remain
[2017-05-10 10:18:34,924: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 181 seconds remain
[2017-05-10 10:18:34,966: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 182 seconds remain
[2017-05-10 10:18:34,966: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 182 seconds remain
[2017-05-10 10:18:35,617: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 182 seconds remain
[2017-05-10 10:18:35,617: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 182 seconds remain
[2017-05-10 10:18:35,763: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 178 seconds remain
[2017-05-10 10:18:35,763: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 178 seconds remain
[2017-05-10 10:18:35,925: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 180 seconds remain
[2017-05-10 10:18:35,925: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 180 seconds remain
[2017-05-10 10:18:35,968: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 181 seconds remain
[2017-05-10 10:18:35,968: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 181 seconds remain
[2017-05-10 10:18:36,618: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 181 seconds remain
[2017-05-10 10:18:36,618: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 181 seconds remain
[2017-05-10 10:18:36,765: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 177 seconds remain
[2017-05-10 10:18:36,765: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 177 seconds remain
[2017-05-10 10:18:36,926: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 179 seconds remain
[2017-05-10 10:18:36,926: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 179 seconds remain
[2017-05-10 10:18:36,968: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 180 seconds remain
[2017-05-10 10:18:36,968: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 180 seconds remain
[2017-05-10 10:18:37,619: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 180 seconds remain
[2017-05-10 10:18:37,619: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 180 seconds remain
[2017-05-10 10:18:37,766: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 176 seconds remain
[2017-05-10 10:18:37,766: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 176 seconds remain
[2017-05-10 10:18:37,927: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 178 seconds remain
[2017-05-10 10:18:37,927: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 178 seconds remain
[2017-05-10 10:18:37,970: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 179 seconds remain
[2017-05-10 10:18:37,970: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 179 seconds remain
[2017-05-10 10:18:38,621: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 179 seconds remain
[2017-05-10 10:18:38,621: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 179 seconds remain
[2017-05-10 10:18:38,768: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 175 seconds remain
[2017-05-10 10:18:38,768: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 175 seconds remain
[2017-05-10 10:18:38,929: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 177 seconds remain
[2017-05-10 10:18:38,929: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 177 seconds remain
[2017-05-10 10:18:38,971: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 178 seconds remain
[2017-05-10 10:18:38,971: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 178 seconds remain
[2017-05-10 10:18:39,622: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 178 seconds remain
[2017-05-10 10:18:39,622: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 178 seconds remain
[2017-05-10 10:18:39,769: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 174 seconds remain
[2017-05-10 10:18:39,769: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 174 seconds remain
[2017-05-10 10:18:39,930: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 176 seconds remain
[2017-05-10 10:18:39,930: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 176 seconds remain
[2017-05-10 10:18:39,972: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 177 seconds remain
[2017-05-10 10:18:39,972: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 177 seconds remain
[2017-05-10 10:18:40,623: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 177 seconds remain
[2017-05-10 10:18:40,623: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 177 seconds remain
[2017-05-10 10:18:40,771: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 173 seconds remain
[2017-05-10 10:18:40,771: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 173 seconds remain
[2017-05-10 10:18:40,932: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 175 seconds remain
[2017-05-10 10:18:40,932: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 175 seconds remain
[2017-05-10 10:18:40,974: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 176 seconds remain
[2017-05-10 10:18:40,974: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 176 seconds remain
[2017-05-10 10:18:41,625: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 176 seconds remain
[2017-05-10 10:18:41,625: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 176 seconds remain
[2017-05-10 10:18:41,772: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 172 seconds remain
[2017-05-10 10:18:41,772: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 172 seconds remain
[2017-05-10 10:18:41,933: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 174 seconds remain
[2017-05-10 10:18:41,933: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 174 seconds remain
[2017-05-10 10:18:41,975: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 175 seconds remain
[2017-05-10 10:18:41,975: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 175 seconds remain
[2017-05-10 10:18:42,626: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 175 seconds remain
[2017-05-10 10:18:42,626: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 175 seconds remain
[2017-05-10 10:18:42,773: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 171 seconds remain
[2017-05-10 10:18:42,773: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 171 seconds remain
[2017-05-10 10:18:42,934: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 173 seconds remain
[2017-05-10 10:18:42,934: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 173 seconds remain
[2017-05-10 10:18:42,975: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 174 seconds remain
[2017-05-10 10:18:42,975: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 174 seconds remain
[2017-05-10 10:18:43,627: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 174 seconds remain
[2017-05-10 10:18:43,627: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 174 seconds remain
[2017-05-10 10:18:43,775: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 170 seconds remain
[2017-05-10 10:18:43,775: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 170 seconds remain
[2017-05-10 10:18:43,935: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 172 seconds remain
[2017-05-10 10:18:43,935: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 172 seconds remain
[2017-05-10 10:18:43,977: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 173 seconds remain
[2017-05-10 10:18:43,977: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 173 seconds remain
[2017-05-10 10:18:44,628: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 173 seconds remain
[2017-05-10 10:18:44,628: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 173 seconds remain
[2017-05-10 10:18:44,776: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 169 seconds remain
[2017-05-10 10:18:44,776: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 169 seconds remain
[2017-05-10 10:18:44,936: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 171 seconds remain
[2017-05-10 10:18:44,936: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 171 seconds remain
[2017-05-10 10:18:44,978: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 172 seconds remain
[2017-05-10 10:18:44,978: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 172 seconds remain
[2017-05-10 10:18:45,630: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 172 seconds remain
[2017-05-10 10:18:45,630: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 172 seconds remain
[2017-05-10 10:18:45,777: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 168 seconds remain
[2017-05-10 10:18:45,777: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 168 seconds remain
[2017-05-10 10:18:45,937: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 170 seconds remain
[2017-05-10 10:18:45,937: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 170 seconds remain
[2017-05-10 10:18:45,979: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 171 seconds remain
[2017-05-10 10:18:45,979: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 171 seconds remain
[2017-05-10 10:18:46,631: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 171 seconds remain
[2017-05-10 10:18:46,631: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 171 seconds remain
[2017-05-10 10:18:46,779: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 167 seconds remain
[2017-05-10 10:18:46,779: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 167 seconds remain
[2017-05-10 10:18:46,938: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 169 seconds remain
[2017-05-10 10:18:46,938: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 169 seconds remain
[2017-05-10 10:18:46,980: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 170 seconds remain
[2017-05-10 10:18:46,980: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 170 seconds remain
[2017-05-10 10:18:47,633: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 170 seconds remain
[2017-05-10 10:18:47,633: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 170 seconds remain
[2017-05-10 10:18:47,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 166 seconds remain
[2017-05-10 10:18:47,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 166 seconds remain
[2017-05-10 10:18:47,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 168 seconds remain
[2017-05-10 10:18:47,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 168 seconds remain
[2017-05-10 10:18:47,981: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 169 seconds remain
[2017-05-10 10:18:47,981: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 169 seconds remain
[2017-05-10 10:18:48,635: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 169 seconds remain
[2017-05-10 10:18:48,635: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 169 seconds remain
[2017-05-10 10:18:48,782: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 165 seconds remain
[2017-05-10 10:18:48,782: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 165 seconds remain
[2017-05-10 10:18:48,941: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 167 seconds remain
[2017-05-10 10:18:48,941: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 167 seconds remain
[2017-05-10 10:18:48,985: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 168 seconds remain
[2017-05-10 10:18:48,985: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 168 seconds remain
[2017-05-10 10:18:49,636: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 168 seconds remain
[2017-05-10 10:18:49,636: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 168 seconds remain
[2017-05-10 10:18:49,783: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 164 seconds remain
[2017-05-10 10:18:49,783: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 164 seconds remain
[2017-05-10 10:18:49,942: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 166 seconds remain
[2017-05-10 10:18:49,942: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 166 seconds remain
[2017-05-10 10:18:49,986: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 167 seconds remain
[2017-05-10 10:18:49,986: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 167 seconds remain
[2017-05-10 10:18:50,637: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 167 seconds remain
[2017-05-10 10:18:50,637: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 167 seconds remain
[2017-05-10 10:18:50,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 163 seconds remain
[2017-05-10 10:18:50,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 163 seconds remain
[2017-05-10 10:18:50,944: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 165 seconds remain
[2017-05-10 10:18:50,944: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 165 seconds remain
[2017-05-10 10:18:50,987: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 166 seconds remain
[2017-05-10 10:18:50,987: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 166 seconds remain
[2017-05-10 10:18:51,639: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 166 seconds remain
[2017-05-10 10:18:51,639: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 166 seconds remain
[2017-05-10 10:18:51,786: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 162 seconds remain
[2017-05-10 10:18:51,786: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 162 seconds remain
[2017-05-10 10:18:51,945: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 164 seconds remain
[2017-05-10 10:18:51,945: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 164 seconds remain
[2017-05-10 10:18:51,988: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 165 seconds remain
[2017-05-10 10:18:51,988: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 165 seconds remain
[2017-05-10 10:18:52,640: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 165 seconds remain
[2017-05-10 10:18:52,640: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 165 seconds remain
[2017-05-10 10:18:52,787: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 161 seconds remain
[2017-05-10 10:18:52,787: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 161 seconds remain
[2017-05-10 10:18:52,946: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 163 seconds remain
[2017-05-10 10:18:52,946: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 163 seconds remain
[2017-05-10 10:18:52,989: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 164 seconds remain
[2017-05-10 10:18:52,989: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 164 seconds remain
[2017-05-10 10:18:53,641: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 164 seconds remain
[2017-05-10 10:18:53,641: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 164 seconds remain
[2017-05-10 10:18:53,789: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 160 seconds remain
[2017-05-10 10:18:53,789: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 160 seconds remain
[2017-05-10 10:18:53,947: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 162 seconds remain
[2017-05-10 10:18:53,947: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 162 seconds remain
[2017-05-10 10:18:53,991: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 163 seconds remain
[2017-05-10 10:18:53,991: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 163 seconds remain
[2017-05-10 10:18:54,642: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 163 seconds remain
[2017-05-10 10:18:54,642: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 163 seconds remain
[2017-05-10 10:18:54,790: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 159 seconds remain
[2017-05-10 10:18:54,790: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 159 seconds remain
[2017-05-10 10:18:54,948: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 161 seconds remain
[2017-05-10 10:18:54,948: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 161 seconds remain
[2017-05-10 10:18:54,992: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 162 seconds remain
[2017-05-10 10:18:54,992: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 162 seconds remain
[2017-05-10 10:18:55,644: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 162 seconds remain
[2017-05-10 10:18:55,644: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 162 seconds remain
[2017-05-10 10:18:55,790: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 158 seconds remain
[2017-05-10 10:18:55,790: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 158 seconds remain
[2017-05-10 10:18:55,950: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 160 seconds remain
[2017-05-10 10:18:55,950: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 160 seconds remain
[2017-05-10 10:18:55,993: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 161 seconds remain
[2017-05-10 10:18:55,993: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 161 seconds remain
[2017-05-10 10:18:56,645: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 161 seconds remain
[2017-05-10 10:18:56,645: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 161 seconds remain
[2017-05-10 10:18:56,792: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 157 seconds remain
[2017-05-10 10:18:56,792: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 157 seconds remain
[2017-05-10 10:18:56,951: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 159 seconds remain
[2017-05-10 10:18:56,951: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 159 seconds remain
[2017-05-10 10:18:56,994: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 160 seconds remain
[2017-05-10 10:18:56,994: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 160 seconds remain
[2017-05-10 10:18:57,646: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 160 seconds remain
[2017-05-10 10:18:57,646: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 160 seconds remain
[2017-05-10 10:18:57,794: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 156 seconds remain
[2017-05-10 10:18:57,794: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 156 seconds remain
[2017-05-10 10:18:57,954: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 158 seconds remain
[2017-05-10 10:18:57,954: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 158 seconds remain
[2017-05-10 10:18:57,995: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 159 seconds remain
[2017-05-10 10:18:57,995: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 159 seconds remain
[2017-05-10 10:18:58,647: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 159 seconds remain
[2017-05-10 10:18:58,647: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 159 seconds remain
[2017-05-10 10:18:58,795: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 155 seconds remain
[2017-05-10 10:18:58,795: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 155 seconds remain
[2017-05-10 10:18:58,955: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 157 seconds remain
[2017-05-10 10:18:58,955: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 157 seconds remain
[2017-05-10 10:18:58,996: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 158 seconds remain
[2017-05-10 10:18:58,996: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 158 seconds remain
[2017-05-10 10:18:59,649: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 158 seconds remain
[2017-05-10 10:18:59,649: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 158 seconds remain
[2017-05-10 10:18:59,796: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 154 seconds remain
[2017-05-10 10:18:59,796: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 154 seconds remain
[2017-05-10 10:18:59,956: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 156 seconds remain
[2017-05-10 10:18:59,956: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 156 seconds remain
[2017-05-10 10:18:59,997: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 157 seconds remain
[2017-05-10 10:18:59,997: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 157 seconds remain
[2017-05-10 10:19:00,650: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 157 seconds remain
[2017-05-10 10:19:00,650: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 157 seconds remain
[2017-05-10 10:19:00,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 153 seconds remain
[2017-05-10 10:19:00,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 153 seconds remain
[2017-05-10 10:19:00,958: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 155 seconds remain
[2017-05-10 10:19:00,958: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 155 seconds remain
[2017-05-10 10:19:00,998: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 156 seconds remain
[2017-05-10 10:19:00,998: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 156 seconds remain
[2017-05-10 10:19:01,651: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 156 seconds remain
[2017-05-10 10:19:01,651: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 156 seconds remain
[2017-05-10 10:19:01,799: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 152 seconds remain
[2017-05-10 10:19:01,799: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 152 seconds remain
[2017-05-10 10:19:01,959: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 154 seconds remain
[2017-05-10 10:19:01,959: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 154 seconds remain
[2017-05-10 10:19:02,000: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 155 seconds remain
[2017-05-10 10:19:02,000: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 155 seconds remain
[2017-05-10 10:19:02,653: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 155 seconds remain
[2017-05-10 10:19:02,653: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 155 seconds remain
[2017-05-10 10:19:02,801: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 151 seconds remain
[2017-05-10 10:19:02,801: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 151 seconds remain
[2017-05-10 10:19:02,961: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 153 seconds remain
[2017-05-10 10:19:02,961: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 153 seconds remain
[2017-05-10 10:19:03,001: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 154 seconds remain
[2017-05-10 10:19:03,001: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 154 seconds remain
[2017-05-10 10:19:03,654: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 154 seconds remain
[2017-05-10 10:19:03,654: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 154 seconds remain
[2017-05-10 10:19:03,802: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 150 seconds remain
[2017-05-10 10:19:03,802: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 150 seconds remain
[2017-05-10 10:19:03,962: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 152 seconds remain
[2017-05-10 10:19:03,962: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 152 seconds remain
[2017-05-10 10:19:04,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 153 seconds remain
[2017-05-10 10:19:04,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 153 seconds remain
[2017-05-10 10:19:04,655: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 153 seconds remain
[2017-05-10 10:19:04,655: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 153 seconds remain
[2017-05-10 10:19:04,803: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 149 seconds remain
[2017-05-10 10:19:04,803: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 149 seconds remain
[2017-05-10 10:19:04,963: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 151 seconds remain
[2017-05-10 10:19:04,963: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 151 seconds remain
[2017-05-10 10:19:05,004: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 152 seconds remain
[2017-05-10 10:19:05,004: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 152 seconds remain
[2017-05-10 10:19:05,656: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 152 seconds remain
[2017-05-10 10:19:05,656: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 152 seconds remain
[2017-05-10 10:19:05,805: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 148 seconds remain
[2017-05-10 10:19:05,805: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 148 seconds remain
[2017-05-10 10:19:05,965: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 150 seconds remain
[2017-05-10 10:19:05,965: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 150 seconds remain
[2017-05-10 10:19:06,005: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 151 seconds remain
[2017-05-10 10:19:06,005: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 151 seconds remain
[2017-05-10 10:19:06,658: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 151 seconds remain
[2017-05-10 10:19:06,658: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 151 seconds remain
[2017-05-10 10:19:06,805: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 147 seconds remain
[2017-05-10 10:19:06,805: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 147 seconds remain
[2017-05-10 10:19:06,966: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 149 seconds remain
[2017-05-10 10:19:06,966: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 149 seconds remain
[2017-05-10 10:19:07,009: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 150 seconds remain
[2017-05-10 10:19:07,009: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 150 seconds remain
[2017-05-10 10:19:07,659: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 150 seconds remain
[2017-05-10 10:19:07,659: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 150 seconds remain
[2017-05-10 10:19:07,807: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 146 seconds remain
[2017-05-10 10:19:07,807: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 146 seconds remain
[2017-05-10 10:19:07,967: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 148 seconds remain
[2017-05-10 10:19:07,967: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 148 seconds remain
[2017-05-10 10:19:08,012: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 149 seconds remain
[2017-05-10 10:19:08,012: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 149 seconds remain
[2017-05-10 10:19:08,660: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 149 seconds remain
[2017-05-10 10:19:08,660: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 149 seconds remain
[2017-05-10 10:19:08,808: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 145 seconds remain
[2017-05-10 10:19:08,808: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 145 seconds remain
[2017-05-10 10:19:08,969: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 147 seconds remain
[2017-05-10 10:19:08,969: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 147 seconds remain
[2017-05-10 10:19:09,012: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 148 seconds remain
[2017-05-10 10:19:09,012: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 148 seconds remain
[2017-05-10 10:19:09,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 148 seconds remain
[2017-05-10 10:19:09,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 148 seconds remain
[2017-05-10 10:19:09,809: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 144 seconds remain
[2017-05-10 10:19:09,809: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 144 seconds remain
[2017-05-10 10:19:09,970: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 146 seconds remain
[2017-05-10 10:19:09,970: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 146 seconds remain
[2017-05-10 10:19:10,014: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 147 seconds remain
[2017-05-10 10:19:10,014: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 147 seconds remain
[2017-05-10 10:19:10,663: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 147 seconds remain
[2017-05-10 10:19:10,663: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 147 seconds remain
[2017-05-10 10:19:10,810: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 143 seconds remain
[2017-05-10 10:19:10,810: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 143 seconds remain
[2017-05-10 10:19:10,971: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 145 seconds remain
[2017-05-10 10:19:10,971: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 145 seconds remain
[2017-05-10 10:19:11,015: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 146 seconds remain
[2017-05-10 10:19:11,015: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 146 seconds remain
[2017-05-10 10:19:11,664: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 146 seconds remain
[2017-05-10 10:19:11,664: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 146 seconds remain
[2017-05-10 10:19:11,812: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 142 seconds remain
[2017-05-10 10:19:11,812: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 142 seconds remain
[2017-05-10 10:19:11,972: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 144 seconds remain
[2017-05-10 10:19:11,972: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 144 seconds remain
[2017-05-10 10:19:12,016: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 145 seconds remain
[2017-05-10 10:19:12,016: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 145 seconds remain
[2017-05-10 10:19:12,665: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 145 seconds remain
[2017-05-10 10:19:12,665: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 145 seconds remain
[2017-05-10 10:19:12,812: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 141 seconds remain
[2017-05-10 10:19:12,812: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 141 seconds remain
[2017-05-10 10:19:12,973: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 143 seconds remain
[2017-05-10 10:19:12,973: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 143 seconds remain
[2017-05-10 10:19:13,017: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 144 seconds remain
[2017-05-10 10:19:13,017: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 144 seconds remain
[2017-05-10 10:19:13,667: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 144 seconds remain
[2017-05-10 10:19:13,667: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 144 seconds remain
[2017-05-10 10:19:13,814: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 140 seconds remain
[2017-05-10 10:19:13,814: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 140 seconds remain
[2017-05-10 10:19:13,974: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 142 seconds remain
[2017-05-10 10:19:13,974: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 142 seconds remain
[2017-05-10 10:19:14,018: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 143 seconds remain
[2017-05-10 10:19:14,018: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 143 seconds remain
[2017-05-10 10:19:14,668: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 143 seconds remain
[2017-05-10 10:19:14,668: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 143 seconds remain
[2017-05-10 10:19:14,815: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 139 seconds remain
[2017-05-10 10:19:14,815: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 139 seconds remain
[2017-05-10 10:19:14,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 141 seconds remain
[2017-05-10 10:19:14,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 141 seconds remain
[2017-05-10 10:19:15,020: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 142 seconds remain
[2017-05-10 10:19:15,020: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 142 seconds remain
[2017-05-10 10:19:15,670: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 142 seconds remain
[2017-05-10 10:19:15,670: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 142 seconds remain
[2017-05-10 10:19:15,816: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 138 seconds remain
[2017-05-10 10:19:15,816: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 138 seconds remain
[2017-05-10 10:19:15,977: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 140 seconds remain
[2017-05-10 10:19:15,977: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 140 seconds remain
[2017-05-10 10:19:16,020: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 141 seconds remain
[2017-05-10 10:19:16,020: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 141 seconds remain
[2017-05-10 10:19:16,671: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 141 seconds remain
[2017-05-10 10:19:16,671: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 141 seconds remain
[2017-05-10 10:19:16,818: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 137 seconds remain
[2017-05-10 10:19:16,818: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 137 seconds remain
[2017-05-10 10:19:16,978: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 139 seconds remain
[2017-05-10 10:19:16,978: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 139 seconds remain
[2017-05-10 10:19:17,022: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 140 seconds remain
[2017-05-10 10:19:17,022: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 140 seconds remain
[2017-05-10 10:19:17,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 140 seconds remain
[2017-05-10 10:19:17,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 140 seconds remain
[2017-05-10 10:19:17,819: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 136 seconds remain
[2017-05-10 10:19:17,819: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 136 seconds remain
[2017-05-10 10:19:17,980: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 138 seconds remain
[2017-05-10 10:19:17,980: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 138 seconds remain
[2017-05-10 10:19:18,023: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 139 seconds remain
[2017-05-10 10:19:18,023: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 139 seconds remain
[2017-05-10 10:19:18,674: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 139 seconds remain
[2017-05-10 10:19:18,674: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 139 seconds remain
[2017-05-10 10:19:18,821: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 135 seconds remain
[2017-05-10 10:19:18,821: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 135 seconds remain
[2017-05-10 10:19:18,981: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 137 seconds remain
[2017-05-10 10:19:18,981: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 137 seconds remain
[2017-05-10 10:19:19,025: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 138 seconds remain
[2017-05-10 10:19:19,025: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 138 seconds remain
[2017-05-10 10:19:19,674: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 138 seconds remain
[2017-05-10 10:19:19,674: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 138 seconds remain
[2017-05-10 10:19:19,822: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 134 seconds remain
[2017-05-10 10:19:19,822: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 134 seconds remain
[2017-05-10 10:19:19,983: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 136 seconds remain
[2017-05-10 10:19:19,983: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 136 seconds remain
[2017-05-10 10:19:20,026: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 137 seconds remain
[2017-05-10 10:19:20,026: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 137 seconds remain
[2017-05-10 10:19:20,069: WARNING/chroma_core.services.job_scheduler.agent_rpc] Unable to reset action_runner sessions
[2017-05-10 10:19:20,069: WARNING/chroma_core.services.job_scheduler.agent_rpc] Unable to reset action_runner sessions
[2017-05-10 10:19:20,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: None/None
[2017-05-10 10:19:20,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: None/None
[2017-05-10 10:19:20,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/62d70472-58ab-4e93-8d14-6d9727df060a
[2017-05-10 10:19:20,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/62d70472-58ab-4e93-8d14-6d9727df060a
[2017-05-10 10:19:20,086: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/c97abb29-f77e-4e47-9fc3-cfacbdded707
[2017-05-10 10:19:20,086: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/c97abb29-f77e-4e47-9fc3-cfacbdded707
[2017-05-10 10:19:20,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/4ac07e17-3406-4bcb-8fdc-4a9d2a2da765
[2017-05-10 10:19:20,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/4ac07e17-3406-4bcb-8fdc-4a9d2a2da765
[2017-05-10 10:19:20,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/cd966cf6-f44e-4de0-85e6-258297056d06
[2017-05-10 10:19:20,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/cd966cf6-f44e-4de0-85e6-258297056d06
[2017-05-10 10:19:20,675: INFO/job_scheduler] Job 8 finished 1 steps successfully
[2017-05-10 10:19:20,709: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 140516297791952
[2017-05-10 10:19:20,716: INFO/job] Job 8: StateChangeJob complete, setting state unconfigured on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:19:20,716: INFO/job] Job 8: not_deleted=True
[2017-05-10 10:19:20,716: INFO/job_scheduler] Job 8: complete_job: Updating cache
[2017-05-10 10:19:20,716: INFO/job_scheduler] Job 8: locked_item 140516297791952 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 10:19:20,720: INFO/job_scheduler] Job 8 completing (errored=False, cancelled=False)
[2017-05-10 10:19:20,786: INFO/job_scheduler] Replaying 3 buffered notifications for ManagedHost-4
[2017-05-10 10:19:20,787: INFO/job_scheduler] _notify: Updating .boot_time of item 4 (lotus-32vm6.lotus.hpdd.lab.intel.com) from None to 2017-04-25 07:09:19+00:00
[2017-05-10 10:19:20,801: INFO/job_scheduler] _notify: Set boot_time=2017-04-25 07:09:19+00:00 on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-4) and saved
[2017-05-10 10:19:20,816: INFO/job_scheduler] _notify: Updating .properties of item 4 (lotus-32vm6.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-05-10 10:19:20,821: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-4) and saved
[2017-05-10 10:19:20,839: INFO/job_scheduler] Job 5 finished 1 steps successfully
[2017-05-10 10:19:20,844: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 10:19:20,859: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 140516297603408
[2017-05-10 10:19:20,865: INFO/job] Job 5: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:19:20,865: INFO/job] Job 5: not_deleted=True
[2017-05-10 10:19:20,865: INFO/job_scheduler] Job 5: complete_job: Updating cache
[2017-05-10 10:19:20,865: INFO/job_scheduler] Job 5: locked_item 140516297603408 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 10:19:20,869: INFO/job_scheduler] Job 5 completing (errored=False, cancelled=False)
[2017-05-10 10:19:20,929: INFO/job_scheduler] Replaying 3 buffered notifications for ManagedHost-1
[2017-05-10 10:19:20,932: INFO/job_scheduler] _notify: Updating .boot_time of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com) from None to 2017-05-03 16:42:07+00:00
[2017-05-10 10:19:20,940: INFO/job_scheduler] _notify: Set boot_time=2017-05-03 16:42:07+00:00 on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-1) and saved
[2017-05-10 10:19:20,946: INFO/job_scheduler] _notify: Updating .properties of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-05-10 10:19:20,951: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-1) and saved
[2017-05-10 10:19:20,970: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-05-10 10:19:20,984: INFO/job_scheduler] Job 7 finished 1 steps successfully
[2017-05-10 10:19:21,001: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 140516297759440
[2017-05-10 10:19:21,006: INFO/job] Job 7: StateChangeJob complete, setting state unconfigured on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:19:21,007: INFO/job] Job 7: not_deleted=True
[2017-05-10 10:19:21,007: INFO/job_scheduler] Job 7: complete_job: Updating cache
[2017-05-10 10:19:21,007: INFO/job_scheduler] Job 7: locked_item 140516297759440 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 10:19:21,010: INFO/job_scheduler] Job 7 completing (errored=False, cancelled=False)
[2017-05-10 10:19:21,029: INFO/job_scheduler] Job 6 finished 1 steps successfully
[2017-05-10 10:19:21,067: INFO/job_scheduler] Replaying 3 buffered notifications for ManagedHost-3
[2017-05-10 10:19:21,067: INFO/job_scheduler] _notify: Updating .boot_time of item 3 (lotus-32vm8.lotus.hpdd.lab.intel.com) from None to 2017-04-25 07:09:56+00:00
[2017-05-10 10:19:21,085: INFO/job_scheduler] _notify: Set boot_time=2017-04-25 07:09:56+00:00 on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-3) and saved
[2017-05-10 10:19:21,104: INFO/job_scheduler] _notify: Updating .properties of item 3 (lotus-32vm8.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-05-10 10:19:21,110: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-3) and saved
[2017-05-10 10:19:21,128: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 10:19:21,142: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 140516297582928
[2017-05-10 10:19:21,148: INFO/job] Job 6: StateChangeJob complete, setting state unconfigured on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:19:21,148: INFO/job] Job 6: not_deleted=True
[2017-05-10 10:19:21,148: INFO/job_scheduler] Job 6: complete_job: Updating cache
[2017-05-10 10:19:21,148: INFO/job_scheduler] Job 6: locked_item 140516297582928 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 10:19:21,152: INFO/job_scheduler] Job 6 completing (errored=False, cancelled=False)
[2017-05-10 10:19:21,206: INFO/job_scheduler] Replaying 3 buffered notifications for ManagedHost-2
[2017-05-10 10:19:21,207: INFO/job_scheduler] _notify: Updating .boot_time of item 2 (lotus-32vm7.lotus.hpdd.lab.intel.com) from None to 2017-04-25 07:09:31+00:00
[2017-05-10 10:19:21,221: INFO/job_scheduler] _notify: Set boot_time=2017-04-25 07:09:31+00:00 on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-2) and saved
[2017-05-10 10:19:21,236: INFO/job_scheduler] _notify: Updating .properties of item 2 (lotus-32vm7.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-05-10 10:19:21,242: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-2) and saved
[2017-05-10 10:19:21,260: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:19:26,180: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:19:26,183: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-1 transitioning from unconfigured to managed
[2017-05-10 10:19:26,348: INFO/command_plan] Created command 9 (Setup managed host lotus-32vm5.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-05-10 10:19:26,389: INFO/job_scheduler] run_next: 1 runnable jobs of (11 pending, 0 tasked)
[2017-05-10 10:19:26,414: INFO/job_scheduler] Job 9: RunJobThread.run
[2017-05-10 10:19:26,415: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-05-10 10:19:26,430: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d7d60159-bbd7-430b-af15-7b1d2bde4db1
[2017-05-10 10:19:26,430: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d7d60159-bbd7-430b-af15-7b1d2bde4db1
[2017-05-10 10:19:26,479: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:19:26,482: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-2 transitioning from unconfigured to managed
[2017-05-10 10:19:26,628: INFO/command_plan] Created command 10 (Setup managed host lotus-32vm7.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-05-10 10:19:26,676: INFO/job_scheduler] run_next: 1 runnable jobs of (21 pending, 1 tasked)
[2017-05-10 10:19:26,702: INFO/job_scheduler] Job 20: RunJobThread.run
[2017-05-10 10:19:26,705: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-05-10 10:19:26,729: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 218afb38-0561-458f-be4a-95f22e43e883
[2017-05-10 10:19:26,729: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 218afb38-0561-458f-be4a-95f22e43e883
[2017-05-10 10:19:26,768: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:19:26,773: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-3 transitioning from unconfigured to managed
[2017-05-10 10:19:26,905: INFO/command_plan] Created command 11 (Setup managed host lotus-32vm8.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-05-10 10:19:26,925: INFO/job_scheduler] run_next: 1 runnable jobs of (31 pending, 2 tasked)
[2017-05-10 10:19:26,957: INFO/job_scheduler] Job 31: RunJobThread.run
[2017-05-10 10:19:26,958: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-05-10 10:19:26,994: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 356e7ef3-e162-41a7-9acd-98cf49d1b7fc
[2017-05-10 10:19:26,994: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 356e7ef3-e162-41a7-9acd-98cf49d1b7fc
[2017-05-10 10:19:27,066: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:19:27,071: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-4 transitioning from unconfigured to managed
[2017-05-10 10:19:27,213: INFO/command_plan] Created command 12 (Setup managed host lotus-32vm6.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-05-10 10:19:27,265: INFO/job_scheduler] run_next: 1 runnable jobs of (41 pending, 3 tasked)
[2017-05-10 10:19:27,287: INFO/job_scheduler] Job 42: RunJobThread.run
[2017-05-10 10:19:27,287: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-05-10 10:19:27,304: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7e14b7c1-25b9-4318-aa00-357574efd8b3
[2017-05-10 10:19:27,304: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7e14b7c1-25b9-4318-aa00-357574efd8b3
[2017-05-10 10:21:13,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/4ac07e17-3406-4bcb-8fdc-4a9d2a2da765
[2017-05-10 10:21:13,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/4ac07e17-3406-4bcb-8fdc-4a9d2a2da765
[2017-05-10 10:21:13,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/4ac07e17-3406-4bcb-8fdc-4a9d2a2da765
[2017-05-10 10:21:13,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/4ac07e17-3406-4bcb-8fdc-4a9d2a2da765
[2017-05-10 10:21:13,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 218afb38-0561-458f-be4a-95f22e43e883
[2017-05-10 10:21:13,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 218afb38-0561-458f-be4a-95f22e43e883
[2017-05-10 10:21:13,113: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 218afb38-0561-458f-be4a-95f22e43e883
[2017-05-10 10:21:13,113: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 218afb38-0561-458f-be4a-95f22e43e883
[2017-05-10 10:21:13,114: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-05-10 10:21:13.114005+00:00:
[2017-05-10 10:21:13,114: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-05-10 10:21:13.114005+00:00:
[2017-05-10 10:21:13,114: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 10:21:13,114: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 10:21:13,128: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-05-10 10:21:13.128911+00:00:
[2017-05-10 10:21:13,128: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-05-10 10:21:13.128911+00:00:
[2017-05-10 10:21:13,129: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 10:21:13,129: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 10:21:13,140: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:21:13,140: INFO/job_scheduler]   Job 20 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 10:21:13,141: INFO/job_scheduler]   Job 21 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:13,141: INFO/job_scheduler]   Job 22 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:13,141: INFO/job_scheduler]   Job 23 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:13,141: INFO/job_scheduler]   Job 24 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:13,142: INFO/job_scheduler]   Job 27 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:13,143: INFO/job_scheduler]   Job 30 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:21:13,144: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 10:21:13,148: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:21:13,149: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com restart_agent {}
[2017-05-10 10:21:13,149: INFO/job_scheduler]   Job 20 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 10:21:13,149: INFO/job_scheduler]   Job 21 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:13,150: INFO/job_scheduler]   Job 22 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:13,158: INFO/job_scheduler]   Job 23 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:13,159: INFO/job_scheduler]   Job 24 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:13,159: INFO/job_scheduler]   Job 27 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:13,159: INFO/job_scheduler]   Job 30 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:21:13,160: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 10:21:13,171: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4977e724-8a9f-487d-a335-e2c81061410a
[2017-05-10 10:21:13,171: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4977e724-8a9f-487d-a335-e2c81061410a
[2017-05-10 10:21:13,259: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/4ac07e17-3406-4bcb-8fdc-4a9d2a2da765
[2017-05-10 10:21:13,259: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/4ac07e17-3406-4bcb-8fdc-4a9d2a2da765
[2017-05-10 10:21:13,260: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/4ac07e17-3406-4bcb-8fdc-4a9d2a2da765
[2017-05-10 10:21:13,260: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/4ac07e17-3406-4bcb-8fdc-4a9d2a2da765
[2017-05-10 10:21:13,260: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4977e724-8a9f-487d-a335-e2c81061410a
[2017-05-10 10:21:13,260: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4977e724-8a9f-487d-a335-e2c81061410a
[2017-05-10 10:21:13,285: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4977e724-8a9f-487d-a335-e2c81061410a
[2017-05-10 10:21:13,285: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4977e724-8a9f-487d-a335-e2c81061410a
[2017-05-10 10:21:13,285: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm7.lotus.hpdd.lab.intel.com (old 4ac07e17-3406-4bcb-8fdc-4a9d2a2da765)
[2017-05-10 10:21:13,285: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm7.lotus.hpdd.lab.intel.com (old 4ac07e17-3406-4bcb-8fdc-4a9d2a2da765)
[2017-05-10 10:21:16,605: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/62d70472-58ab-4e93-8d14-6d9727df060a
[2017-05-10 10:21:16,605: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/62d70472-58ab-4e93-8d14-6d9727df060a
[2017-05-10 10:21:16,606: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/62d70472-58ab-4e93-8d14-6d9727df060a
[2017-05-10 10:21:16,606: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/62d70472-58ab-4e93-8d14-6d9727df060a
[2017-05-10 10:21:16,606: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d7d60159-bbd7-430b-af15-7b1d2bde4db1
[2017-05-10 10:21:16,606: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d7d60159-bbd7-430b-af15-7b1d2bde4db1
[2017-05-10 10:21:16,610: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d7d60159-bbd7-430b-af15-7b1d2bde4db1
[2017-05-10 10:21:16,610: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d7d60159-bbd7-430b-af15-7b1d2bde4db1
[2017-05-10 10:21:16,610: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-10 10:21:16.610562+00:00:
[2017-05-10 10:21:16,610: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-10 10:21:16.610562+00:00:
[2017-05-10 10:21:16,610: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 10:21:16,610: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 10:21:16,624: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-10 10:21:16.624192+00:00:
[2017-05-10 10:21:16,624: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-10 10:21:16.624192+00:00:
[2017-05-10 10:21:16,624: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 10:21:16,624: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 10:21:16,634: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:21:16,635: INFO/job_scheduler]   Job 9 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 10:21:16,635: INFO/job_scheduler]   Job 10 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:16,635: INFO/job_scheduler]   Job 11 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:16,635: INFO/job_scheduler]   Job 12 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:16,635: INFO/job_scheduler]   Job 13 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:16,635: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:16,635: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:21:16,636: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 10:21:16,641: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:21:16,642: INFO/job_scheduler]   Job 9 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 10:21:16,642: INFO/job_scheduler]   Job 10 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:16,642: INFO/job_scheduler]   Job 11 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:16,642: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com restart_agent {}
[2017-05-10 10:21:16,642: INFO/job_scheduler]   Job 12 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:16,643: INFO/job_scheduler]   Job 13 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:16,651: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:16,651: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:21:16,652: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 10:21:16,662: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b7caef8e-bf29-4d4e-9afd-3f1de589622e
[2017-05-10 10:21:16,662: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b7caef8e-bf29-4d4e-9afd-3f1de589622e
[2017-05-10 10:21:16,778: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/62d70472-58ab-4e93-8d14-6d9727df060a
[2017-05-10 10:21:16,778: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/62d70472-58ab-4e93-8d14-6d9727df060a
[2017-05-10 10:21:16,779: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/62d70472-58ab-4e93-8d14-6d9727df060a
[2017-05-10 10:21:16,779: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/62d70472-58ab-4e93-8d14-6d9727df060a
[2017-05-10 10:21:16,779: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b7caef8e-bf29-4d4e-9afd-3f1de589622e
[2017-05-10 10:21:16,779: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b7caef8e-bf29-4d4e-9afd-3f1de589622e
[2017-05-10 10:21:16,826: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b7caef8e-bf29-4d4e-9afd-3f1de589622e
[2017-05-10 10:21:16,826: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b7caef8e-bf29-4d4e-9afd-3f1de589622e
[2017-05-10 10:21:16,826: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old 62d70472-58ab-4e93-8d14-6d9727df060a)
[2017-05-10 10:21:16,826: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old 62d70472-58ab-4e93-8d14-6d9727df060a)
[2017-05-10 10:21:18,507: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/cd966cf6-f44e-4de0-85e6-258297056d06
[2017-05-10 10:21:18,507: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/cd966cf6-f44e-4de0-85e6-258297056d06
[2017-05-10 10:21:18,507: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/cd966cf6-f44e-4de0-85e6-258297056d06
[2017-05-10 10:21:18,507: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/cd966cf6-f44e-4de0-85e6-258297056d06
[2017-05-10 10:21:18,507: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7e14b7c1-25b9-4318-aa00-357574efd8b3
[2017-05-10 10:21:18,507: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7e14b7c1-25b9-4318-aa00-357574efd8b3
[2017-05-10 10:21:18,542: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7e14b7c1-25b9-4318-aa00-357574efd8b3
[2017-05-10 10:21:18,542: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7e14b7c1-25b9-4318-aa00-357574efd8b3
[2017-05-10 10:21:18,542: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-05-10 10:21:18.542413+00:00:
[2017-05-10 10:21:18,542: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-05-10 10:21:18.542413+00:00:
[2017-05-10 10:21:18,542: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 10:21:18,542: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 10:21:18,561: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:21:18,561: INFO/job_scheduler]   Job 42 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 10:21:18,562: INFO/job_scheduler]   Job 43 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:18,562: INFO/job_scheduler]   Job 44 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:18,562: INFO/job_scheduler]   Job 45 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:18,562: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-05-10 10:21:18.562076+00:00:
[2017-05-10 10:21:18,562: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-05-10 10:21:18.562076+00:00:
[2017-05-10 10:21:18,563: INFO/job_scheduler]   Job 46 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:18,563: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 10:21:18,563: INFO/job_scheduler]   Job 50 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:18,563: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 10:21:18,564: INFO/job_scheduler]   Job 52 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:21:18,573: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 10:21:18,579: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com restart_agent {}
[2017-05-10 10:21:18,579: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:21:18,580: INFO/job_scheduler]   Job 42 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 10:21:18,588: INFO/job_scheduler]   Job 43 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:18,588: INFO/job_scheduler]   Job 44 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:18,589: INFO/job_scheduler]   Job 45 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:18,589: INFO/job_scheduler]   Job 46 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:18,589: INFO/job_scheduler]   Job 50 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:18,589: INFO/job_scheduler]   Job 52 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:21:18,590: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 10:21:18,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b026641f-0847-4820-b5a4-c2e8c781257d
[2017-05-10 10:21:18,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b026641f-0847-4820-b5a4-c2e8c781257d
[2017-05-10 10:21:18,675: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/cd966cf6-f44e-4de0-85e6-258297056d06
[2017-05-10 10:21:18,675: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/cd966cf6-f44e-4de0-85e6-258297056d06
[2017-05-10 10:21:18,675: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/cd966cf6-f44e-4de0-85e6-258297056d06
[2017-05-10 10:21:18,675: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/cd966cf6-f44e-4de0-85e6-258297056d06
[2017-05-10 10:21:18,676: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b026641f-0847-4820-b5a4-c2e8c781257d
[2017-05-10 10:21:18,676: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b026641f-0847-4820-b5a4-c2e8c781257d
[2017-05-10 10:21:18,709: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b026641f-0847-4820-b5a4-c2e8c781257d
[2017-05-10 10:21:18,709: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b026641f-0847-4820-b5a4-c2e8c781257d
[2017-05-10 10:21:18,710: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm6.lotus.hpdd.lab.intel.com (old cd966cf6-f44e-4de0-85e6-258297056d06)
[2017-05-10 10:21:18,710: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm6.lotus.hpdd.lab.intel.com (old cd966cf6-f44e-4de0-85e6-258297056d06)
[2017-05-10 10:21:21,199: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/c97abb29-f77e-4e47-9fc3-cfacbdded707
[2017-05-10 10:21:21,199: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/c97abb29-f77e-4e47-9fc3-cfacbdded707
[2017-05-10 10:21:21,200: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/c97abb29-f77e-4e47-9fc3-cfacbdded707
[2017-05-10 10:21:21,200: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/c97abb29-f77e-4e47-9fc3-cfacbdded707
[2017-05-10 10:21:21,200: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 356e7ef3-e162-41a7-9acd-98cf49d1b7fc
[2017-05-10 10:21:21,200: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 356e7ef3-e162-41a7-9acd-98cf49d1b7fc
[2017-05-10 10:21:21,228: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 356e7ef3-e162-41a7-9acd-98cf49d1b7fc
[2017-05-10 10:21:21,228: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 356e7ef3-e162-41a7-9acd-98cf49d1b7fc
[2017-05-10 10:21:21,229: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-05-10 10:21:21.229085+00:00:
[2017-05-10 10:21:21,229: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-05-10 10:21:21.229085+00:00:
[2017-05-10 10:21:21,229: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 10:21:21,229: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 10:21:21,244: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:21:21,244: INFO/job_scheduler]   Job 31 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 10:21:21,244: INFO/job_scheduler]   Job 32 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:21,245: INFO/job_scheduler]   Job 33 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:21,247: INFO/job_scheduler]   Job 34 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:21,247: INFO/job_scheduler]   Job 35 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:21,247: INFO/job_scheduler]   Job 38 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:21,248: INFO/job_scheduler]   Job 41 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:21:21,250: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-05-10 10:21:21.250518+00:00:
[2017-05-10 10:21:21,250: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-05-10 10:21:21.250518+00:00:
[2017-05-10 10:21:21,252: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 10:21:21,252: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 10:21:21,261: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 10:21:21,267: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:21:21,268: INFO/job_scheduler]   Job 31 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 10:21:21,268: INFO/job_scheduler]   Job 32 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:21,268: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com restart_agent {}
[2017-05-10 10:21:21,269: INFO/job_scheduler]   Job 33 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:21,278: INFO/job_scheduler]   Job 34 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:21,279: INFO/job_scheduler]   Job 35 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:21,279: INFO/job_scheduler]   Job 38 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:21,280: INFO/job_scheduler]   Job 41 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:21:21,281: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 10:21:21,286: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cc4b2937-ea57-4426-8401-c1612d3acbac
[2017-05-10 10:21:21,286: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cc4b2937-ea57-4426-8401-c1612d3acbac
[2017-05-10 10:21:21,368: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/c97abb29-f77e-4e47-9fc3-cfacbdded707
[2017-05-10 10:21:21,368: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/c97abb29-f77e-4e47-9fc3-cfacbdded707
[2017-05-10 10:21:21,368: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/c97abb29-f77e-4e47-9fc3-cfacbdded707
[2017-05-10 10:21:21,368: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/c97abb29-f77e-4e47-9fc3-cfacbdded707
[2017-05-10 10:21:21,369: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cc4b2937-ea57-4426-8401-c1612d3acbac
[2017-05-10 10:21:21,369: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cc4b2937-ea57-4426-8401-c1612d3acbac
[2017-05-10 10:21:21,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cc4b2937-ea57-4426-8401-c1612d3acbac
[2017-05-10 10:21:21,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cc4b2937-ea57-4426-8401-c1612d3acbac
[2017-05-10 10:21:21,400: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm8.lotus.hpdd.lab.intel.com (old c97abb29-f77e-4e47-9fc3-cfacbdded707)
[2017-05-10 10:21:21,400: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm8.lotus.hpdd.lab.intel.com (old c97abb29-f77e-4e47-9fc3-cfacbdded707)
[2017-05-10 10:21:43,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/4ac07e17-3406-4bcb-8fdc-4a9d2a2da765
[2017-05-10 10:21:43,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/4ac07e17-3406-4bcb-8fdc-4a9d2a2da765
[2017-05-10 10:21:43,271: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 4ac07e17-3406-4bcb-8fdc-4a9d2a2da765 because session terminated
[2017-05-10 10:21:43,271: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 4ac07e17-3406-4bcb-8fdc-4a9d2a2da765 because session terminated
[2017-05-10 10:21:43,273: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:21:43,273: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:21:43,313: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm7.lotus.hpdd.lab.intel.com new c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:21:43,313: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm7.lotus.hpdd.lab.intel.com new c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:21:43,355: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 10:21:43,374: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 46ac9ef4-6abb-4ac5-a761-996a51080368
[2017-05-10 10:21:43,374: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 46ac9ef4-6abb-4ac5-a761-996a51080368
[2017-05-10 10:21:43,488: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/62d70472-58ab-4e93-8d14-6d9727df060a
[2017-05-10 10:21:43,488: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/62d70472-58ab-4e93-8d14-6d9727df060a
[2017-05-10 10:21:43,489: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 62d70472-58ab-4e93-8d14-6d9727df060a because session terminated
[2017-05-10 10:21:43,489: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 62d70472-58ab-4e93-8d14-6d9727df060a because session terminated
[2017-05-10 10:21:43,492: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:21:43,492: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:21:43,498: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:21:43,498: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:21:43,498: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:21:43,498: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:21:43,498: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 46ac9ef4-6abb-4ac5-a761-996a51080368
[2017-05-10 10:21:43,498: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 46ac9ef4-6abb-4ac5-a761-996a51080368
[2017-05-10 10:21:43,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 46ac9ef4-6abb-4ac5-a761-996a51080368
[2017-05-10 10:21:43,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 46ac9ef4-6abb-4ac5-a761-996a51080368
[2017-05-10 10:21:43,540: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-05-10 10:21:43,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 23aece51-e73d-4100-8fd1-f1f545e3298e
[2017-05-10 10:21:43,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 23aece51-e73d-4100-8fd1-f1f545e3298e
[2017-05-10 10:21:43,850: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new 04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:21:43,850: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new 04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:21:43,852: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 10:21:43,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6e7df222-e7b8-437c-b125-1f503f565027
[2017-05-10 10:21:43,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6e7df222-e7b8-437c-b125-1f503f565027
[2017-05-10 10:21:43,973: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:21:43,973: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:21:43,973: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:21:43,973: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:21:43,974: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6e7df222-e7b8-437c-b125-1f503f565027
[2017-05-10 10:21:43,974: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6e7df222-e7b8-437c-b125-1f503f565027
[2017-05-10 10:21:43,984: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6e7df222-e7b8-437c-b125-1f503f565027
[2017-05-10 10:21:43,984: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6e7df222-e7b8-437c-b125-1f503f565027
[2017-05-10 10:21:43,984: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-05-10 10:21:43,998: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 42bc93c5-19bf-4979-9a2a-9f1127d3c158
[2017-05-10 10:21:43,998: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 42bc93c5-19bf-4979-9a2a-9f1127d3c158
[2017-05-10 10:21:45,513: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/cd966cf6-f44e-4de0-85e6-258297056d06
[2017-05-10 10:21:45,513: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/cd966cf6-f44e-4de0-85e6-258297056d06
[2017-05-10 10:21:45,513: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session cd966cf6-f44e-4de0-85e6-258297056d06 because session terminated
[2017-05-10 10:21:45,513: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session cd966cf6-f44e-4de0-85e6-258297056d06 because session terminated
[2017-05-10 10:21:45,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:21:45,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:21:45,737: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm6.lotus.hpdd.lab.intel.com new 37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:21:45,737: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm6.lotus.hpdd.lab.intel.com new 37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:21:45,738: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 10:21:45,754: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 10d37304-7847-4583-9ea1-30de58dd0f90
[2017-05-10 10:21:45,754: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 10d37304-7847-4583-9ea1-30de58dd0f90
[2017-05-10 10:21:45,840: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:21:45,840: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:21:45,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:21:45,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:21:45,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 10d37304-7847-4583-9ea1-30de58dd0f90
[2017-05-10 10:21:45,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 10d37304-7847-4583-9ea1-30de58dd0f90
[2017-05-10 10:21:45,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 10d37304-7847-4583-9ea1-30de58dd0f90
[2017-05-10 10:21:45,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 10d37304-7847-4583-9ea1-30de58dd0f90
[2017-05-10 10:21:45,869: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-05-10 10:21:45,889: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a07ce5e3-8b89-4964-a438-452ae36f0c10
[2017-05-10 10:21:45,889: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a07ce5e3-8b89-4964-a438-452ae36f0c10
[2017-05-10 10:21:49,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/c97abb29-f77e-4e47-9fc3-cfacbdded707
[2017-05-10 10:21:49,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/c97abb29-f77e-4e47-9fc3-cfacbdded707
[2017-05-10 10:21:49,123: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session c97abb29-f77e-4e47-9fc3-cfacbdded707 because session terminated
[2017-05-10 10:21:49,123: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session c97abb29-f77e-4e47-9fc3-cfacbdded707 because session terminated
[2017-05-10 10:21:49,126: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:21:49,126: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:21:49,429: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm8.lotus.hpdd.lab.intel.com new aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:21:49,429: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm8.lotus.hpdd.lab.intel.com new aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:21:49,430: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 10:21:49,443: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0deb581a-19a2-4461-a91d-b3e7444c2310
[2017-05-10 10:21:49,443: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0deb581a-19a2-4461-a91d-b3e7444c2310
[2017-05-10 10:21:50,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:21:50,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:21:50,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:21:50,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:21:50,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0deb581a-19a2-4461-a91d-b3e7444c2310
[2017-05-10 10:21:50,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0deb581a-19a2-4461-a91d-b3e7444c2310
[2017-05-10 10:21:50,557: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0deb581a-19a2-4461-a91d-b3e7444c2310
[2017-05-10 10:21:50,557: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0deb581a-19a2-4461-a91d-b3e7444c2310
[2017-05-10 10:21:50,558: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-05-10 10:21:50,569: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ec5356ac-94a9-4da0-bf42-bfeb4afdb624
[2017-05-10 10:21:50,569: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ec5356ac-94a9-4da0-bf42-bfeb4afdb624
[2017-05-10 10:21:52,361: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:21:52,361: INFO/job_scheduler]   Job 20 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 10:21:52,361: INFO/job_scheduler]   Job 21 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:52,361: INFO/job_scheduler]   Job 22 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:52,361: INFO/job_scheduler]   Job 23 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:52,361: INFO/job_scheduler]   Job 24 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:52,361: INFO/job_scheduler]   Job 27 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:52,362: INFO/job_scheduler]   Job 30 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:21:52,362: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 10:21:53,580: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:21:53,580: INFO/job_scheduler]   Job 9 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 10:21:53,580: INFO/job_scheduler]   Job 10 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:53,580: INFO/job_scheduler]   Job 11 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:53,580: INFO/job_scheduler]   Job 12 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:53,580: INFO/job_scheduler]   Job 13 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:53,580: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:53,581: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:21:53,581: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 10:21:55,577: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:21:55,578: INFO/job_scheduler]   Job 42 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 10:21:55,578: INFO/job_scheduler]   Job 43 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:55,578: INFO/job_scheduler]   Job 44 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:55,578: INFO/job_scheduler]   Job 45 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:55,578: INFO/job_scheduler]   Job 46 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:55,578: INFO/job_scheduler]   Job 50 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:21:55,579: INFO/job_scheduler]   Job 52 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:21:55,579: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 10:22:00,196: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:22:00,196: INFO/job_scheduler]   Job 31 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 10:22:00,196: INFO/job_scheduler]   Job 32 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:22:00,197: INFO/job_scheduler]   Job 33 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:22:00,197: INFO/job_scheduler]   Job 34 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:22:00,197: INFO/job_scheduler]   Job 35 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:22:00,197: INFO/job_scheduler]   Job 38 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:22:00,197: INFO/job_scheduler]   Job 41 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:22:00,198: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 10:22:25,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:25,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:25,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:25,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:25,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 23aece51-e73d-4100-8fd1-f1f545e3298e
[2017-05-10 10:22:25,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 23aece51-e73d-4100-8fd1-f1f545e3298e
[2017-05-10 10:22:25,562: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 23aece51-e73d-4100-8fd1-f1f545e3298e
[2017-05-10 10:22:25,562: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 23aece51-e73d-4100-8fd1-f1f545e3298e
[2017-05-10 10:22:25,939: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com install_packages {'repos': [u'lustre', u'e2fsprogs', u'zfs', u'iml-agent'], 'packages': [u'chroma-agent-management', u'kernel-devel-3.10.0-514.10.2.el7_lustre', u'lustre-osd-zfs', u'lustre-osd-ldiskfs', u'lustre-modules', u'lustre', u'zfs']}
[2017-05-10 10:22:25,966: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bc6e9661-7935-42b7-8b1a-7f509279a638
[2017-05-10 10:22:25,966: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bc6e9661-7935-42b7-8b1a-7f509279a638
[2017-05-10 10:22:27,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:27,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:27,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:27,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:27,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a07ce5e3-8b89-4964-a438-452ae36f0c10
[2017-05-10 10:22:27,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a07ce5e3-8b89-4964-a438-452ae36f0c10
[2017-05-10 10:22:27,616: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a07ce5e3-8b89-4964-a438-452ae36f0c10
[2017-05-10 10:22:27,616: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a07ce5e3-8b89-4964-a438-452ae36f0c10
[2017-05-10 10:22:28,195: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com install_packages {'repos': [u'lustre', u'e2fsprogs', u'zfs', u'iml-agent'], 'packages': [u'chroma-agent-management', u'kernel-devel-3.10.0-514.10.2.el7_lustre', u'lustre-osd-zfs', u'lustre-osd-ldiskfs', u'lustre-modules', u'lustre', u'zfs']}
[2017-05-10 10:22:28,215: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d0a0b9c3-30fe-42a0-a1fb-2704f42c3d54
[2017-05-10 10:22:28,215: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d0a0b9c3-30fe-42a0-a1fb-2704f42c3d54
[2017-05-10 10:22:30,674: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:22:30,674: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:22:30,675: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:22:30,675: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:22:30,675: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 42bc93c5-19bf-4979-9a2a-9f1127d3c158
[2017-05-10 10:22:30,675: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 42bc93c5-19bf-4979-9a2a-9f1127d3c158
[2017-05-10 10:22:30,675: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 42bc93c5-19bf-4979-9a2a-9f1127d3c158
[2017-05-10 10:22:30,675: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 42bc93c5-19bf-4979-9a2a-9f1127d3c158
[2017-05-10 10:22:31,398: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com install_packages {'repos': [u'lustre', u'e2fsprogs', u'zfs', u'iml-agent'], 'packages': [u'chroma-agent-management', u'kernel-devel-3.10.0-514.10.2.el7_lustre', u'lustre-osd-zfs', u'lustre-osd-ldiskfs', u'lustre-modules', u'lustre', u'zfs']}
[2017-05-10 10:22:31,410: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4a62097f-7bbc-4754-a199-b1ad8c87ca18
[2017-05-10 10:22:31,410: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4a62097f-7bbc-4754-a199-b1ad8c87ca18
[2017-05-10 10:22:34,199: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:22:34,199: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:22:34,200: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:22:34,200: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:22:34,200: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ec5356ac-94a9-4da0-bf42-bfeb4afdb624
[2017-05-10 10:22:34,200: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ec5356ac-94a9-4da0-bf42-bfeb4afdb624
[2017-05-10 10:22:34,245: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ec5356ac-94a9-4da0-bf42-bfeb4afdb624
[2017-05-10 10:22:34,245: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ec5356ac-94a9-4da0-bf42-bfeb4afdb624
[2017-05-10 10:22:35,331: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com install_packages {'repos': [u'lustre', u'e2fsprogs', u'zfs', u'iml-agent'], 'packages': [u'chroma-agent-management', u'kernel-devel-3.10.0-514.10.2.el7_lustre', u'lustre-osd-zfs', u'lustre-osd-ldiskfs', u'lustre-modules', u'lustre', u'zfs']}
[2017-05-10 10:22:35,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d6a68266-4d90-455e-9d78-d91b22a2fb01
[2017-05-10 10:22:35,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d6a68266-4d90-455e-9d78-d91b22a2fb01
[2017-05-10 10:22:45,519: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:45,519: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:45,520: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:45,520: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:45,520: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bc6e9661-7935-42b7-8b1a-7f509279a638
[2017-05-10 10:22:45,520: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bc6e9661-7935-42b7-8b1a-7f509279a638
[2017-05-10 10:22:45,535: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bc6e9661-7935-42b7-8b1a-7f509279a638
[2017-05-10 10:22:45,535: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bc6e9661-7935-42b7-8b1a-7f509279a638
[2017-05-10 10:22:46,079: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com kernel_status {}
[2017-05-10 10:22:46,091: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 79c62960-495e-4bf6-84ab-496b717156cc
[2017-05-10 10:22:46,091: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 79c62960-495e-4bf6-84ab-496b717156cc
[2017-05-10 10:22:49,004: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:49,004: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:49,005: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:49,005: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:49,005: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 79c62960-495e-4bf6-84ab-496b717156cc
[2017-05-10 10:22:49,005: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 79c62960-495e-4bf6-84ab-496b717156cc
[2017-05-10 10:22:49,007: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 79c62960-495e-4bf6-84ab-496b717156cc
[2017-05-10 10:22:49,007: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 79c62960-495e-4bf6-84ab-496b717156cc
[2017-05-10 10:22:49,008: INFO/job_scheduler] Job 20 finished 4 steps successfully
[2017-05-10 10:22:49,250: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 140516297613520
[2017-05-10 10:22:49,255: INFO/job] Job 20: StateChangeJob complete, setting state packages_installed on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:22:49,256: INFO/job] Job 20: not_deleted=True
[2017-05-10 10:22:49,256: INFO/job_scheduler] Job 20: complete_job: Updating cache
[2017-05-10 10:22:49,256: INFO/job_scheduler] Job 20: locked_item 140516297766352 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 10:22:49,259: INFO/job_scheduler] Job 20 completing (errored=False, cancelled=False)
[2017-05-10 10:22:49,299: INFO/job_scheduler] run_next: 4 runnable jobs of (40 pending, 3 tasked)
[2017-05-10 10:22:49,319: INFO/job_scheduler] Job 21: RunJobThread.run
[2017-05-10 10:22:49,319: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-05-10 10:22:49,328: INFO/job_scheduler] Job 22: RunJobThread.run
[2017-05-10 10:22:49,329: INFO/job_scheduler] Job 24: open DB connection during completion
[2017-05-10 10:22:49,329: INFO/job_scheduler] Job 23: RunJobThread.run
[2017-05-10 10:22:49,332: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-05-10 10:22:49,334: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-05-10 10:22:49,337: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7bae4088-0678-4160-a4ed-39b4d81b18f4
[2017-05-10 10:22:49,337: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7bae4088-0678-4160-a4ed-39b4d81b18f4
[2017-05-10 10:22:49,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ed1aebcb-1435-430a-8854-70ef489465e5
[2017-05-10 10:22:49,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ed1aebcb-1435-430a-8854-70ef489465e5
[2017-05-10 10:22:49,364: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2253aacf-a2f9-47d9-ba59-951c21ce3fa2
[2017-05-10 10:22:49,364: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2253aacf-a2f9-47d9-ba59-951c21ce3fa2
[2017-05-10 10:22:49,634: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 140516499187152
[2017-05-10 10:22:49,642: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 10:22:49,646: INFO/job] Job 24: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 10:22:49,648: INFO/job] Job 24: not_deleted=True
[2017-05-10 10:22:49,649: INFO/job_scheduler] Job 24: complete_job: Updating cache
[2017-05-10 10:22:49,649: INFO/job_scheduler] Job 24: locked_item 140516499187152 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 10:22:49,657: INFO/job_scheduler] Job 24 completing (errored=False, cancelled=False)
[2017-05-10 10:22:49,699: INFO/job_scheduler] run_next: 1 runnable jobs of (36 pending, 6 tasked)
[2017-05-10 10:22:49,704: INFO/job_scheduler] Job 25: RunJobThread.run
[2017-05-10 10:22:49,705: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com load_lnet {}
[2017-05-10 10:22:49,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b0a45609-688b-4163-9fbe-aa5dcb7e2638
[2017-05-10 10:22:49,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b0a45609-688b-4163-9fbe-aa5dcb7e2638
[2017-05-10 10:22:49,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:49,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:49,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:49,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:49,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d0a0b9c3-30fe-42a0-a1fb-2704f42c3d54
[2017-05-10 10:22:49,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d0a0b9c3-30fe-42a0-a1fb-2704f42c3d54
[2017-05-10 10:22:49,748: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d0a0b9c3-30fe-42a0-a1fb-2704f42c3d54
[2017-05-10 10:22:49,748: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d0a0b9c3-30fe-42a0-a1fb-2704f42c3d54
[2017-05-10 10:22:50,190: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com kernel_status {}
[2017-05-10 10:22:50,205: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3f7ad3e8-da75-4be9-90ad-835a57dac7f8
[2017-05-10 10:22:50,205: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3f7ad3e8-da75-4be9-90ad-835a57dac7f8
[2017-05-10 10:22:50,830: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:50,830: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:50,830: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:50,830: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:50,830: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b0a45609-688b-4163-9fbe-aa5dcb7e2638
[2017-05-10 10:22:50,830: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b0a45609-688b-4163-9fbe-aa5dcb7e2638
[2017-05-10 10:22:50,833: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b0a45609-688b-4163-9fbe-aa5dcb7e2638
[2017-05-10 10:22:50,833: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b0a45609-688b-4163-9fbe-aa5dcb7e2638
[2017-05-10 10:22:50,834: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 10:22:50,848: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e16d7761-e9b4-42f3-8357-ca45486fe983
[2017-05-10 10:22:50,848: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e16d7761-e9b4-42f3-8357-ca45486fe983
[2017-05-10 10:22:50,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:50,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:50,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:50,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:50,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e16d7761-e9b4-42f3-8357-ca45486fe983
[2017-05-10 10:22:50,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e16d7761-e9b4-42f3-8357-ca45486fe983
[2017-05-10 10:22:50,912: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e16d7761-e9b4-42f3-8357-ca45486fe983
[2017-05-10 10:22:50,912: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e16d7761-e9b4-42f3-8357-ca45486fe983
[2017-05-10 10:22:51,639: INFO/job_scheduler] Job 25 finished 2 steps successfully
[2017-05-10 10:22:51,684: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 140516499187152
[2017-05-10 10:22:51,690: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 10:22:51,697: INFO/job] Job 25: StateChangeJob complete, setting state lnet_down on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 10:22:51,698: INFO/job] Job 25: not_deleted=True
[2017-05-10 10:22:51,698: INFO/job_scheduler] Job 25: complete_job: Updating cache
[2017-05-10 10:22:51,699: INFO/job_scheduler] Job 25: locked_item 140516499187152 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 10:22:51,709: INFO/job_scheduler] Job 25 completing (errored=False, cancelled=False)
[2017-05-10 10:22:51,923: INFO/job_scheduler] run_next: 1 runnable jobs of (35 pending, 6 tasked)
[2017-05-10 10:22:51,927: INFO/job_scheduler] Job 28: RunJobThread.run
[2017-05-10 10:22:51,929: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_lnet {}
[2017-05-10 10:22:51,941: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f1834cc5-5e23-453b-880e-80bd535b93bf
[2017-05-10 10:22:51,941: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f1834cc5-5e23-453b-880e-80bd535b93bf
[2017-05-10 10:22:53,300: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:53,300: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:53,301: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:53,301: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:53,301: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3f7ad3e8-da75-4be9-90ad-835a57dac7f8
[2017-05-10 10:22:53,301: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3f7ad3e8-da75-4be9-90ad-835a57dac7f8
[2017-05-10 10:22:53,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3f7ad3e8-da75-4be9-90ad-835a57dac7f8
[2017-05-10 10:22:53,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3f7ad3e8-da75-4be9-90ad-835a57dac7f8
[2017-05-10 10:22:53,322: INFO/job_scheduler] Job 42 finished 4 steps successfully
[2017-05-10 10:22:53,383: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 140516499040912
[2017-05-10 10:22:53,387: INFO/job] Job 42: StateChangeJob complete, setting state packages_installed on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:22:53,387: INFO/job] Job 42: not_deleted=True
[2017-05-10 10:22:53,387: INFO/job_scheduler] Job 42: complete_job: Updating cache
[2017-05-10 10:22:53,388: INFO/job_scheduler] Job 42: locked_item 140516499244368 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 10:22:53,390: INFO/job_scheduler] Job 42 completing (errored=False, cancelled=False)
[2017-05-10 10:22:53,455: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:53,455: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:53,455: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:53,455: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:53,455: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f1834cc5-5e23-453b-880e-80bd535b93bf
[2017-05-10 10:22:53,455: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f1834cc5-5e23-453b-880e-80bd535b93bf
[2017-05-10 10:22:53,456: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f1834cc5-5e23-453b-880e-80bd535b93bf
[2017-05-10 10:22:53,456: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f1834cc5-5e23-453b-880e-80bd535b93bf
[2017-05-10 10:22:53,457: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 10:22:53,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b049893b-1e21-4204-a8c2-10b163b3c1b1
[2017-05-10 10:22:53,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b049893b-1e21-4204-a8c2-10b163b3c1b1
[2017-05-10 10:22:53,522: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:53,522: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:53,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:53,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:53,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b049893b-1e21-4204-a8c2-10b163b3c1b1
[2017-05-10 10:22:53,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b049893b-1e21-4204-a8c2-10b163b3c1b1
[2017-05-10 10:22:53,534: INFO/job_scheduler] run_next: 4 runnable jobs of (34 pending, 6 tasked)
[2017-05-10 10:22:53,537: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b049893b-1e21-4204-a8c2-10b163b3c1b1
[2017-05-10 10:22:53,537: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b049893b-1e21-4204-a8c2-10b163b3c1b1
[2017-05-10 10:22:53,561: INFO/job_scheduler] Job 43: RunJobThread.run
[2017-05-10 10:22:53,562: INFO/job_scheduler] Job 44: open DB connection during completion
[2017-05-10 10:22:53,565: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-05-10 10:22:53,575: INFO/job_scheduler] Job 45: RunJobThread.run
[2017-05-10 10:22:53,578: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-05-10 10:22:53,580: INFO/job_scheduler] Job 46: RunJobThread.run
[2017-05-10 10:22:53,586: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-05-10 10:22:53,591: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 01a9829d-0bdf-4fb5-b52c-2fffb911a90f
[2017-05-10 10:22:53,591: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 01a9829d-0bdf-4fb5-b52c-2fffb911a90f
[2017-05-10 10:22:53,606: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7b5b4e7c-43ba-49ae-b133-c448fd76f05f
[2017-05-10 10:22:53,606: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7b5b4e7c-43ba-49ae-b133-c448fd76f05f
[2017-05-10 10:22:53,631: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7b3fdfe4-b7c6-45ac-8db4-849cd2eea5e8
[2017-05-10 10:22:53,631: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7b3fdfe4-b7c6-45ac-8db4-849cd2eea5e8
[2017-05-10 10:22:53,807: INFO/job_scheduler] Job 28 finished 2 steps successfully
[2017-05-10 10:22:53,875: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 140516499068560
[2017-05-10 10:22:53,884: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 10:22:53,889: INFO/job] Job 44: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 10:22:53,890: INFO/job] Job 44: not_deleted=True
[2017-05-10 10:22:53,891: INFO/job_scheduler] Job 44: complete_job: Updating cache
[2017-05-10 10:22:53,892: INFO/job_scheduler] Job 44: locked_item 140516499068560 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 10:22:53,899: INFO/job_scheduler] Job 44 completing (errored=False, cancelled=False)
[2017-05-10 10:22:53,948: INFO/job_scheduler] run_next: 1 runnable jobs of (30 pending, 9 tasked)
[2017-05-10 10:22:53,953: INFO/job_scheduler] Job 48: RunJobThread.run
[2017-05-10 10:22:53,954: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com load_lnet {}
[2017-05-10 10:22:53,971: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 68e9aba8-bf47-4dc0-bb19-5591141a9779
[2017-05-10 10:22:53,971: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 68e9aba8-bf47-4dc0-bb19-5591141a9779
[2017-05-10 10:22:54,056: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 140516499187152
[2017-05-10 10:22:54,061: INFO/job] Job 28: StateChangeJob complete, setting state lnet_up on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 10:22:54,061: INFO/job] Job 28: not_deleted=True
[2017-05-10 10:22:54,061: INFO/job_scheduler] Job 28: complete_job: Updating cache
[2017-05-10 10:22:54,061: INFO/job_scheduler] Job 28: locked_item 140516499187152 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 10:22:54,064: INFO/job_scheduler] Job 28 completing (errored=False, cancelled=False)
[2017-05-10 10:22:54,089: INFO/job_scheduler] run_next: 0 runnable jobs of (29 pending, 9 tasked)
[2017-05-10 10:22:54,150: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:54,150: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:54,150: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:54,150: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:54,150: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 68e9aba8-bf47-4dc0-bb19-5591141a9779
[2017-05-10 10:22:54,150: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 68e9aba8-bf47-4dc0-bb19-5591141a9779
[2017-05-10 10:22:54,186: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 68e9aba8-bf47-4dc0-bb19-5591141a9779
[2017-05-10 10:22:54,186: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 68e9aba8-bf47-4dc0-bb19-5591141a9779
[2017-05-10 10:22:54,187: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 10:22:54,202: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8ff023c8-7a2d-4c9f-b0a1-a8c6e91a81b4
[2017-05-10 10:22:54,202: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8ff023c8-7a2d-4c9f-b0a1-a8c6e91a81b4
[2017-05-10 10:22:54,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:54,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:54,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:54,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:54,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8ff023c8-7a2d-4c9f-b0a1-a8c6e91a81b4
[2017-05-10 10:22:54,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8ff023c8-7a2d-4c9f-b0a1-a8c6e91a81b4
[2017-05-10 10:22:54,318: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8ff023c8-7a2d-4c9f-b0a1-a8c6e91a81b4
[2017-05-10 10:22:54,318: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8ff023c8-7a2d-4c9f-b0a1-a8c6e91a81b4
[2017-05-10 10:22:54,765: INFO/job_scheduler] Job 48 finished 2 steps successfully
[2017-05-10 10:22:54,814: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 140516499068560
[2017-05-10 10:22:54,820: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 10:22:54,834: INFO/job] Job 48: StateChangeJob complete, setting state lnet_down on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 10:22:54,835: INFO/job] Job 48: not_deleted=True
[2017-05-10 10:22:54,837: INFO/job_scheduler] Job 48: complete_job: Updating cache
[2017-05-10 10:22:54,838: INFO/job_scheduler] Job 48: locked_item 140516499068560 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 10:22:54,840: INFO/job_scheduler] Job 48 completing (errored=False, cancelled=False)
[2017-05-10 10:22:54,870: INFO/job_scheduler] run_next: 1 runnable jobs of (29 pending, 8 tasked)
[2017-05-10 10:22:54,874: INFO/job_scheduler] Job 49: RunJobThread.run
[2017-05-10 10:22:54,875: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_lnet {}
[2017-05-10 10:22:54,888: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 43fe0d92-9292-4803-9dd0-d8a324476097
[2017-05-10 10:22:54,888: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 43fe0d92-9292-4803-9dd0-d8a324476097
[2017-05-10 10:22:55,121: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:22:55,121: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:22:55,122: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:22:55,122: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:22:55,122: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4a62097f-7bbc-4754-a199-b1ad8c87ca18
[2017-05-10 10:22:55,122: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4a62097f-7bbc-4754-a199-b1ad8c87ca18
[2017-05-10 10:22:55,134: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4a62097f-7bbc-4754-a199-b1ad8c87ca18
[2017-05-10 10:22:55,134: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4a62097f-7bbc-4754-a199-b1ad8c87ca18
[2017-05-10 10:22:55,477: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:55,477: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:55,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:55,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:55,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ed1aebcb-1435-430a-8854-70ef489465e5
[2017-05-10 10:22:55,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ed1aebcb-1435-430a-8854-70ef489465e5
[2017-05-10 10:22:55,482: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ed1aebcb-1435-430a-8854-70ef489465e5
[2017-05-10 10:22:55,482: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ed1aebcb-1435-430a-8854-70ef489465e5
[2017-05-10 10:22:55,484: INFO/job_scheduler] Job 22 finished 1 steps successfully
[2017-05-10 10:22:55,590: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 140516499187216
[2017-05-10 10:22:55,596: INFO/job] Job 22: StateChangeJob complete, setting state configured on lotus-32vm7.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-10 10:22:55,600: INFO/job] Job 22: not_deleted=True
[2017-05-10 10:22:55,601: INFO/job_scheduler] Job 22: complete_job: Updating cache
[2017-05-10 10:22:55,604: INFO/job_scheduler] Job 22: locked_item 140516499187216 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-10 10:22:55,620: INFO/job_scheduler] Job 22 completing (errored=False, cancelled=False)
[2017-05-10 10:22:55,664: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com kernel_status {}
[2017-05-10 10:22:55,682: INFO/job_scheduler] run_next: 0 runnable jobs of (28 pending, 8 tasked)
[2017-05-10 10:22:55,705: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b8284cc8-b116-4c2e-bc6a-e668ba90de6e
[2017-05-10 10:22:55,705: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b8284cc8-b116-4c2e-bc6a-e668ba90de6e
[2017-05-10 10:22:56,546: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:56,546: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:56,547: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:56,547: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:22:56,547: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7bae4088-0678-4160-a4ed-39b4d81b18f4
[2017-05-10 10:22:56,547: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7bae4088-0678-4160-a4ed-39b4d81b18f4
[2017-05-10 10:22:56,565: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7bae4088-0678-4160-a4ed-39b4d81b18f4
[2017-05-10 10:22:56,565: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7bae4088-0678-4160-a4ed-39b4d81b18f4
[2017-05-10 10:22:56,565: INFO/job_scheduler] Job 21 finished 1 steps successfully
[2017-05-10 10:22:56,698: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 140516499186896
[2017-05-10 10:22:56,702: INFO/job] Job 21: StateChangeJob complete, setting state configured on lotus-32vm7.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-10 10:22:56,702: INFO/job] Job 21: not_deleted=True
[2017-05-10 10:22:56,703: INFO/job_scheduler] Job 21: complete_job: Updating cache
[2017-05-10 10:22:56,703: INFO/job_scheduler] Job 21: locked_item 140516499186896 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-10 10:22:56,705: INFO/job_scheduler] Job 21 completing (errored=False, cancelled=False)
[2017-05-10 10:22:56,739: INFO/job_scheduler] run_next: 0 runnable jobs of (28 pending, 7 tasked)
[2017-05-10 10:22:57,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:57,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:57,190: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:57,190: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:57,190: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 43fe0d92-9292-4803-9dd0-d8a324476097
[2017-05-10 10:22:57,190: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 43fe0d92-9292-4803-9dd0-d8a324476097
[2017-05-10 10:22:57,205: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 43fe0d92-9292-4803-9dd0-d8a324476097
[2017-05-10 10:22:57,205: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 43fe0d92-9292-4803-9dd0-d8a324476097
[2017-05-10 10:22:57,208: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 10:22:57,226: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b24e86f2-1f64-42d9-a70c-9b258afd7c56
[2017-05-10 10:22:57,226: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b24e86f2-1f64-42d9-a70c-9b258afd7c56
[2017-05-10 10:22:57,403: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:57,403: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:57,403: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:57,403: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:57,403: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b24e86f2-1f64-42d9-a70c-9b258afd7c56
[2017-05-10 10:22:57,403: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b24e86f2-1f64-42d9-a70c-9b258afd7c56
[2017-05-10 10:22:57,450: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b24e86f2-1f64-42d9-a70c-9b258afd7c56
[2017-05-10 10:22:57,450: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b24e86f2-1f64-42d9-a70c-9b258afd7c56
[2017-05-10 10:22:57,494: INFO/job_scheduler] Job 49 finished 2 steps successfully
[2017-05-10 10:22:57,514: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 140516499068560
[2017-05-10 10:22:57,520: INFO/job] Job 49: StateChangeJob complete, setting state lnet_up on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 10:22:57,520: INFO/job] Job 49: not_deleted=True
[2017-05-10 10:22:57,520: INFO/job_scheduler] Job 49: complete_job: Updating cache
[2017-05-10 10:22:57,520: INFO/job_scheduler] Job 49: locked_item 140516499068560 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 10:22:57,523: INFO/job_scheduler] Job 49 completing (errored=False, cancelled=False)
[2017-05-10 10:22:57,546: INFO/job_scheduler] run_next: 0 runnable jobs of (28 pending, 6 tasked)
[2017-05-10 10:22:57,796: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:22:57,796: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:22:57,796: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:22:57,796: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:22:57,796: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b8284cc8-b116-4c2e-bc6a-e668ba90de6e
[2017-05-10 10:22:57,796: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b8284cc8-b116-4c2e-bc6a-e668ba90de6e
[2017-05-10 10:22:57,820: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b8284cc8-b116-4c2e-bc6a-e668ba90de6e
[2017-05-10 10:22:57,820: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b8284cc8-b116-4c2e-bc6a-e668ba90de6e
[2017-05-10 10:22:57,820: INFO/job_scheduler] Job 9 finished 4 steps successfully
[2017-05-10 10:22:57,931: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 140516297760400
[2017-05-10 10:22:57,935: INFO/job] Job 9: StateChangeJob complete, setting state packages_installed on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:22:57,935: INFO/job] Job 9: not_deleted=True
[2017-05-10 10:22:57,936: INFO/job_scheduler] Job 9: complete_job: Updating cache
[2017-05-10 10:22:57,936: INFO/job_scheduler] Job 9: locked_item 140516499180240 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 10:22:57,939: INFO/job_scheduler] Job 9 completing (errored=False, cancelled=False)
[2017-05-10 10:22:57,954: INFO/job_scheduler] run_next: 4 runnable jobs of (28 pending, 5 tasked)
[2017-05-10 10:22:57,973: INFO/job_scheduler] Job 10: RunJobThread.run
[2017-05-10 10:22:57,974: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-05-10 10:22:57,982: INFO/job_scheduler] Job 11: RunJobThread.run
[2017-05-10 10:22:57,982: INFO/job_scheduler] Job 12: open DB connection during completion
[2017-05-10 10:22:57,985: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-05-10 10:22:57,989: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8f29636b-bdff-4cce-b6ed-f7652b82d662
[2017-05-10 10:22:57,989: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8f29636b-bdff-4cce-b6ed-f7652b82d662
[2017-05-10 10:22:58,000: INFO/job_scheduler] Job 13: RunJobThread.run
[2017-05-10 10:22:58,001: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-05-10 10:22:58,006: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b6336212-4844-482a-bf45-ca24eb09a359
[2017-05-10 10:22:58,006: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b6336212-4844-482a-bf45-ca24eb09a359
[2017-05-10 10:22:58,019: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3dc263bf-ac73-40b6-ae58-a662a27a6170
[2017-05-10 10:22:58,019: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3dc263bf-ac73-40b6-ae58-a662a27a6170
[2017-05-10 10:22:58,023: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 140516633538064
[2017-05-10 10:22:58,030: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 10:22:58,041: INFO/job] Job 12: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 10:22:58,042: INFO/job] Job 12: not_deleted=True
[2017-05-10 10:22:58,043: INFO/job_scheduler] Job 12: complete_job: Updating cache
[2017-05-10 10:22:58,043: INFO/job_scheduler] Job 12: locked_item 140516633538064 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 10:22:58,051: INFO/job_scheduler] Job 12 completing (errored=False, cancelled=False)
[2017-05-10 10:22:58,113: INFO/job_scheduler] run_next: 1 runnable jobs of (24 pending, 8 tasked)
[2017-05-10 10:22:58,117: INFO/job_scheduler] Job 14: RunJobThread.run
[2017-05-10 10:22:58,118: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com load_lnet {}
[2017-05-10 10:22:58,135: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1444474a-023f-4340-b085-a43f03b097f4
[2017-05-10 10:22:58,135: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1444474a-023f-4340-b085-a43f03b097f4
[2017-05-10 10:22:58,851: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:58,851: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:58,852: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:58,852: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:58,852: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 01a9829d-0bdf-4fb5-b52c-2fffb911a90f
[2017-05-10 10:22:58,852: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 01a9829d-0bdf-4fb5-b52c-2fffb911a90f
[2017-05-10 10:22:58,866: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 01a9829d-0bdf-4fb5-b52c-2fffb911a90f
[2017-05-10 10:22:58,866: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 01a9829d-0bdf-4fb5-b52c-2fffb911a90f
[2017-05-10 10:22:58,867: INFO/job_scheduler] Job 43 finished 1 steps successfully
[2017-05-10 10:22:58,939: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 140516499070288
[2017-05-10 10:22:58,943: INFO/job] Job 43: StateChangeJob complete, setting state configured on lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-10 10:22:58,943: INFO/job] Job 43: not_deleted=True
[2017-05-10 10:22:58,943: INFO/job_scheduler] Job 43: complete_job: Updating cache
[2017-05-10 10:22:58,943: INFO/job_scheduler] Job 43: locked_item 140516499070288 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-10 10:22:58,946: INFO/job_scheduler] Job 43 completing (errored=False, cancelled=False)
[2017-05-10 10:22:58,963: INFO/job_scheduler] run_next: 0 runnable jobs of (23 pending, 8 tasked)
[2017-05-10 10:22:59,274: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:59,274: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:59,274: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:59,274: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:22:59,274: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7b3fdfe4-b7c6-45ac-8db4-849cd2eea5e8
[2017-05-10 10:22:59,274: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7b3fdfe4-b7c6-45ac-8db4-849cd2eea5e8
[2017-05-10 10:22:59,277: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:22:59,277: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:22:59,277: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:22:59,277: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:22:59,277: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1444474a-023f-4340-b085-a43f03b097f4
[2017-05-10 10:22:59,277: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1444474a-023f-4340-b085-a43f03b097f4
[2017-05-10 10:22:59,299: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1444474a-023f-4340-b085-a43f03b097f4
[2017-05-10 10:22:59,299: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1444474a-023f-4340-b085-a43f03b097f4
[2017-05-10 10:22:59,301: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 10:22:59,301: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7b3fdfe4-b7c6-45ac-8db4-849cd2eea5e8
[2017-05-10 10:22:59,301: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7b3fdfe4-b7c6-45ac-8db4-849cd2eea5e8
[2017-05-10 10:22:59,312: INFO/job_scheduler] Job 46 finished 1 steps successfully
[2017-05-10 10:22:59,319: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 62f20ebc-d4e4-486f-a603-e79ae5b4fb7f
[2017-05-10 10:22:59,319: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 62f20ebc-d4e4-486f-a603-e79ae5b4fb7f
[2017-05-10 10:22:59,382: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:22:59,382: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:22:59,382: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:22:59,382: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:22:59,382: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 62f20ebc-d4e4-486f-a603-e79ae5b4fb7f
[2017-05-10 10:22:59,382: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 62f20ebc-d4e4-486f-a603-e79ae5b4fb7f
[2017-05-10 10:22:59,383: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 62f20ebc-d4e4-486f-a603-e79ae5b4fb7f
[2017-05-10 10:22:59,383: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 62f20ebc-d4e4-486f-a603-e79ae5b4fb7f
[2017-05-10 10:22:59,606: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 140516499067216
[2017-05-10 10:22:59,611: INFO/job] Job 46: StateChangeJob complete, setting state configured on lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-10 10:22:59,611: INFO/job] Job 46: not_deleted=True
[2017-05-10 10:22:59,611: INFO/job_scheduler] Job 46: complete_job: Updating cache
[2017-05-10 10:22:59,611: INFO/job_scheduler] Job 46: locked_item 140516499067216 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-10 10:22:59,615: INFO/job_scheduler] Job 46 completing (errored=False, cancelled=False)
[2017-05-10 10:22:59,638: INFO/job_scheduler] run_next: 0 runnable jobs of (23 pending, 7 tasked)
[2017-05-10 10:22:59,828: INFO/job_scheduler] Job 14 finished 2 steps successfully
[2017-05-10 10:22:59,872: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 140516633538064
[2017-05-10 10:22:59,878: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 10:22:59,884: INFO/job] Job 14: StateChangeJob complete, setting state lnet_down on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 10:22:59,884: INFO/job] Job 14: not_deleted=True
[2017-05-10 10:22:59,885: INFO/job_scheduler] Job 14: complete_job: Updating cache
[2017-05-10 10:22:59,885: INFO/job_scheduler] Job 14: locked_item 140516633538064 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 10:22:59,894: INFO/job_scheduler] Job 14 completing (errored=False, cancelled=False)
[2017-05-10 10:22:59,929: INFO/job_scheduler] run_next: 1 runnable jobs of (23 pending, 6 tasked)
[2017-05-10 10:22:59,933: INFO/job_scheduler] Job 17: RunJobThread.run
[2017-05-10 10:22:59,934: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_lnet {}
[2017-05-10 10:22:59,959: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3c800ddf-f128-4860-8962-34ffa6d1073d
[2017-05-10 10:22:59,959: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3c800ddf-f128-4860-8962-34ffa6d1073d
[2017-05-10 10:23:00,060: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:00,060: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:00,060: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:00,060: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:00,060: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3c800ddf-f128-4860-8962-34ffa6d1073d
[2017-05-10 10:23:00,060: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3c800ddf-f128-4860-8962-34ffa6d1073d
[2017-05-10 10:23:00,073: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3c800ddf-f128-4860-8962-34ffa6d1073d
[2017-05-10 10:23:00,073: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3c800ddf-f128-4860-8962-34ffa6d1073d
[2017-05-10 10:23:00,074: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 10:23:00,090: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 40c00274-590a-443c-82d3-cf4fe46face1
[2017-05-10 10:23:00,090: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 40c00274-590a-443c-82d3-cf4fe46face1
[2017-05-10 10:23:00,181: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:00,181: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:00,181: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:00,181: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:00,181: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 40c00274-590a-443c-82d3-cf4fe46face1
[2017-05-10 10:23:00,181: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 40c00274-590a-443c-82d3-cf4fe46face1
[2017-05-10 10:23:00,207: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 40c00274-590a-443c-82d3-cf4fe46face1
[2017-05-10 10:23:00,207: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 40c00274-590a-443c-82d3-cf4fe46face1
[2017-05-10 10:23:00,440: INFO/job_scheduler] Job 17 finished 2 steps successfully
[2017-05-10 10:23:00,470: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 140516633538064
[2017-05-10 10:23:00,511: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 10:23:00,518: INFO/job] Job 17: StateChangeJob complete, setting state lnet_up on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 10:23:00,518: INFO/job] Job 17: not_deleted=True
[2017-05-10 10:23:00,518: INFO/job_scheduler] Job 17: complete_job: Updating cache
[2017-05-10 10:23:00,519: INFO/job_scheduler] Job 17: locked_item 140516633538064 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 10:23:00,524: INFO/job_scheduler] Job 17 completing (errored=False, cancelled=False)
[2017-05-10 10:23:00,543: INFO/job_scheduler] run_next: 0 runnable jobs of (22 pending, 6 tasked)
[2017-05-10 10:23:00,981: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:00,981: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:00,981: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:00,981: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:00,981: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d6a68266-4d90-455e-9d78-d91b22a2fb01
[2017-05-10 10:23:00,981: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d6a68266-4d90-455e-9d78-d91b22a2fb01
[2017-05-10 10:23:00,983: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d6a68266-4d90-455e-9d78-d91b22a2fb01
[2017-05-10 10:23:00,983: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d6a68266-4d90-455e-9d78-d91b22a2fb01
[2017-05-10 10:23:01,448: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com kernel_status {}
[2017-05-10 10:23:01,463: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 99d94b2f-0725-4ce5-b300-566309ace7d7
[2017-05-10 10:23:01,463: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 99d94b2f-0725-4ce5-b300-566309ace7d7
[2017-05-10 10:23:03,257: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:03,257: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:03,257: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:03,257: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:03,257: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3dc263bf-ac73-40b6-ae58-a662a27a6170
[2017-05-10 10:23:03,257: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3dc263bf-ac73-40b6-ae58-a662a27a6170
[2017-05-10 10:23:03,288: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3dc263bf-ac73-40b6-ae58-a662a27a6170
[2017-05-10 10:23:03,288: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3dc263bf-ac73-40b6-ae58-a662a27a6170
[2017-05-10 10:23:03,289: INFO/job_scheduler] Job 13 finished 1 steps successfully
[2017-05-10 10:23:03,445: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 140516297816720
[2017-05-10 10:23:03,450: INFO/job] Job 13: StateChangeJob complete, setting state configured on lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-10 10:23:03,450: INFO/job] Job 13: not_deleted=True
[2017-05-10 10:23:03,450: INFO/job_scheduler] Job 13: complete_job: Updating cache
[2017-05-10 10:23:03,451: INFO/job_scheduler] Job 13: locked_item 140516297816720 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-10 10:23:03,455: INFO/job_scheduler] Job 13 completing (errored=False, cancelled=False)
[2017-05-10 10:23:03,486: INFO/job_scheduler] run_next: 0 runnable jobs of (22 pending, 5 tasked)
[2017-05-10 10:23:03,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:03,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:03,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:03,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:03,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 99d94b2f-0725-4ce5-b300-566309ace7d7
[2017-05-10 10:23:03,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 99d94b2f-0725-4ce5-b300-566309ace7d7
[2017-05-10 10:23:03,628: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 99d94b2f-0725-4ce5-b300-566309ace7d7
[2017-05-10 10:23:03,628: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 99d94b2f-0725-4ce5-b300-566309ace7d7
[2017-05-10 10:23:03,629: INFO/job_scheduler] Job 31 finished 4 steps successfully
[2017-05-10 10:23:03,737: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 140516499070352
[2017-05-10 10:23:03,742: INFO/job] Job 31: StateChangeJob complete, setting state packages_installed on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:23:03,743: INFO/job] Job 31: not_deleted=True
[2017-05-10 10:23:03,743: INFO/job_scheduler] Job 31: complete_job: Updating cache
[2017-05-10 10:23:03,743: INFO/job_scheduler] Job 31: locked_item 140516499068368 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 10:23:03,747: INFO/job_scheduler] Job 31 completing (errored=False, cancelled=False)
[2017-05-10 10:23:03,922: INFO/job_scheduler] run_next: 4 runnable jobs of (22 pending, 4 tasked)
[2017-05-10 10:23:03,944: INFO/job_scheduler] Job 32: RunJobThread.run
[2017-05-10 10:23:03,946: INFO/job_scheduler] Job 33: RunJobThread.run
[2017-05-10 10:23:03,948: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-05-10 10:23:03,964: INFO/job_scheduler] Job 34: RunJobThread.run
[2017-05-10 10:23:03,966: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-05-10 10:23:03,968: INFO/job_scheduler] Job 35: open DB connection during completion
[2017-05-10 10:23:03,968: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-05-10 10:23:03,974: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc da593318-c3f6-4197-b2bc-8693bacc6a36
[2017-05-10 10:23:03,974: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc da593318-c3f6-4197-b2bc-8693bacc6a36
[2017-05-10 10:23:03,992: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d6104ae9-29f2-4123-9470-c1098d8818d2
[2017-05-10 10:23:03,992: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d6104ae9-29f2-4123-9470-c1098d8818d2
[2017-05-10 10:23:04,011: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 96815398-2183-43d2-842e-1637ad625979
[2017-05-10 10:23:04,011: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 96815398-2183-43d2-842e-1637ad625979
[2017-05-10 10:23:04,054: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 140516499188048
[2017-05-10 10:23:04,063: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 10:23:04,069: INFO/job] Job 35: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 10:23:04,071: INFO/job] Job 35: not_deleted=True
[2017-05-10 10:23:04,071: INFO/job_scheduler] Job 35: complete_job: Updating cache
[2017-05-10 10:23:04,072: INFO/job_scheduler] Job 35: locked_item 140516499188048 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 10:23:04,083: INFO/job_scheduler] Job 35 completing (errored=False, cancelled=False)
[2017-05-10 10:23:04,202: INFO/job_scheduler] run_next: 1 runnable jobs of (18 pending, 7 tasked)
[2017-05-10 10:23:04,207: INFO/job_scheduler] Job 36: RunJobThread.run
[2017-05-10 10:23:04,208: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com load_lnet {}
[2017-05-10 10:23:04,226: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a3c2adc4-13ab-4d5d-aa90-ec411f2faf4f
[2017-05-10 10:23:04,226: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a3c2adc4-13ab-4d5d-aa90-ec411f2faf4f
[2017-05-10 10:23:04,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:04,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:04,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:04,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:04,272: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b6336212-4844-482a-bf45-ca24eb09a359
[2017-05-10 10:23:04,272: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b6336212-4844-482a-bf45-ca24eb09a359
[2017-05-10 10:23:04,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b6336212-4844-482a-bf45-ca24eb09a359
[2017-05-10 10:23:04,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b6336212-4844-482a-bf45-ca24eb09a359
[2017-05-10 10:23:04,283: INFO/job_scheduler] Job 11 finished 1 steps successfully
[2017-05-10 10:23:04,404: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 140516633537360
[2017-05-10 10:23:04,407: INFO/job] Job 11: StateChangeJob complete, setting state configured on lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-10 10:23:04,407: INFO/job] Job 11: not_deleted=True
[2017-05-10 10:23:04,408: INFO/job_scheduler] Job 11: complete_job: Updating cache
[2017-05-10 10:23:04,408: INFO/job_scheduler] Job 11: locked_item 140516633537360 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-10 10:23:04,410: INFO/job_scheduler] Job 11 completing (errored=False, cancelled=False)
[2017-05-10 10:23:04,485: INFO/job_scheduler] run_next: 0 runnable jobs of (17 pending, 7 tasked)
[2017-05-10 10:23:05,312: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:05,312: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:05,313: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:05,313: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:05,313: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a3c2adc4-13ab-4d5d-aa90-ec411f2faf4f
[2017-05-10 10:23:05,313: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a3c2adc4-13ab-4d5d-aa90-ec411f2faf4f
[2017-05-10 10:23:05,340: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a3c2adc4-13ab-4d5d-aa90-ec411f2faf4f
[2017-05-10 10:23:05,340: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a3c2adc4-13ab-4d5d-aa90-ec411f2faf4f
[2017-05-10 10:23:05,341: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 10:23:05,359: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d2e68398-6bad-4d7a-9cf4-e68326ef9de6
[2017-05-10 10:23:05,359: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d2e68398-6bad-4d7a-9cf4-e68326ef9de6
[2017-05-10 10:23:06,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:06,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:06,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:06,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:06,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d2e68398-6bad-4d7a-9cf4-e68326ef9de6
[2017-05-10 10:23:06,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d2e68398-6bad-4d7a-9cf4-e68326ef9de6
[2017-05-10 10:23:06,473: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d2e68398-6bad-4d7a-9cf4-e68326ef9de6
[2017-05-10 10:23:06,473: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d2e68398-6bad-4d7a-9cf4-e68326ef9de6
[2017-05-10 10:23:07,001: INFO/job_scheduler] Job 36 finished 2 steps successfully
[2017-05-10 10:23:07,128: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 140516499188048
[2017-05-10 10:23:07,135: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 10:23:07,142: INFO/job] Job 36: StateChangeJob complete, setting state lnet_down on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 10:23:07,143: INFO/job] Job 36: not_deleted=True
[2017-05-10 10:23:07,144: INFO/job_scheduler] Job 36: complete_job: Updating cache
[2017-05-10 10:23:07,145: INFO/job_scheduler] Job 36: locked_item 140516499188048 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 10:23:07,157: INFO/job_scheduler] Job 36 completing (errored=False, cancelled=False)
[2017-05-10 10:23:07,185: INFO/job_scheduler] run_next: 1 runnable jobs of (17 pending, 6 tasked)
[2017-05-10 10:23:07,191: INFO/job_scheduler] Job 39: RunJobThread.run
[2017-05-10 10:23:07,192: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_lnet {}
[2017-05-10 10:23:07,214: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f1d0b1d3-6366-426f-8190-a413e4c80dff
[2017-05-10 10:23:07,214: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f1d0b1d3-6366-426f-8190-a413e4c80dff
[2017-05-10 10:23:08,418: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:08,418: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:08,419: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:08,419: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:08,419: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f1d0b1d3-6366-426f-8190-a413e4c80dff
[2017-05-10 10:23:08,419: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f1d0b1d3-6366-426f-8190-a413e4c80dff
[2017-05-10 10:23:08,428: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f1d0b1d3-6366-426f-8190-a413e4c80dff
[2017-05-10 10:23:08,428: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f1d0b1d3-6366-426f-8190-a413e4c80dff
[2017-05-10 10:23:08,430: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 10:23:08,454: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f0898864-65e2-40bd-932e-3fd538e99d3f
[2017-05-10 10:23:08,454: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f0898864-65e2-40bd-932e-3fd538e99d3f
[2017-05-10 10:23:08,657: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:08,657: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:08,657: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:08,657: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:08,658: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f0898864-65e2-40bd-932e-3fd538e99d3f
[2017-05-10 10:23:08,658: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f0898864-65e2-40bd-932e-3fd538e99d3f
[2017-05-10 10:23:08,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f0898864-65e2-40bd-932e-3fd538e99d3f
[2017-05-10 10:23:08,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f0898864-65e2-40bd-932e-3fd538e99d3f
[2017-05-10 10:23:08,852: INFO/job_scheduler] Job 39 finished 2 steps successfully
[2017-05-10 10:23:08,886: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 140516499188048
[2017-05-10 10:23:08,891: INFO/job] Job 39: StateChangeJob complete, setting state lnet_up on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 10:23:08,891: INFO/job] Job 39: not_deleted=True
[2017-05-10 10:23:08,891: INFO/job_scheduler] Job 39: complete_job: Updating cache
[2017-05-10 10:23:08,892: INFO/job_scheduler] Job 39: locked_item 140516499188048 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 10:23:08,894: INFO/job_scheduler] Job 39 completing (errored=False, cancelled=False)
[2017-05-10 10:23:08,918: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 6 tasked)
[2017-05-10 10:23:09,439: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:09,439: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:09,439: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:09,439: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:09,439: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 96815398-2183-43d2-842e-1637ad625979
[2017-05-10 10:23:09,439: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 96815398-2183-43d2-842e-1637ad625979
[2017-05-10 10:23:09,476: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 96815398-2183-43d2-842e-1637ad625979
[2017-05-10 10:23:09,476: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 96815398-2183-43d2-842e-1637ad625979
[2017-05-10 10:23:09,477: INFO/job_scheduler] Job 32 finished 1 steps successfully
[2017-05-10 10:23:09,561: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 140516499185744
[2017-05-10 10:23:09,566: INFO/job] Job 32: StateChangeJob complete, setting state configured on lotus-32vm8.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-10 10:23:09,566: INFO/job] Job 32: not_deleted=True
[2017-05-10 10:23:09,566: INFO/job_scheduler] Job 32: complete_job: Updating cache
[2017-05-10 10:23:09,566: INFO/job_scheduler] Job 32: locked_item 140516499185744 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-10 10:23:09,569: INFO/job_scheduler] Job 32 completing (errored=False, cancelled=False)
[2017-05-10 10:23:09,593: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 5 tasked)
[2017-05-10 10:23:10,169: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:10,169: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:10,169: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:10,169: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:10,169: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc da593318-c3f6-4197-b2bc-8693bacc6a36
[2017-05-10 10:23:10,169: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc da593318-c3f6-4197-b2bc-8693bacc6a36
[2017-05-10 10:23:10,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc da593318-c3f6-4197-b2bc-8693bacc6a36
[2017-05-10 10:23:10,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc da593318-c3f6-4197-b2bc-8693bacc6a36
[2017-05-10 10:23:10,190: INFO/job_scheduler] Job 33 finished 1 steps successfully
[2017-05-10 10:23:10,386: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 140516499188688
[2017-05-10 10:23:10,390: INFO/job] Job 33: StateChangeJob complete, setting state configured on lotus-32vm8.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-10 10:23:10,391: INFO/job] Job 33: not_deleted=True
[2017-05-10 10:23:10,391: INFO/job_scheduler] Job 33: complete_job: Updating cache
[2017-05-10 10:23:10,391: INFO/job_scheduler] Job 33: locked_item 140516499188688 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-10 10:23:10,394: INFO/job_scheduler] Job 33 completing (errored=False, cancelled=False)
[2017-05-10 10:23:10,418: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 4 tasked)
[2017-05-10 10:23:17,768: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:17,768: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:17,769: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:17,769: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:17,769: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8f29636b-bdff-4cce-b6ed-f7652b82d662
[2017-05-10 10:23:17,769: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8f29636b-bdff-4cce-b6ed-f7652b82d662
[2017-05-10 10:23:17,771: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8f29636b-bdff-4cce-b6ed-f7652b82d662
[2017-05-10 10:23:17,771: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8f29636b-bdff-4cce-b6ed-f7652b82d662
[2017-05-10 10:23:17,772: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.224', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-05-10 10:23:17,792: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cf19170b-f6a6-43df-80a1-cdbc56471541
[2017-05-10 10:23:17,792: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cf19170b-f6a6-43df-80a1-cdbc56471541
[2017-05-10 10:23:17,855: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:17,855: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:17,855: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:17,855: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:17,855: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cf19170b-f6a6-43df-80a1-cdbc56471541
[2017-05-10 10:23:17,855: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cf19170b-f6a6-43df-80a1-cdbc56471541
[2017-05-10 10:23:17,860: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cf19170b-f6a6-43df-80a1-cdbc56471541
[2017-05-10 10:23:17,860: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cf19170b-f6a6-43df-80a1-cdbc56471541
[2017-05-10 10:23:17,878: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 49545, 'pcs_password': 'UFyN9K9YIxj8RsuzAS1L'}
[2017-05-10 10:23:17,893: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bd29ea54-8d01-4dd2-8a0d-6ac9b8215b2c
[2017-05-10 10:23:17,893: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bd29ea54-8d01-4dd2-8a0d-6ac9b8215b2c
[2017-05-10 10:23:20,565: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:20,565: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:20,565: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:20,565: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:20,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d6104ae9-29f2-4123-9470-c1098d8818d2
[2017-05-10 10:23:20,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d6104ae9-29f2-4123-9470-c1098d8818d2
[2017-05-10 10:23:20,611: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d6104ae9-29f2-4123-9470-c1098d8818d2
[2017-05-10 10:23:20,611: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d6104ae9-29f2-4123-9470-c1098d8818d2
[2017-05-10 10:23:20,612: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.227', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-05-10 10:23:20,627: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f0fb7af2-0ba9-44dd-bb2f-40e44c9bc917
[2017-05-10 10:23:20,627: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f0fb7af2-0ba9-44dd-bb2f-40e44c9bc917
[2017-05-10 10:23:20,688: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:20,688: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:20,690: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:20,690: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:20,693: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f0fb7af2-0ba9-44dd-bb2f-40e44c9bc917
[2017-05-10 10:23:20,693: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f0fb7af2-0ba9-44dd-bb2f-40e44c9bc917
[2017-05-10 10:23:20,694: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f0fb7af2-0ba9-44dd-bb2f-40e44c9bc917
[2017-05-10 10:23:20,694: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f0fb7af2-0ba9-44dd-bb2f-40e44c9bc917
[2017-05-10 10:23:20,718: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 58263, 'pcs_password': 'nZfmQFzvPPnxJP7Pofg3'}
[2017-05-10 10:23:20,732: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4c5dbd76-e7e9-4168-b224-2e0544e25b8e
[2017-05-10 10:23:20,732: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4c5dbd76-e7e9-4168-b224-2e0544e25b8e
[2017-05-10 10:23:30,031: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:30,031: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:30,031: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:30,031: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:30,031: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bd29ea54-8d01-4dd2-8a0d-6ac9b8215b2c
[2017-05-10 10:23:30,031: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bd29ea54-8d01-4dd2-8a0d-6ac9b8215b2c
[2017-05-10 10:23:30,061: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bd29ea54-8d01-4dd2-8a0d-6ac9b8215b2c
[2017-05-10 10:23:30,061: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bd29ea54-8d01-4dd2-8a0d-6ac9b8215b2c
[2017-05-10 10:23:30,062: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': True, 'mcast_port': 49545, 'new_node_fqdn': u'lotus-32vm5.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'UFyN9K9YIxj8RsuzAS1L'}
[2017-05-10 10:23:30,079: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4f866877-1c7c-4978-9013-a2c8d2f78f88
[2017-05-10 10:23:30,079: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4f866877-1c7c-4978-9013-a2c8d2f78f88
[2017-05-10 10:23:32,876: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:32,876: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:32,876: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:32,876: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:32,876: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4c5dbd76-e7e9-4168-b224-2e0544e25b8e
[2017-05-10 10:23:32,876: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4c5dbd76-e7e9-4168-b224-2e0544e25b8e
[2017-05-10 10:23:32,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4c5dbd76-e7e9-4168-b224-2e0544e25b8e
[2017-05-10 10:23:32,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4c5dbd76-e7e9-4168-b224-2e0544e25b8e
[2017-05-10 10:23:32,898: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': True, 'mcast_port': 58263, 'new_node_fqdn': u'lotus-32vm8.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'nZfmQFzvPPnxJP7Pofg3'}
[2017-05-10 10:23:32,911: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc edd43805-5d0a-4ea2-b59f-3d72b52f0db5
[2017-05-10 10:23:32,911: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc edd43805-5d0a-4ea2-b59f-3d72b52f0db5
[2017-05-10 10:23:36,161: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:36,161: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:36,161: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:36,161: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:36,161: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4f866877-1c7c-4978-9013-a2c8d2f78f88
[2017-05-10 10:23:36,161: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4f866877-1c7c-4978-9013-a2c8d2f78f88
[2017-05-10 10:23:36,193: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4f866877-1c7c-4978-9013-a2c8d2f78f88
[2017-05-10 10:23:36,193: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4f866877-1c7c-4978-9013-a2c8d2f78f88
[2017-05-10 10:23:36,194: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 10:23:36.194318+00:00:
[2017-05-10 10:23:36,194: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 10:23:36.194318+00:00:
[2017-05-10 10:23:36,194: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->49545
[2017-05-10 10:23:36,194: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->49545
[2017-05-10 10:23:36,198: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 10:23:36,198: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 10:23:36,213: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:23:36,214: INFO/job_scheduler]   Job 10 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-05-10 10:23:36,214: INFO/job_scheduler]   Job 15 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-05-10 10:23:36,214: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:23:36,214: INFO/job_scheduler]   Job 18 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:23:36,215: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 4 tasked)
[2017-05-10 10:23:36,214: INFO/job_scheduler] Job 10 finished 1 steps successfully
[2017-05-10 10:23:36,249: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 140516499041424
[2017-05-10 10:23:36,257: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-10 10:23:36,265: INFO/job] Job 10: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:23:36,266: INFO/job] Job 10: not_deleted=True
[2017-05-10 10:23:36,267: INFO/job_scheduler] Job 10: complete_job: Updating cache
[2017-05-10 10:23:36,267: INFO/job_scheduler] Job 10: locked_item 140516499041424 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 10:23:36,281: INFO/job_scheduler] Job 10 completing (errored=False, cancelled=False)
[2017-05-10 10:23:36,432: INFO/job_scheduler] run_next: 1 runnable jobs of (16 pending, 3 tasked)
[2017-05-10 10:23:36,442: INFO/job_scheduler] Job 15: RunJobThread.run
[2017-05-10 10:23:36,442: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-05-10 10:23:36,455: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 16e55b72-6d29-4c0f-bbaf-320da57332a5
[2017-05-10 10:23:36,455: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 16e55b72-6d29-4c0f-bbaf-320da57332a5
[2017-05-10 10:23:37,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:23:37,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:23:37,554: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:23:37,554: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:23:37,554: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7b5b4e7c-43ba-49ae-b133-c448fd76f05f
[2017-05-10 10:23:37,554: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7b5b4e7c-43ba-49ae-b133-c448fd76f05f
[2017-05-10 10:23:37,581: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7b5b4e7c-43ba-49ae-b133-c448fd76f05f
[2017-05-10 10:23:37,581: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7b5b4e7c-43ba-49ae-b133-c448fd76f05f
[2017-05-10 10:23:37,582: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.225', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-05-10 10:23:37,602: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 76b0a24f-291b-46bd-adc9-ffde56732ebb
[2017-05-10 10:23:37,602: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 76b0a24f-291b-46bd-adc9-ffde56732ebb
[2017-05-10 10:23:37,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:23:37,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:23:37,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:23:37,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:23:37,673: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 76b0a24f-291b-46bd-adc9-ffde56732ebb
[2017-05-10 10:23:37,673: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 76b0a24f-291b-46bd-adc9-ffde56732ebb
[2017-05-10 10:23:37,718: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 76b0a24f-291b-46bd-adc9-ffde56732ebb
[2017-05-10 10:23:37,718: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 76b0a24f-291b-46bd-adc9-ffde56732ebb
[2017-05-10 10:23:37,734: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 49545, 'pcs_password': 'd8xrZIAdurI1oXZjX1x0'}
[2017-05-10 10:23:37,749: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0b8aab09-8996-4782-9ae2-bbb3e8c77379
[2017-05-10 10:23:37,749: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0b8aab09-8996-4782-9ae2-bbb3e8c77379
[2017-05-10 10:23:39,009: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:39,009: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:39,010: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:39,010: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:39,010: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc edd43805-5d0a-4ea2-b59f-3d72b52f0db5
[2017-05-10 10:23:39,010: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc edd43805-5d0a-4ea2-b59f-3d72b52f0db5
[2017-05-10 10:23:39,028: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc edd43805-5d0a-4ea2-b59f-3d72b52f0db5
[2017-05-10 10:23:39,028: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc edd43805-5d0a-4ea2-b59f-3d72b52f0db5
[2017-05-10 10:23:39,029: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 10:23:39.029369+00:00:
[2017-05-10 10:23:39,029: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 10:23:39.029369+00:00:
[2017-05-10 10:23:39,029: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->58263
[2017-05-10 10:23:39,029: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->58263
[2017-05-10 10:23:39,034: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 10:23:39,034: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 10:23:39,050: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:23:39,051: INFO/job_scheduler]   Job 34 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-05-10 10:23:39,051: INFO/job_scheduler]   Job 37 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-05-10 10:23:39,051: INFO/job_scheduler]   Job 38 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:23:39,052: INFO/job_scheduler]   Job 40 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:23:39,053: INFO/job_scheduler] run_next: 0 runnable jobs of (15 pending, 4 tasked)
[2017-05-10 10:23:39,051: INFO/job_scheduler] Job 34 finished 1 steps successfully
[2017-05-10 10:23:39,074: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 140516499068304
[2017-05-10 10:23:39,082: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-10 10:23:39,098: INFO/job] Job 34: StateChangeJob complete, setting state stopped on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:23:39,098: INFO/job] Job 34: not_deleted=True
[2017-05-10 10:23:39,099: INFO/job_scheduler] Job 34: complete_job: Updating cache
[2017-05-10 10:23:39,099: INFO/job_scheduler] Job 34: locked_item 140516499068304 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 10:23:39,110: INFO/job_scheduler] Job 34 completing (errored=False, cancelled=False)
[2017-05-10 10:23:39,149: INFO/job_scheduler] run_next: 1 runnable jobs of (15 pending, 3 tasked)
[2017-05-10 10:23:39,154: INFO/job_scheduler] Job 37: RunJobThread.run
[2017-05-10 10:23:39,154: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-05-10 10:23:39,169: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0b339272-78cc-4068-b46a-869757561af3
[2017-05-10 10:23:39,169: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0b339272-78cc-4068-b46a-869757561af3
[2017-05-10 10:23:39,330: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:23:39,330: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:23:39,331: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:23:39,331: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:23:39,331: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2253aacf-a2f9-47d9-ba59-951c21ce3fa2
[2017-05-10 10:23:39,331: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2253aacf-a2f9-47d9-ba59-951c21ce3fa2
[2017-05-10 10:23:39,344: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2253aacf-a2f9-47d9-ba59-951c21ce3fa2
[2017-05-10 10:23:39,344: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2253aacf-a2f9-47d9-ba59-951c21ce3fa2
[2017-05-10 10:23:39,345: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.226', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-05-10 10:23:39,363: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e75f7eaf-ac75-4eb9-938d-590cfc99894e
[2017-05-10 10:23:39,363: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e75f7eaf-ac75-4eb9-938d-590cfc99894e
[2017-05-10 10:23:39,456: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:23:39,456: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:23:39,457: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:23:39,457: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:23:39,457: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e75f7eaf-ac75-4eb9-938d-590cfc99894e
[2017-05-10 10:23:39,457: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e75f7eaf-ac75-4eb9-938d-590cfc99894e
[2017-05-10 10:23:39,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e75f7eaf-ac75-4eb9-938d-590cfc99894e
[2017-05-10 10:23:39,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e75f7eaf-ac75-4eb9-938d-590cfc99894e
[2017-05-10 10:23:39,502: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 58263, 'pcs_password': '7WGxEapABzlZfKQPvBIm'}
[2017-05-10 10:23:39,516: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e0c50633-2a67-4def-9b17-ae01a46fc9c9
[2017-05-10 10:23:39,516: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e0c50633-2a67-4def-9b17-ae01a46fc9c9
[2017-05-10 10:23:43,737: INFO/job_scheduler] run_next: 0 runnable jobs of (14 pending, 4 tasked)
[2017-05-10 10:23:43,754: INFO/job_scheduler] run_next: 0 runnable jobs of (14 pending, 4 tasked)
[2017-05-10 10:23:44,540: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:44,540: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:44,541: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:44,541: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:44,541: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 16e55b72-6d29-4c0f-bbaf-320da57332a5
[2017-05-10 10:23:44,541: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 16e55b72-6d29-4c0f-bbaf-320da57332a5
[2017-05-10 10:23:44,572: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 16e55b72-6d29-4c0f-bbaf-320da57332a5
[2017-05-10 10:23:44,572: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 16e55b72-6d29-4c0f-bbaf-320da57332a5
[2017-05-10 10:23:44,572: INFO/job_scheduler] Job 15 finished 1 steps successfully
[2017-05-10 10:23:44,689: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 140516499041424
[2017-05-10 10:23:44,720: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 10:23:44,728: INFO/job] Job 15: StateChangeJob complete, setting state started on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:23:44,728: INFO/job] Job 15: not_deleted=True
[2017-05-10 10:23:44,729: INFO/job_scheduler] Job 15: complete_job: Updating cache
[2017-05-10 10:23:44,729: INFO/job_scheduler] Job 15: locked_item 140516499041424 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 10:23:44,734: INFO/job_scheduler] Job 15 completing (errored=False, cancelled=False)
[2017-05-10 10:23:44,789: INFO/job_scheduler] run_next: 1 runnable jobs of (14 pending, 3 tasked)
[2017-05-10 10:23:44,828: INFO/job_scheduler] Job 16: RunJobThread.run
[2017-05-10 10:23:44,829: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 10:23:44,844: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bb1e0e17-e806-4792-8ad9-ee9c0ad53e82
[2017-05-10 10:23:44,844: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bb1e0e17-e806-4792-8ad9-ee9c0ad53e82
[2017-05-10 10:23:47,283: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:47,283: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:47,283: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:47,283: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:47,284: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0b339272-78cc-4068-b46a-869757561af3
[2017-05-10 10:23:47,284: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0b339272-78cc-4068-b46a-869757561af3
[2017-05-10 10:23:47,285: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0b339272-78cc-4068-b46a-869757561af3
[2017-05-10 10:23:47,285: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0b339272-78cc-4068-b46a-869757561af3
[2017-05-10 10:23:47,286: INFO/job_scheduler] Job 37 finished 1 steps successfully
[2017-05-10 10:23:47,489: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 140516499068304
[2017-05-10 10:23:47,522: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm8.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 10:23:47,542: INFO/job] Job 37: StateChangeJob complete, setting state started on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:23:47,544: INFO/job] Job 37: not_deleted=True
[2017-05-10 10:23:47,544: INFO/job_scheduler] Job 37: complete_job: Updating cache
[2017-05-10 10:23:47,545: INFO/job_scheduler] Job 37: locked_item 140516499068304 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 10:23:47,555: INFO/job_scheduler] Job 37 completing (errored=False, cancelled=False)
[2017-05-10 10:23:47,606: INFO/job_scheduler] run_next: 1 runnable jobs of (13 pending, 3 tasked)
[2017-05-10 10:23:47,619: INFO/job_scheduler] Job 38: RunJobThread.run
[2017-05-10 10:23:47,621: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 10:23:47,637: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 53760013-19e1-41f5-88f3-d8713ee9015d
[2017-05-10 10:23:47,637: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 53760013-19e1-41f5-88f3-d8713ee9015d
[2017-05-10 10:23:48,695: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-10 10:23:48,710: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-10 10:23:49,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:23:49,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:23:49,899: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:23:49,899: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:23:49,899: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0b8aab09-8996-4782-9ae2-bbb3e8c77379
[2017-05-10 10:23:49,899: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0b8aab09-8996-4782-9ae2-bbb3e8c77379
[2017-05-10 10:23:49,915: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0b8aab09-8996-4782-9ae2-bbb3e8c77379
[2017-05-10 10:23:49,915: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0b8aab09-8996-4782-9ae2-bbb3e8c77379
[2017-05-10 10:23:49,916: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': False, 'mcast_port': 49545, 'new_node_fqdn': u'lotus-32vm6.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'd8xrZIAdurI1oXZjX1x0'}
[2017-05-10 10:23:49,930: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 593aff6e-6036-4f3d-9e04-006261a13bbc
[2017-05-10 10:23:49,930: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 593aff6e-6036-4f3d-9e04-006261a13bbc
[2017-05-10 10:23:51,705: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:23:51,705: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:23:51,705: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:23:51,705: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:23:51,706: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e0c50633-2a67-4def-9b17-ae01a46fc9c9
[2017-05-10 10:23:51,706: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e0c50633-2a67-4def-9b17-ae01a46fc9c9
[2017-05-10 10:23:51,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e0c50633-2a67-4def-9b17-ae01a46fc9c9
[2017-05-10 10:23:51,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e0c50633-2a67-4def-9b17-ae01a46fc9c9
[2017-05-10 10:23:51,733: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': False, 'mcast_port': 58263, 'new_node_fqdn': u'lotus-32vm7.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': '7WGxEapABzlZfKQPvBIm'}
[2017-05-10 10:23:51,748: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dae39698-ffa5-41ff-ac78-98dc5f3de212
[2017-05-10 10:23:51,748: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dae39698-ffa5-41ff-ac78-98dc5f3de212
[2017-05-10 10:23:51,925: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:51,925: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:51,925: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:51,925: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:51,926: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bb1e0e17-e806-4792-8ad9-ee9c0ad53e82
[2017-05-10 10:23:51,926: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bb1e0e17-e806-4792-8ad9-ee9c0ad53e82
[2017-05-10 10:23:51,962: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bb1e0e17-e806-4792-8ad9-ee9c0ad53e82
[2017-05-10 10:23:51,962: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bb1e0e17-e806-4792-8ad9-ee9c0ad53e82
[2017-05-10 10:23:51,963: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-05-10 10:23:51,980: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a5e72d75-1778-4560-8e1a-52427df36d95
[2017-05-10 10:23:51,980: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a5e72d75-1778-4560-8e1a-52427df36d95
[2017-05-10 10:23:53,793: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-10 10:23:53,812: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-10 10:23:54,755: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:54,755: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:54,756: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:54,756: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:54,756: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 53760013-19e1-41f5-88f3-d8713ee9015d
[2017-05-10 10:23:54,756: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 53760013-19e1-41f5-88f3-d8713ee9015d
[2017-05-10 10:23:54,803: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 53760013-19e1-41f5-88f3-d8713ee9015d
[2017-05-10 10:23:54,803: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 53760013-19e1-41f5-88f3-d8713ee9015d
[2017-05-10 10:23:54,804: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-05-10 10:23:54,820: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 04ab4a44-fbfc-48ff-83d5-8dccfa867583
[2017-05-10 10:23:54,820: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 04ab4a44-fbfc-48ff-83d5-8dccfa867583
[2017-05-10 10:23:56,081: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:56,081: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:56,082: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:56,082: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:23:56,082: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 593aff6e-6036-4f3d-9e04-006261a13bbc
[2017-05-10 10:23:56,082: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 593aff6e-6036-4f3d-9e04-006261a13bbc
[2017-05-10 10:23:56,096: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 593aff6e-6036-4f3d-9e04-006261a13bbc
[2017-05-10 10:23:56,096: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 593aff6e-6036-4f3d-9e04-006261a13bbc
[2017-05-10 10:23:56,096: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 10:23:56.096779+00:00:
[2017-05-10 10:23:56,096: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 10:23:56.096779+00:00:
[2017-05-10 10:23:56,097: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->49545
[2017-05-10 10:23:56,097: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->49545
[2017-05-10 10:23:56,099: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 10:23:56,099: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 10:23:56,117: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:23:56,117: INFO/job_scheduler]   Job 45 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-05-10 10:23:56,117: INFO/job_scheduler]   Job 47 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-05-10 10:23:56,117: INFO/job_scheduler]   Job 50 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:23:56,118: INFO/job_scheduler]   Job 51 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:23:56,119: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-10 10:23:56,118: INFO/job_scheduler] Job 45 finished 1 steps successfully
[2017-05-10 10:23:56,135: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 140516499253136
[2017-05-10 10:23:56,142: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-10 10:23:56,150: INFO/job] Job 45: StateChangeJob complete, setting state stopped on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:23:56,151: INFO/job] Job 45: not_deleted=True
[2017-05-10 10:23:56,152: INFO/job_scheduler] Job 45: complete_job: Updating cache
[2017-05-10 10:23:56,152: INFO/job_scheduler] Job 45: locked_item 140516499253136 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 10:23:56,161: INFO/job_scheduler] Job 45 completing (errored=False, cancelled=False)
[2017-05-10 10:23:56,210: INFO/job_scheduler] run_next: 1 runnable jobs of (12 pending, 3 tasked)
[2017-05-10 10:23:56,216: INFO/job_scheduler] Job 47: RunJobThread.run
[2017-05-10 10:23:56,216: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-05-10 10:23:56,230: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8a449b6c-d4e3-4723-a526-52fc23578bce
[2017-05-10 10:23:56,230: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8a449b6c-d4e3-4723-a526-52fc23578bce
[2017-05-10 10:23:58,743: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 4 tasked)
[2017-05-10 10:23:58,761: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 4 tasked)
[2017-05-10 10:23:58,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:58,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:58,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:58,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:23:58,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dae39698-ffa5-41ff-ac78-98dc5f3de212
[2017-05-10 10:23:58,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dae39698-ffa5-41ff-ac78-98dc5f3de212
[2017-05-10 10:23:58,915: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dae39698-ffa5-41ff-ac78-98dc5f3de212
[2017-05-10 10:23:58,915: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dae39698-ffa5-41ff-ac78-98dc5f3de212
[2017-05-10 10:23:58,916: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 10:23:58.916329+00:00:
[2017-05-10 10:23:58,916: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 10:23:58.916329+00:00:
[2017-05-10 10:23:58,916: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->58263
[2017-05-10 10:23:58,916: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->58263
[2017-05-10 10:23:58,918: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 10:23:58,918: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 10:23:58,932: INFO/job_scheduler] Job 23 finished 1 steps successfully
[2017-05-10 10:23:58,933: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:23:58,933: INFO/job_scheduler]   Job 23 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-05-10 10:23:58,933: INFO/job_scheduler]   Job 26 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-05-10 10:23:58,933: INFO/job_scheduler]   Job 27 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:23:58,934: INFO/job_scheduler]   Job 29 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:23:58,934: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 4 tasked)
[2017-05-10 10:23:58,985: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 140516297790288
[2017-05-10 10:23:58,994: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-10 10:23:59,013: INFO/job] Job 23: StateChangeJob complete, setting state stopped on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:23:59,014: INFO/job] Job 23: not_deleted=True
[2017-05-10 10:23:59,014: INFO/job_scheduler] Job 23: complete_job: Updating cache
[2017-05-10 10:23:59,014: INFO/job_scheduler] Job 23: locked_item 140516297790288 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 10:23:59,019: INFO/job_scheduler] Job 23 completing (errored=False, cancelled=False)
[2017-05-10 10:23:59,057: INFO/job_scheduler] run_next: 1 runnable jobs of (11 pending, 3 tasked)
[2017-05-10 10:23:59,061: INFO/job_scheduler] Job 26: RunJobThread.run
[2017-05-10 10:23:59,062: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-05-10 10:23:59,075: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f5521e2d-239c-4d72-a2dc-ef1e839363e3
[2017-05-10 10:23:59,075: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f5521e2d-239c-4d72-a2dc-ef1e839363e3
[2017-05-10 10:24:00,503: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 4 tasked)
[2017-05-10 10:24:00,527: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 4 tasked)
[2017-05-10 10:24:03,874: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 4 tasked)
[2017-05-10 10:24:03,893: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 4 tasked)
[2017-05-10 10:24:04,428: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:24:04,428: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:24:04,429: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:24:04,429: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:24:04,429: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8a449b6c-d4e3-4723-a526-52fc23578bce
[2017-05-10 10:24:04,429: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8a449b6c-d4e3-4723-a526-52fc23578bce
[2017-05-10 10:24:04,446: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8a449b6c-d4e3-4723-a526-52fc23578bce
[2017-05-10 10:24:04,446: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8a449b6c-d4e3-4723-a526-52fc23578bce
[2017-05-10 10:24:04,447: INFO/job_scheduler] Job 47 finished 1 steps successfully
[2017-05-10 10:24:04,567: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 140516499253136
[2017-05-10 10:24:04,596: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm6.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 10:24:04,605: INFO/job] Job 47: StateChangeJob complete, setting state started on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:24:04,605: INFO/job] Job 47: not_deleted=True
[2017-05-10 10:24:04,605: INFO/job_scheduler] Job 47: complete_job: Updating cache
[2017-05-10 10:24:04,606: INFO/job_scheduler] Job 47: locked_item 140516499253136 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 10:24:04,612: INFO/job_scheduler] Job 47 completing (errored=False, cancelled=False)
[2017-05-10 10:24:04,664: INFO/job_scheduler] run_next: 1 runnable jobs of (10 pending, 3 tasked)
[2017-05-10 10:24:04,678: INFO/job_scheduler] Job 50: RunJobThread.run
[2017-05-10 10:24:04,679: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 10:24:04,693: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1a895d4a-49fc-406b-82ef-69ede452cee6
[2017-05-10 10:24:04,693: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1a895d4a-49fc-406b-82ef-69ede452cee6
[2017-05-10 10:24:07,266: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:24:07,266: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:24:07,266: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:24:07,266: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:24:07,266: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f5521e2d-239c-4d72-a2dc-ef1e839363e3
[2017-05-10 10:24:07,266: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f5521e2d-239c-4d72-a2dc-ef1e839363e3
[2017-05-10 10:24:07,291: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f5521e2d-239c-4d72-a2dc-ef1e839363e3
[2017-05-10 10:24:07,291: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f5521e2d-239c-4d72-a2dc-ef1e839363e3
[2017-05-10 10:24:07,302: INFO/job_scheduler] Job 26 finished 1 steps successfully
[2017-05-10 10:24:07,500: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 140516297790288
[2017-05-10 10:24:07,528: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 10:24:07,539: INFO/job] Job 26: StateChangeJob complete, setting state started on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:24:07,540: INFO/job] Job 26: not_deleted=True
[2017-05-10 10:24:07,540: INFO/job_scheduler] Job 26: complete_job: Updating cache
[2017-05-10 10:24:07,540: INFO/job_scheduler] Job 26: locked_item 140516297790288 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 10:24:07,550: INFO/job_scheduler] Job 26 completing (errored=False, cancelled=False)
[2017-05-10 10:24:07,585: INFO/job_scheduler] run_next: 1 runnable jobs of (9 pending, 3 tasked)
[2017-05-10 10:24:07,600: INFO/job_scheduler] Job 27: RunJobThread.run
[2017-05-10 10:24:07,601: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 10:24:07,617: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f4901fdf-d5a1-4eb4-a6b3-f9372c4ff894
[2017-05-10 10:24:07,617: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f4901fdf-d5a1-4eb4-a6b3-f9372c4ff894
[2017-05-10 10:24:08,121: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:08,137: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:10,546: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:10,565: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:10,604: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:10,605: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:10,605: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:10,605: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:10,627: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:10,627: INFO/job_scheduler]   Job 50 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:10,627: INFO/job_scheduler]   Job 52 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:10,628: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:11,797: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:24:11,797: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:24:11,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:24:11,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:24:11,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1a895d4a-49fc-406b-82ef-69ede452cee6
[2017-05-10 10:24:11,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1a895d4a-49fc-406b-82ef-69ede452cee6
[2017-05-10 10:24:11,808: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1a895d4a-49fc-406b-82ef-69ede452cee6
[2017-05-10 10:24:11,808: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1a895d4a-49fc-406b-82ef-69ede452cee6
[2017-05-10 10:24:11,810: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-05-10 10:24:11,825: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 70af60ec-1524-40bc-b6e6-8aaa683e91ec
[2017-05-10 10:24:11,825: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 70af60ec-1524-40bc-b6e6-8aaa683e91ec
[2017-05-10 10:24:13,884: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:13,884: INFO/job_scheduler]   Job 27 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:13,884: INFO/job_scheduler]   Job 30 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:13,885: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:13,906: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:13,906: INFO/job_scheduler]   Job 38 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:13,906: INFO/job_scheduler]   Job 41 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:13,906: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:14,713: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:24:14,713: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:24:14,714: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:24:14,714: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:24:14,714: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f4901fdf-d5a1-4eb4-a6b3-f9372c4ff894
[2017-05-10 10:24:14,714: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f4901fdf-d5a1-4eb4-a6b3-f9372c4ff894
[2017-05-10 10:24:14,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f4901fdf-d5a1-4eb4-a6b3-f9372c4ff894
[2017-05-10 10:24:14,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f4901fdf-d5a1-4eb4-a6b3-f9372c4ff894
[2017-05-10 10:24:14,734: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-05-10 10:24:14,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ead67df8-be5e-49d7-b306-8a97ceeed3c6
[2017-05-10 10:24:14,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ead67df8-be5e-49d7-b306-8a97ceeed3c6
[2017-05-10 10:24:18,240: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:24:18,240: INFO/job_scheduler]   Job 27 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:24:18,240: INFO/job_scheduler]   Job 29 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:24:18,241: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:18,250: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:18,250: INFO/job_scheduler]   Job 27 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:18,250: INFO/job_scheduler]   Job 30 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:18,250: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:18,273: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:18,273: INFO/job_scheduler]   Job 38 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:18,273: INFO/job_scheduler]   Job 41 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:18,274: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:18,792: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:18,808: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:20,590: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:20,607: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:23,909: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:23,928: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:23,963: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:23,963: INFO/job_scheduler]   Job 27 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:23,963: INFO/job_scheduler]   Job 30 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:23,964: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:23,989: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:23,989: INFO/job_scheduler]   Job 38 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:23,989: INFO/job_scheduler]   Job 41 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:23,990: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:28,224: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:28,242: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:28,273: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:28,274: INFO/job_scheduler]   Job 27 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:28,274: INFO/job_scheduler]   Job 30 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:28,274: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:28,296: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:28,296: INFO/job_scheduler]   Job 38 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:28,297: INFO/job_scheduler]   Job 41 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:28,297: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:28,820: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:28,820: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:28,820: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:28,821: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:28,841: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:28,841: INFO/job_scheduler]   Job 50 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:28,842: INFO/job_scheduler]   Job 52 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:28,842: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:30,587: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:30,602: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:30,628: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:30,628: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:30,629: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:30,629: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:30,647: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:30,647: INFO/job_scheduler]   Job 50 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:30,647: INFO/job_scheduler]   Job 52 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:30,647: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:33,996: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:33,996: INFO/job_scheduler]   Job 27 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:33,996: INFO/job_scheduler]   Job 30 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:33,996: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:34,018: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:34,018: INFO/job_scheduler]   Job 38 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:34,018: INFO/job_scheduler]   Job 41 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:34,018: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:38,283: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:38,284: INFO/job_scheduler]   Job 27 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:38,284: INFO/job_scheduler]   Job 30 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:38,284: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:38,305: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:38,305: INFO/job_scheduler]   Job 38 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:38,305: INFO/job_scheduler]   Job 41 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:38,305: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:38,929: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:38,929: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:38,929: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:38,930: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:38,949: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:38,949: INFO/job_scheduler]   Job 50 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:38,949: INFO/job_scheduler]   Job 52 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:38,950: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:40,615: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:40,616: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:40,616: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:40,616: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:40,637: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:40,637: INFO/job_scheduler]   Job 50 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:40,637: INFO/job_scheduler]   Job 52 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:40,637: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:44,019: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:44,019: INFO/job_scheduler]   Job 27 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:44,019: INFO/job_scheduler]   Job 30 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:44,019: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:44,039: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:44,039: INFO/job_scheduler]   Job 38 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:44,039: INFO/job_scheduler]   Job 41 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:44,040: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:48,297: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:24:48,297: INFO/job_scheduler]   Job 27 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:24:48,297: INFO/job_scheduler]   Job 29 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:24:48,298: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:48,310: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:48,310: INFO/job_scheduler]   Job 27 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:48,311: INFO/job_scheduler]   Job 30 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:48,311: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:48,346: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:24:48,346: INFO/job_scheduler]   Job 38 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:24:48,346: INFO/job_scheduler]   Job 40 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:24:48,346: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:48,358: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:48,358: INFO/job_scheduler]   Job 38 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:48,358: INFO/job_scheduler]   Job 41 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:48,359: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:48,977: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:24:48,977: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:24:48,978: INFO/job_scheduler]   Job 18 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:24:48,978: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:48,992: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:48,993: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:48,993: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:48,993: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:49,031: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:24:49,031: INFO/job_scheduler]   Job 50 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:24:49,031: INFO/job_scheduler]   Job 51 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:24:49,032: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:49,047: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:49,047: INFO/job_scheduler]   Job 50 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:49,047: INFO/job_scheduler]   Job 52 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:49,047: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:50,669: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:24:50,670: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:24:50,670: INFO/job_scheduler]   Job 18 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:24:50,670: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:50,684: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:50,684: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:50,684: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:50,684: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:50,705: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:24:50,705: INFO/job_scheduler]   Job 50 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:24:50,705: INFO/job_scheduler]   Job 51 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:24:50,706: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:50,723: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:50,723: INFO/job_scheduler]   Job 50 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:50,723: INFO/job_scheduler]   Job 52 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:50,724: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:54,024: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:24:54,025: INFO/job_scheduler]   Job 27 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:24:54,025: INFO/job_scheduler]   Job 29 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:24:54,025: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:54,042: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:54,042: INFO/job_scheduler]   Job 27 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:54,043: INFO/job_scheduler]   Job 30 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:54,043: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:54,066: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:24:54,066: INFO/job_scheduler]   Job 38 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:24:54,066: INFO/job_scheduler]   Job 40 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:24:54,067: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:54,084: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:54,084: INFO/job_scheduler]   Job 38 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:54,084: INFO/job_scheduler]   Job 41 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:54,085: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:56,165: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:24:56,165: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:24:56,165: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:24:56,165: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:24:56,165: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 04ab4a44-fbfc-48ff-83d5-8dccfa867583
[2017-05-10 10:24:56,165: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 04ab4a44-fbfc-48ff-83d5-8dccfa867583
[2017-05-10 10:24:56,195: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 04ab4a44-fbfc-48ff-83d5-8dccfa867583
[2017-05-10 10:24:56,195: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 04ab4a44-fbfc-48ff-83d5-8dccfa867583
[2017-05-10 10:24:56,196: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-10 10:24:56,217: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 044db9d6-0969-4e9d-9dc9-fec888a316c1
[2017-05-10 10:24:56,217: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 044db9d6-0969-4e9d-9dc9-fec888a316c1
[2017-05-10 10:24:58,310: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:24:58,311: INFO/job_scheduler]   Job 27 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:24:58,311: INFO/job_scheduler]   Job 29 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:24:58,311: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:58,330: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:58,330: INFO/job_scheduler]   Job 27 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:58,330: INFO/job_scheduler]   Job 30 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:58,330: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:58,356: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:24:58,356: INFO/job_scheduler]   Job 38 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:24:58,357: INFO/job_scheduler]   Job 40 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:24:58,357: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:58,376: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:58,376: INFO/job_scheduler]   Job 38 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:58,376: INFO/job_scheduler]   Job 41 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:58,377: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:58,960: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:24:58,960: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:24:58,961: INFO/job_scheduler]   Job 18 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:24:58,961: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:58,979: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:58,979: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:58,980: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:58,980: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:59,002: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:24:59,002: INFO/job_scheduler]   Job 50 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:24:59,002: INFO/job_scheduler]   Job 51 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:24:59,003: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:24:59,023: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:24:59,023: INFO/job_scheduler]   Job 50 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:24:59,023: INFO/job_scheduler]   Job 52 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:24:59,023: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:25:00,678: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:25:00,678: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:25:00,678: INFO/job_scheduler]   Job 18 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:25:00,679: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:25:00,696: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:25:00,697: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:25:00,697: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:25:00,697: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:25:00,720: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:25:00,720: INFO/job_scheduler]   Job 50 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:25:00,720: INFO/job_scheduler]   Job 51 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:25:00,720: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:25:00,738: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:25:00,738: INFO/job_scheduler]   Job 50 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:25:00,738: INFO/job_scheduler]   Job 52 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:25:00,738: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:25:03,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:25:03,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:25:03,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:25:03,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:25:03,351: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 044db9d6-0969-4e9d-9dc9-fec888a316c1
[2017-05-10 10:25:03,351: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 044db9d6-0969-4e9d-9dc9-fec888a316c1
[2017-05-10 10:25:03,382: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 044db9d6-0969-4e9d-9dc9-fec888a316c1
[2017-05-10 10:25:03,382: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 044db9d6-0969-4e9d-9dc9-fec888a316c1
[2017-05-10 10:25:03,382: INFO/job_scheduler] Job 38 finished 3 steps successfully
[2017-05-10 10:25:03,450: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 140516499186256
[2017-05-10 10:25:03,461: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-10 10:25:03,464: INFO/job] Job 38: StateChangeJob complete, setting state stopped on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 10:25:03,464: INFO/job] Job 38: not_deleted=True
[2017-05-10 10:25:03,464: INFO/job_scheduler] Job 38: complete_job: Updating cache
[2017-05-10 10:25:03,464: INFO/job_scheduler] Job 38: locked_item 140516499186256 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 10:25:03,473: INFO/job_scheduler] Job 38 completing (errored=False, cancelled=False)
[2017-05-10 10:25:03,529: INFO/job_scheduler] run_next: 1 runnable jobs of (8 pending, 3 tasked)
[2017-05-10 10:25:03,541: INFO/job_scheduler] Job 40: RunJobThread.run
[2017-05-10 10:25:03,542: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 10:25:03,558: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 30b8ef1d-95bd-4b4d-ba0e-6e972a326e6a
[2017-05-10 10:25:03,558: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 30b8ef1d-95bd-4b4d-ba0e-6e972a326e6a
[2017-05-10 10:25:04,052: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-05-10 10:25:04,067: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-05-10 10:25:08,351: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:25:08,352: INFO/job_scheduler]   Job 27 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:25:08,352: INFO/job_scheduler]   Job 29 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:25:08,352: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-05-10 10:25:08,370: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:25:08,370: INFO/job_scheduler]   Job 27 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:25:08,370: INFO/job_scheduler]   Job 30 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:25:08,370: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-05-10 10:25:08,394: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:25:08,394: INFO/job_scheduler]   Job 40 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:25:08,395: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-05-10 10:25:08,412: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:25:08,412: INFO/job_scheduler]   Job 41 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:25:08,413: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-05-10 10:25:09,099: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:25:09,099: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:25:09,099: INFO/job_scheduler]   Job 18 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:25:09,100: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-05-10 10:25:09,117: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:25:09,117: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:25:09,117: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:25:09,117: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-05-10 10:25:09,143: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:25:09,143: INFO/job_scheduler]   Job 50 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:25:09,143: INFO/job_scheduler]   Job 51 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:25:09,144: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-05-10 10:25:09,159: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:25:09,159: INFO/job_scheduler]   Job 50 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:25:09,159: INFO/job_scheduler]   Job 52 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:25:09,159: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-05-10 10:25:09,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:25:09,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:25:09,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:25:09,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:25:09,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ead67df8-be5e-49d7-b306-8a97ceeed3c6
[2017-05-10 10:25:09,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ead67df8-be5e-49d7-b306-8a97ceeed3c6
[2017-05-10 10:25:09,221: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ead67df8-be5e-49d7-b306-8a97ceeed3c6
[2017-05-10 10:25:09,221: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ead67df8-be5e-49d7-b306-8a97ceeed3c6
[2017-05-10 10:25:09,224: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-10 10:25:09,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5cb0fb30-9e79-43ba-a614-0a64131fb693
[2017-05-10 10:25:09,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5cb0fb30-9e79-43ba-a614-0a64131fb693
[2017-05-10 10:25:10,716: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:25:10,716: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:25:10,716: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:25:10,716: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:25:10,716: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 30b8ef1d-95bd-4b4d-ba0e-6e972a326e6a
[2017-05-10 10:25:10,716: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 30b8ef1d-95bd-4b4d-ba0e-6e972a326e6a
[2017-05-10 10:25:10,723: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 30b8ef1d-95bd-4b4d-ba0e-6e972a326e6a
[2017-05-10 10:25:10,723: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 30b8ef1d-95bd-4b4d-ba0e-6e972a326e6a
[2017-05-10 10:25:10,724: INFO/job_scheduler] Job 40 finished 1 steps successfully
[2017-05-10 10:25:10,744: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:25:10,744: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:25:10,744: INFO/job_scheduler]   Job 18 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:25:10,745: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-05-10 10:25:10,762: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:25:10,762: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:25:10,762: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:25:10,763: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-05-10 10:25:10,784: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:25:10,784: INFO/job_scheduler]   Job 50 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:25:10,784: INFO/job_scheduler]   Job 51 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:25:10,784: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-05-10 10:25:10,801: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:25:10,801: INFO/job_scheduler]   Job 50 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:25:10,801: INFO/job_scheduler]   Job 52 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:25:10,802: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-05-10 10:25:10,817: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 140516499186256
[2017-05-10 10:25:10,845: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm8.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 10:25:10,870: INFO/job] Job 40: StateChangeJob complete, setting state started on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 10:25:10,870: INFO/job] Job 40: not_deleted=True
[2017-05-10 10:25:10,870: INFO/job_scheduler] Job 40: complete_job: Updating cache
[2017-05-10 10:25:10,870: INFO/job_scheduler] Job 40: locked_item 140516499186256 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 10:25:10,873: INFO/job_scheduler] Job 40 completing (errored=False, cancelled=False)
[2017-05-10 10:25:10,894: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-3
[2017-05-10 10:25:10,894: INFO/job_scheduler] _notify: Updating .mcast_port of item 3 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 58263
[2017-05-10 10:25:10,898: INFO/job_scheduler] _notify: Set mcast_port=58263 on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-3) and saved
[2017-05-10 10:25:10,901: INFO/job_scheduler] _notify: Updating .network_interfaces of item 3 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-05-10 10:25:10,930: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-3) and saved
[2017-05-10 10:25:10,950: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 3 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-10 10:25:10,955: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-3) and saved
[2017-05-10 10:25:10,976: INFO/job_scheduler] run_next: 1 runnable jobs of (7 pending, 3 tasked)
[2017-05-10 10:25:11,000: INFO/job_scheduler] Job 41: RunJobThread.run
[2017-05-10 10:25:11,002: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-05-10 10:25:11,025: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8c7f829a-1318-4f40-9bb3-c715d480c06f
[2017-05-10 10:25:11,025: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8c7f829a-1318-4f40-9bb3-c715d480c06f
[2017-05-10 10:25:13,223: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:25:13,223: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:25:13,223: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:25:13,223: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:25:13,223: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 70af60ec-1524-40bc-b6e6-8aaa683e91ec
[2017-05-10 10:25:13,223: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 70af60ec-1524-40bc-b6e6-8aaa683e91ec
[2017-05-10 10:25:13,256: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 70af60ec-1524-40bc-b6e6-8aaa683e91ec
[2017-05-10 10:25:13,256: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 70af60ec-1524-40bc-b6e6-8aaa683e91ec
[2017-05-10 10:25:13,259: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-10 10:25:13,274: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b29f221c-5ba9-49c0-acb5-5a04e5835516
[2017-05-10 10:25:13,274: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b29f221c-5ba9-49c0-acb5-5a04e5835516
[2017-05-10 10:25:14,081: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-05-10 10:25:14,099: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-05-10 10:25:14,124: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:25:14,124: INFO/job_scheduler]   Job 27 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:25:14,125: INFO/job_scheduler]   Job 29 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:25:14,125: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-05-10 10:25:14,146: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:25:14,146: INFO/job_scheduler]   Job 27 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:25:14,146: INFO/job_scheduler]   Job 30 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:25:14,146: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-05-10 10:25:14,170: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:25:14,170: INFO/job_scheduler]   Job 41 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:25:14,171: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-05-10 10:25:14,477: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:25:14,477: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:25:14,477: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:25:14,477: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:25:14,477: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a5e72d75-1778-4560-8e1a-52427df36d95
[2017-05-10 10:25:14,477: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a5e72d75-1778-4560-8e1a-52427df36d95
[2017-05-10 10:25:14,481: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a5e72d75-1778-4560-8e1a-52427df36d95
[2017-05-10 10:25:14,481: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a5e72d75-1778-4560-8e1a-52427df36d95
[2017-05-10 10:25:14,484: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-10 10:25:14,499: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 045aba3b-8f51-4e0d-bb9b-1cde8011b478
[2017-05-10 10:25:14,499: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 045aba3b-8f51-4e0d-bb9b-1cde8011b478
[2017-05-10 10:25:17,326: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:25:17,326: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:25:17,326: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:25:17,326: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:25:17,326: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5cb0fb30-9e79-43ba-a614-0a64131fb693
[2017-05-10 10:25:17,326: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5cb0fb30-9e79-43ba-a614-0a64131fb693
[2017-05-10 10:25:17,358: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5cb0fb30-9e79-43ba-a614-0a64131fb693
[2017-05-10 10:25:17,358: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5cb0fb30-9e79-43ba-a614-0a64131fb693
[2017-05-10 10:25:17,359: INFO/job_scheduler] Job 27 finished 3 steps successfully
[2017-05-10 10:25:17,404: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 140516499186320
[2017-05-10 10:25:17,411: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-10 10:25:17,418: INFO/job] Job 27: StateChangeJob complete, setting state stopped on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 10:25:17,419: INFO/job] Job 27: not_deleted=True
[2017-05-10 10:25:17,420: INFO/job_scheduler] Job 27: complete_job: Updating cache
[2017-05-10 10:25:17,421: INFO/job_scheduler] Job 27: locked_item 140516499186320 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 10:25:17,439: INFO/job_scheduler] Job 27 completing (errored=False, cancelled=False)
[2017-05-10 10:25:17,466: INFO/job_scheduler] run_next: 1 runnable jobs of (6 pending, 3 tasked)
[2017-05-10 10:25:17,480: INFO/job_scheduler] Job 29: RunJobThread.run
[2017-05-10 10:25:17,481: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 10:25:17,497: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1718c32f-fd76-4bb4-b845-fcd8e7c3b753
[2017-05-10 10:25:17,497: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1718c32f-fd76-4bb4-b845-fcd8e7c3b753
[2017-05-10 10:25:18,359: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-10 10:25:18,376: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-10 10:25:19,152: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-10 10:25:19,171: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-10 10:25:20,365: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:25:20,365: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:25:20,366: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:25:20,366: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:25:20,366: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b29f221c-5ba9-49c0-acb5-5a04e5835516
[2017-05-10 10:25:20,366: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b29f221c-5ba9-49c0-acb5-5a04e5835516
[2017-05-10 10:25:20,389: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b29f221c-5ba9-49c0-acb5-5a04e5835516
[2017-05-10 10:25:20,389: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b29f221c-5ba9-49c0-acb5-5a04e5835516
[2017-05-10 10:25:20,389: INFO/job_scheduler] Job 50 finished 3 steps successfully
[2017-05-10 10:25:20,453: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 140516499068816
[2017-05-10 10:25:20,462: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-10 10:25:20,481: INFO/job] Job 50: StateChangeJob complete, setting state stopped on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 10:25:20,483: INFO/job] Job 50: not_deleted=True
[2017-05-10 10:25:20,483: INFO/job_scheduler] Job 50: complete_job: Updating cache
[2017-05-10 10:25:20,491: INFO/job_scheduler] Job 50: locked_item 140516499068816 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 10:25:20,495: INFO/job_scheduler] Job 50 completing (errored=False, cancelled=False)
[2017-05-10 10:25:20,541: INFO/job_scheduler] run_next: 1 runnable jobs of (5 pending, 3 tasked)
[2017-05-10 10:25:20,555: INFO/job_scheduler] Job 51: RunJobThread.run
[2017-05-10 10:25:20,556: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 10:25:20,577: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dcd657f5-3272-46bd-9821-7917001e20de
[2017-05-10 10:25:20,577: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dcd657f5-3272-46bd-9821-7917001e20de
[2017-05-10 10:25:20,779: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 4 tasked)
[2017-05-10 10:25:20,794: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 4 tasked)
[2017-05-10 10:25:21,636: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:25:21,636: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:25:21,636: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:25:21,636: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:25:21,637: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 045aba3b-8f51-4e0d-bb9b-1cde8011b478
[2017-05-10 10:25:21,637: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 045aba3b-8f51-4e0d-bb9b-1cde8011b478
[2017-05-10 10:25:21,669: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 045aba3b-8f51-4e0d-bb9b-1cde8011b478
[2017-05-10 10:25:21,669: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 045aba3b-8f51-4e0d-bb9b-1cde8011b478
[2017-05-10 10:25:21,669: INFO/job_scheduler] Job 16 finished 3 steps successfully
[2017-05-10 10:25:21,726: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 140516633537808
[2017-05-10 10:25:21,735: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-10 10:25:21,739: INFO/job] Job 16: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 10:25:21,740: INFO/job] Job 16: not_deleted=True
[2017-05-10 10:25:21,742: INFO/job_scheduler] Job 16: complete_job: Updating cache
[2017-05-10 10:25:21,743: INFO/job_scheduler] Job 16: locked_item 140516633537808 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 10:25:21,755: INFO/job_scheduler] Job 16 completing (errored=False, cancelled=False)
[2017-05-10 10:25:21,812: INFO/job_scheduler] run_next: 1 runnable jobs of (4 pending, 3 tasked)
[2017-05-10 10:25:21,825: INFO/job_scheduler] Job 18: RunJobThread.run
[2017-05-10 10:25:21,826: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 10:25:21,844: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bd05e040-47eb-41b3-9c14-b0a1fb09c130
[2017-05-10 10:25:21,844: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bd05e040-47eb-41b3-9c14-b0a1fb09c130
[2017-05-10 10:25:24,097: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:25:24,098: INFO/job_scheduler]   Job 29 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:25:24,098: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 4 tasked)
[2017-05-10 10:25:24,114: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:25:24,114: INFO/job_scheduler]   Job 30 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:25:24,115: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 4 tasked)
[2017-05-10 10:25:24,136: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:25:24,136: INFO/job_scheduler]   Job 41 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:25:24,136: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 4 tasked)
[2017-05-10 10:25:24,594: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:25:24,594: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:25:24,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:25:24,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:25:24,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1718c32f-fd76-4bb4-b845-fcd8e7c3b753
[2017-05-10 10:25:24,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1718c32f-fd76-4bb4-b845-fcd8e7c3b753
[2017-05-10 10:25:24,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1718c32f-fd76-4bb4-b845-fcd8e7c3b753
[2017-05-10 10:25:24,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1718c32f-fd76-4bb4-b845-fcd8e7c3b753
[2017-05-10 10:25:24,613: INFO/job_scheduler] Job 29 finished 1 steps successfully
[2017-05-10 10:25:24,650: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 140516499186320
[2017-05-10 10:25:24,700: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 10:25:24,713: INFO/job] Job 29: StateChangeJob complete, setting state started on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 10:25:24,713: INFO/job] Job 29: not_deleted=True
[2017-05-10 10:25:24,713: INFO/job_scheduler] Job 29: complete_job: Updating cache
[2017-05-10 10:25:24,713: INFO/job_scheduler] Job 29: locked_item 140516499186320 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 10:25:24,716: INFO/job_scheduler] Job 29 completing (errored=False, cancelled=False)
[2017-05-10 10:25:24,762: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-2
[2017-05-10 10:25:24,762: INFO/job_scheduler] _notify: Updating .mcast_port of item 2 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 58263
[2017-05-10 10:25:24,767: INFO/job_scheduler] _notify: Set mcast_port=58263 on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-2) and saved
[2017-05-10 10:25:24,768: INFO/job_scheduler] _notify: Updating .network_interfaces of item 2 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-05-10 10:25:24,777: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-2) and saved
[2017-05-10 10:25:24,784: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 2 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-10 10:25:24,788: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-2) and saved
[2017-05-10 10:25:24,797: INFO/job_scheduler] run_next: 1 runnable jobs of (3 pending, 3 tasked)
[2017-05-10 10:25:24,821: INFO/job_scheduler] Job 30: RunJobThread.run
[2017-05-10 10:25:24,823: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-05-10 10:25:24,844: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 594d05fc-251f-48f7-bd4c-b93837e97004
[2017-05-10 10:25:24,844: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 594d05fc-251f-48f7-bd4c-b93837e97004
[2017-05-10 10:25:27,635: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:25:27,635: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:25:27,636: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:25:27,636: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:25:27,636: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dcd657f5-3272-46bd-9821-7917001e20de
[2017-05-10 10:25:27,636: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dcd657f5-3272-46bd-9821-7917001e20de
[2017-05-10 10:25:27,642: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dcd657f5-3272-46bd-9821-7917001e20de
[2017-05-10 10:25:27,642: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dcd657f5-3272-46bd-9821-7917001e20de
[2017-05-10 10:25:27,643: INFO/job_scheduler] Job 51 finished 1 steps successfully
[2017-05-10 10:25:27,700: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 140516499068816
[2017-05-10 10:25:27,731: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm6.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 10:25:27,751: INFO/job] Job 51: StateChangeJob complete, setting state started on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 10:25:27,752: INFO/job] Job 51: not_deleted=True
[2017-05-10 10:25:27,752: INFO/job_scheduler] Job 51: complete_job: Updating cache
[2017-05-10 10:25:27,753: INFO/job_scheduler] Job 51: locked_item 140516499068816 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 10:25:27,763: INFO/job_scheduler] Job 51 completing (errored=False, cancelled=False)
[2017-05-10 10:25:27,798: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-4
[2017-05-10 10:25:27,799: INFO/job_scheduler] _notify: Updating .mcast_port of item 4 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 49545
[2017-05-10 10:25:27,803: INFO/job_scheduler] _notify: Set mcast_port=49545 on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-4) and saved
[2017-05-10 10:25:27,804: INFO/job_scheduler] _notify: Updating .network_interfaces of item 4 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-05-10 10:25:27,813: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-4) and saved
[2017-05-10 10:25:27,819: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 4 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-10 10:25:27,824: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-4) and saved
[2017-05-10 10:25:27,833: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 3 tasked)
[2017-05-10 10:25:27,856: INFO/job_scheduler] Job 52: RunJobThread.run
[2017-05-10 10:25:27,858: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-05-10 10:25:27,874: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b9950639-756c-4d74-af51-4cdbe1b1358d
[2017-05-10 10:25:27,874: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b9950639-756c-4d74-af51-4cdbe1b1358d
[2017-05-10 10:25:28,397: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 4 tasked)
[2017-05-10 10:25:28,408: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 4 tasked)
[2017-05-10 10:25:28,432: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:25:28,432: INFO/job_scheduler]   Job 30 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:25:28,433: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 4 tasked)
[2017-05-10 10:25:28,449: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:25:28,449: INFO/job_scheduler]   Job 41 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:25:28,449: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 4 tasked)
[2017-05-10 10:25:28,924: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:25:28,924: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:25:28,924: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:25:28,924: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:25:28,924: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bd05e040-47eb-41b3-9c14-b0a1fb09c130
[2017-05-10 10:25:28,924: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bd05e040-47eb-41b3-9c14-b0a1fb09c130
[2017-05-10 10:25:28,960: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bd05e040-47eb-41b3-9c14-b0a1fb09c130
[2017-05-10 10:25:28,960: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bd05e040-47eb-41b3-9c14-b0a1fb09c130
[2017-05-10 10:25:28,961: INFO/job_scheduler] Job 18 finished 1 steps successfully
[2017-05-10 10:25:29,024: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 140516633537808
[2017-05-10 10:25:29,059: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 10:25:29,099: INFO/job] Job 18: StateChangeJob complete, setting state started on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 10:25:29,102: INFO/job] Job 18: not_deleted=True
[2017-05-10 10:25:29,107: INFO/job_scheduler] Job 18: complete_job: Updating cache
[2017-05-10 10:25:29,108: INFO/job_scheduler] Job 18: locked_item 140516633537808 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 10:25:29,119: INFO/job_scheduler] Job 18 completing (errored=False, cancelled=False)
[2017-05-10 10:25:29,161: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-1
[2017-05-10 10:25:29,161: INFO/job_scheduler] _notify: Updating .mcast_port of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 49545
[2017-05-10 10:25:29,166: INFO/job_scheduler] _notify: Set mcast_port=49545 on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-05-10 10:25:29,167: INFO/job_scheduler] _notify: Updating .network_interfaces of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-05-10 10:25:29,175: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-05-10 10:25:29,181: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-10 10:25:29,185: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-05-10 10:25:29,195: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 10:25:29,224: INFO/job_scheduler] Job 19: RunJobThread.run
[2017-05-10 10:25:29,226: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-05-10 10:25:29,238: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dbe63597-499a-4885-96ec-c8ce53c65f1e
[2017-05-10 10:25:29,238: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dbe63597-499a-4885-96ec-c8ce53c65f1e
[2017-05-10 10:25:29,253: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 10:25:29,259: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 10:25:29,266: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from True to False
[2017-05-10 10:25:29,271: INFO/job_scheduler] _notify: Set corosync_reported_up=False on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-05-10 10:25:29,281: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 10:25:29,297: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:25:29,297: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:25:29,297: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 10:25:29,302: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 4 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from True to False
[2017-05-10 10:25:29,306: INFO/job_scheduler] _notify: Set corosync_reported_up=False on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-4) and saved
[2017-05-10 10:25:29,323: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 10:25:29,333: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:25:29,333: INFO/job_scheduler]   Job 52 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:25:29,334: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 10:25:30,799: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 10:25:30,811: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 10:25:30,841: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:25:30,842: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:25:30,842: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 10:25:30,863: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:25:30,863: INFO/job_scheduler]   Job 52 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:25:30,863: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 10:25:34,166: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:25:34,167: INFO/job_scheduler]   Job 30 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:25:34,167: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 10:25:34,189: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:25:34,189: INFO/job_scheduler]   Job 41 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:25:34,189: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 10:25:35,338: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:25:35,338: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:25:35,338: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:25:35,338: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:25:35,339: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8c7f829a-1318-4f40-9bb3-c715d480c06f
[2017-05-10 10:25:35,339: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8c7f829a-1318-4f40-9bb3-c715d480c06f
[2017-05-10 10:25:35,344: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8c7f829a-1318-4f40-9bb3-c715d480c06f
[2017-05-10 10:25:35,344: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8c7f829a-1318-4f40-9bb3-c715d480c06f
[2017-05-10 10:25:35,346: INFO/job_scheduler] Job 41 finished 1 steps successfully
[2017-05-10 10:25:35,739: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 140516499068368
[2017-05-10 10:25:35,744: INFO/job] Job 41: StateChangeJob complete, setting state managed on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:25:35,744: INFO/job] Job 41: not_deleted=True
[2017-05-10 10:25:35,744: INFO/job_scheduler] Job 41: complete_job: Updating cache
[2017-05-10 10:25:35,745: INFO/job_scheduler] Job 41: locked_item 140516499068368 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 10:25:35,747: INFO/job_scheduler] Job 41 completing (errored=False, cancelled=False)
[2017-05-10 10:25:35,793: INFO/job_scheduler] Replaying 4 buffered notifications for ManagedHost-3
[2017-05-10 10:25:35,801: INFO/job_scheduler] _notify: Updating .immutable_state of item 3 (lotus-32vm8.lotus.hpdd.lab.intel.com) from True to False
[2017-05-10 10:25:35,810: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-3) and saved
[2017-05-10 10:25:35,851: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 3 (lotus-32vm8.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7fcc9010ec10> to [2]
[2017-05-10 10:25:35,864: INFO/job_scheduler] _notify: Set ha_cluster_peers=[2] on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-3) and saved
[2017-05-10 10:25:35,926: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 10:25:39,251: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:25:39,251: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:25:39,251: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 10:25:39,273: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:25:39,273: INFO/job_scheduler]   Job 52 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:25:39,274: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 10:25:40,816: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:25:40,817: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:25:40,817: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 10:25:40,842: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:25:40,842: INFO/job_scheduler]   Job 52 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:25:40,842: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 10:25:49,302: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-10 10:25:49,307: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-05-10 10:25:49,314: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 10:25:49,321: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:25:49,321: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:25:49,322: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 10:25:49,361: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 4 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-10 10:25:49,365: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-4) and saved
[2017-05-10 10:25:49,372: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 10:25:49,380: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:25:49,380: INFO/job_scheduler]   Job 52 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:25:49,380: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 10:25:50,847: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:25:50,847: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:25:50,848: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 10:25:50,871: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:25:50,871: INFO/job_scheduler]   Job 52 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:25:50,871: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 10:25:51,242: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:25:51,242: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:25:51,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:25:51,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:25:51,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 594d05fc-251f-48f7-bd4c-b93837e97004
[2017-05-10 10:25:51,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 594d05fc-251f-48f7-bd4c-b93837e97004
[2017-05-10 10:25:51,273: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 594d05fc-251f-48f7-bd4c-b93837e97004
[2017-05-10 10:25:51,273: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 594d05fc-251f-48f7-bd4c-b93837e97004
[2017-05-10 10:25:51,275: INFO/job_scheduler] Job 30 finished 1 steps successfully
[2017-05-10 10:25:51,678: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 140516297766352
[2017-05-10 10:25:51,683: INFO/job] Job 30: StateChangeJob complete, setting state managed on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:25:51,684: INFO/job] Job 30: not_deleted=True
[2017-05-10 10:25:51,684: INFO/job_scheduler] Job 30: complete_job: Updating cache
[2017-05-10 10:25:51,684: INFO/job_scheduler] Job 30: locked_item 140516297766352 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 10:25:51,687: INFO/job_scheduler] Job 30 completing (errored=False, cancelled=False)
[2017-05-10 10:25:51,754: INFO/job_scheduler] Replaying 4 buffered notifications for ManagedHost-2
[2017-05-10 10:25:51,774: INFO/job_scheduler] _notify: Updating .immutable_state of item 2 (lotus-32vm7.lotus.hpdd.lab.intel.com) from True to False
[2017-05-10 10:25:51,779: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-2) and saved
[2017-05-10 10:25:51,794: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 2 (lotus-32vm7.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7fcc9009e3d0> to [3]
[2017-05-10 10:25:51,804: INFO/job_scheduler] _notify: Set ha_cluster_peers=[3] on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-2) and saved
[2017-05-10 10:25:51,864: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-05-10 10:25:54,292: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:25:54,292: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:25:54,292: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:25:54,292: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:25:54,292: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b9950639-756c-4d74-af51-4cdbe1b1358d
[2017-05-10 10:25:54,292: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b9950639-756c-4d74-af51-4cdbe1b1358d
[2017-05-10 10:25:54,343: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b9950639-756c-4d74-af51-4cdbe1b1358d
[2017-05-10 10:25:54,343: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b9950639-756c-4d74-af51-4cdbe1b1358d
[2017-05-10 10:25:54,345: INFO/job_scheduler] Job 52 finished 1 steps successfully
[2017-05-10 10:25:54,754: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 140516499244368
[2017-05-10 10:25:54,760: INFO/job] Job 52: StateChangeJob complete, setting state managed on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:25:54,761: INFO/job] Job 52: not_deleted=True
[2017-05-10 10:25:54,761: INFO/job_scheduler] Job 52: complete_job: Updating cache
[2017-05-10 10:25:54,761: INFO/job_scheduler] Job 52: locked_item 140516499244368 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 10:25:54,764: INFO/job_scheduler] Job 52 completing (errored=False, cancelled=False)
[2017-05-10 10:25:54,838: INFO/job_scheduler] Replaying 4 buffered notifications for ManagedHost-4
[2017-05-10 10:25:54,846: INFO/job_scheduler] _notify: Updating .immutable_state of item 4 (lotus-32vm6.lotus.hpdd.lab.intel.com) from True to False
[2017-05-10 10:25:54,851: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-4) and saved
[2017-05-10 10:25:54,871: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 4 (lotus-32vm6.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7fcc900b4750> to [1]
[2017-05-10 10:25:54,882: INFO/job_scheduler] _notify: Set ha_cluster_peers=[1] on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-4) and saved
[2017-05-10 10:25:54,951: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 10:25:56,636: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:25:56,636: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:25:56,636: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:25:56,636: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:25:56,636: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dbe63597-499a-4885-96ec-c8ce53c65f1e
[2017-05-10 10:25:56,636: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dbe63597-499a-4885-96ec-c8ce53c65f1e
[2017-05-10 10:25:56,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dbe63597-499a-4885-96ec-c8ce53c65f1e
[2017-05-10 10:25:56,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dbe63597-499a-4885-96ec-c8ce53c65f1e
[2017-05-10 10:25:56,663: INFO/job_scheduler] Job 19 finished 1 steps successfully
[2017-05-10 10:25:57,029: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 140516499180240
[2017-05-10 10:25:57,035: INFO/job] Job 19: StateChangeJob complete, setting state managed on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:25:57,035: INFO/job] Job 19: not_deleted=True
[2017-05-10 10:25:57,036: INFO/job_scheduler] Job 19: complete_job: Updating cache
[2017-05-10 10:25:57,036: INFO/job_scheduler] Job 19: locked_item 140516499180240 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 10:25:57,039: INFO/job_scheduler] Job 19 completing (errored=False, cancelled=False)
[2017-05-10 10:25:57,123: INFO/job_scheduler] Replaying 4 buffered notifications for ManagedHost-1
[2017-05-10 10:25:57,130: INFO/job_scheduler] _notify: Updating .immutable_state of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com) from True to False
[2017-05-10 10:25:57,135: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-1) and saved
[2017-05-10 10:25:57,152: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7fcc9005c450> to [4]
[2017-05-10 10:25:57,164: INFO/job_scheduler] _notify: Set ha_cluster_peers=[4] on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-1) and saved
[2017-05-10 10:25:57,251: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:27:02,312: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:27:02,312: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 10:27:02,341: INFO/command_plan] add_jobs: created Job 53 (Trigger plugin poll for linux plugins)
[2017-05-10 10:27:02,385: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 10:27:02,389: INFO/job_scheduler] Job 53: RunJobThread.run
[2017-05-10 10:27:02,390: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 10:27:02,401: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 10:27:02,411: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ee9485d8-a493-47a5-b111-d0ba406c94d0
[2017-05-10 10:27:02,411: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ee9485d8-a493-47a5-b111-d0ba406c94d0
[2017-05-10 10:27:02,423: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 10:27:02,427: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c8eb5d4a-9600-4de7-9bdd-2d65225dcdd2
[2017-05-10 10:27:02,427: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c8eb5d4a-9600-4de7-9bdd-2d65225dcdd2
[2017-05-10 10:27:02,440: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0ac1c108-69b7-43d4-a6fe-aa333825353a
[2017-05-10 10:27:02,440: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0ac1c108-69b7-43d4-a6fe-aa333825353a
[2017-05-10 10:27:02,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:27:02,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:27:02,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:27:02,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:27:02,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c8eb5d4a-9600-4de7-9bdd-2d65225dcdd2
[2017-05-10 10:27:02,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c8eb5d4a-9600-4de7-9bdd-2d65225dcdd2
[2017-05-10 10:27:02,497: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:27:02,497: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:27:02,497: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:27:02,497: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:27:02,498: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0ac1c108-69b7-43d4-a6fe-aa333825353a
[2017-05-10 10:27:02,498: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0ac1c108-69b7-43d4-a6fe-aa333825353a
[2017-05-10 10:27:02,503: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:27:02,503: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:27:02,503: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:27:02,503: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:27:02,503: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ee9485d8-a493-47a5-b111-d0ba406c94d0
[2017-05-10 10:27:02,503: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ee9485d8-a493-47a5-b111-d0ba406c94d0
[2017-05-10 10:27:02,504: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c8eb5d4a-9600-4de7-9bdd-2d65225dcdd2
[2017-05-10 10:27:02,504: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c8eb5d4a-9600-4de7-9bdd-2d65225dcdd2
[2017-05-10 10:27:02,504: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 10:27:02,505: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0ac1c108-69b7-43d4-a6fe-aa333825353a
[2017-05-10 10:27:02,505: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0ac1c108-69b7-43d4-a6fe-aa333825353a
[2017-05-10 10:27:02,505: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 10:27:02,548: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ee9485d8-a493-47a5-b111-d0ba406c94d0
[2017-05-10 10:27:02,548: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ee9485d8-a493-47a5-b111-d0ba406c94d0
[2017-05-10 10:27:02,548: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 10:27:02,548: INFO/job_scheduler] Job 53 finished 1 steps successfully
[2017-05-10 10:27:02,609: INFO/job_scheduler] Job 53: complete_job: Updating cache
[2017-05-10 10:27:02,609: INFO/job_scheduler] Job 53 completing (errored=False, cancelled=False)
[2017-05-10 10:27:02,633: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:27:52,974: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:27:52,975: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 10:27:52,996: INFO/command_plan] add_jobs: created Job 54 (Trigger plugin poll for linux plugins)
[2017-05-10 10:27:53,011: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 10:27:53,015: INFO/job_scheduler] Job 54: RunJobThread.run
[2017-05-10 10:27:53,016: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 10:27:53,028: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 10:27:53,028: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 10:27:53,038: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cf3c4644-3d49-440c-94c3-8895bc2f031c
[2017-05-10 10:27:53,038: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cf3c4644-3d49-440c-94c3-8895bc2f031c
[2017-05-10 10:27:53,076: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc adb32ed5-c26b-4f26-8519-f28a25c5baf4
[2017-05-10 10:27:53,076: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc adb32ed5-c26b-4f26-8519-f28a25c5baf4
[2017-05-10 10:27:53,105: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bf5f256c-e7e3-491b-ae37-b37b9c640bf9
[2017-05-10 10:27:53,105: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bf5f256c-e7e3-491b-ae37-b37b9c640bf9
[2017-05-10 10:27:53,113: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:27:53,113: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:27:53,113: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:27:53,113: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:27:53,113: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cf3c4644-3d49-440c-94c3-8895bc2f031c
[2017-05-10 10:27:53,113: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cf3c4644-3d49-440c-94c3-8895bc2f031c
[2017-05-10 10:27:53,142: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:27:53,142: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:27:53,143: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:27:53,143: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:27:53,143: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc adb32ed5-c26b-4f26-8519-f28a25c5baf4
[2017-05-10 10:27:53,143: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc adb32ed5-c26b-4f26-8519-f28a25c5baf4
[2017-05-10 10:27:53,143: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cf3c4644-3d49-440c-94c3-8895bc2f031c
[2017-05-10 10:27:53,143: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cf3c4644-3d49-440c-94c3-8895bc2f031c
[2017-05-10 10:27:53,143: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 10:27:53,160: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc adb32ed5-c26b-4f26-8519-f28a25c5baf4
[2017-05-10 10:27:53,160: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc adb32ed5-c26b-4f26-8519-f28a25c5baf4
[2017-05-10 10:27:53,160: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 10:27:53,180: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:27:53,180: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:27:53,181: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:27:53,181: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:27:53,181: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bf5f256c-e7e3-491b-ae37-b37b9c640bf9
[2017-05-10 10:27:53,181: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bf5f256c-e7e3-491b-ae37-b37b9c640bf9
[2017-05-10 10:27:53,221: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bf5f256c-e7e3-491b-ae37-b37b9c640bf9
[2017-05-10 10:27:53,221: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bf5f256c-e7e3-491b-ae37-b37b9c640bf9
[2017-05-10 10:27:53,221: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 10:27:53,221: INFO/job_scheduler] Job 54 finished 1 steps successfully
[2017-05-10 10:27:53,238: INFO/job_scheduler] Job 54: complete_job: Updating cache
[2017-05-10 10:27:53,238: INFO/job_scheduler] Job 54 completing (errored=False, cancelled=False)
[2017-05-10 10:27:53,280: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:27:57,810: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:27:57,812: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 10:27:57,835: INFO/command_plan] add_jobs: created Job 55 (Trigger plugin poll for linux plugins)
[2017-05-10 10:27:57,847: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 10:27:57,850: INFO/job_scheduler] Job 55: RunJobThread.run
[2017-05-10 10:27:57,852: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 10:27:57,870: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 026224d0-cbe6-454d-b120-7fc0ca468215
[2017-05-10 10:27:57,870: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 026224d0-cbe6-454d-b120-7fc0ca468215
[2017-05-10 10:27:57,871: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 10:27:57,889: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 10:27:57,890: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7091a85d-ecf3-415e-b081-3de0bb721136
[2017-05-10 10:27:57,890: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7091a85d-ecf3-415e-b081-3de0bb721136
[2017-05-10 10:27:57,904: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 18bd625f-18de-4326-a6d5-13230884c153
[2017-05-10 10:27:57,904: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 18bd625f-18de-4326-a6d5-13230884c153
[2017-05-10 10:27:57,937: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:27:57,937: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:27:57,938: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:27:57,938: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:27:57,938: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 026224d0-cbe6-454d-b120-7fc0ca468215
[2017-05-10 10:27:57,938: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 026224d0-cbe6-454d-b120-7fc0ca468215
[2017-05-10 10:27:57,953: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 026224d0-cbe6-454d-b120-7fc0ca468215
[2017-05-10 10:27:57,953: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 026224d0-cbe6-454d-b120-7fc0ca468215
[2017-05-10 10:27:57,953: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 10:27:57,957: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:27:57,957: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:27:57,957: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:27:57,957: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:27:57,957: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7091a85d-ecf3-415e-b081-3de0bb721136
[2017-05-10 10:27:57,957: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7091a85d-ecf3-415e-b081-3de0bb721136
[2017-05-10 10:27:57,973: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7091a85d-ecf3-415e-b081-3de0bb721136
[2017-05-10 10:27:57,973: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7091a85d-ecf3-415e-b081-3de0bb721136
[2017-05-10 10:27:57,974: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 10:27:58,002: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:27:58,002: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:27:58,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:27:58,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:27:58,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 18bd625f-18de-4326-a6d5-13230884c153
[2017-05-10 10:27:58,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 18bd625f-18de-4326-a6d5-13230884c153
[2017-05-10 10:27:58,025: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 18bd625f-18de-4326-a6d5-13230884c153
[2017-05-10 10:27:58,025: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 18bd625f-18de-4326-a6d5-13230884c153
[2017-05-10 10:27:58,025: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 10:27:58,025: INFO/job_scheduler] Job 55 finished 1 steps successfully
[2017-05-10 10:27:58,068: INFO/job_scheduler] Job 55: complete_job: Updating cache
[2017-05-10 10:27:58,069: INFO/job_scheduler] Job 55 completing (errored=False, cancelled=False)
[2017-05-10 10:27:58,119: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:27:58,422: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:27:58,423: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 10:27:58,436: INFO/command_plan] add_jobs: created Job 56 (Trigger plugin poll for linux plugins)
[2017-05-10 10:27:58,450: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 10:27:58,454: INFO/job_scheduler] Job 56: RunJobThread.run
[2017-05-10 10:27:58,455: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 10:27:58,464: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 10:27:58,465: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 10:27:58,469: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 26477ec3-1f64-4d14-994c-f50314d36658
[2017-05-10 10:27:58,469: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 26477ec3-1f64-4d14-994c-f50314d36658
[2017-05-10 10:27:58,483: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3402650a-5796-469d-b0d7-ca256f1afd5e
[2017-05-10 10:27:58,483: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3402650a-5796-469d-b0d7-ca256f1afd5e
[2017-05-10 10:27:58,496: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 23b94145-bf28-48bb-b555-43511ad3a7d3
[2017-05-10 10:27:58,496: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 23b94145-bf28-48bb-b555-43511ad3a7d3
[2017-05-10 10:27:58,529: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:27:58,529: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:27:58,529: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:27:58,529: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:27:58,530: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 26477ec3-1f64-4d14-994c-f50314d36658
[2017-05-10 10:27:58,530: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 26477ec3-1f64-4d14-994c-f50314d36658
[2017-05-10 10:27:58,531: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:27:58,531: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:27:58,531: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:27:58,531: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:27:58,531: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3402650a-5796-469d-b0d7-ca256f1afd5e
[2017-05-10 10:27:58,531: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3402650a-5796-469d-b0d7-ca256f1afd5e
[2017-05-10 10:27:58,541: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 26477ec3-1f64-4d14-994c-f50314d36658
[2017-05-10 10:27:58,541: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 26477ec3-1f64-4d14-994c-f50314d36658
[2017-05-10 10:27:58,541: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 10:27:58,560: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3402650a-5796-469d-b0d7-ca256f1afd5e
[2017-05-10 10:27:58,560: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3402650a-5796-469d-b0d7-ca256f1afd5e
[2017-05-10 10:27:58,560: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 10:27:58,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:27:58,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:27:58,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:27:58,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:27:58,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 23b94145-bf28-48bb-b555-43511ad3a7d3
[2017-05-10 10:27:58,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 23b94145-bf28-48bb-b555-43511ad3a7d3
[2017-05-10 10:27:58,614: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 23b94145-bf28-48bb-b555-43511ad3a7d3
[2017-05-10 10:27:58,614: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 23b94145-bf28-48bb-b555-43511ad3a7d3
[2017-05-10 10:27:58,615: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 10:27:58,615: INFO/job_scheduler] Job 56 finished 1 steps successfully
[2017-05-10 10:27:58,670: INFO/job_scheduler] Job 56: complete_job: Updating cache
[2017-05-10 10:27:58,670: INFO/job_scheduler] Job 56 completing (errored=False, cancelled=False)
[2017-05-10 10:27:58,728: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:28:00,511: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:28:00,513: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 10:28:00,527: INFO/command_plan] add_jobs: created Job 57 (Trigger plugin poll for linux plugins)
[2017-05-10 10:28:00,546: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 10:28:00,550: INFO/job_scheduler] Job 57: RunJobThread.run
[2017-05-10 10:28:00,551: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 10:28:00,568: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 89c56f7b-f3a4-44c8-9aa4-e4ead2b76f9e
[2017-05-10 10:28:00,568: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 89c56f7b-f3a4-44c8-9aa4-e4ead2b76f9e
[2017-05-10 10:28:00,569: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 10:28:00,583: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1f826223-d953-4ea3-b922-836f9fef4b46
[2017-05-10 10:28:00,583: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1f826223-d953-4ea3-b922-836f9fef4b46
[2017-05-10 10:28:00,583: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 10:28:00,597: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5f82b6a3-6ada-48ef-aa9f-093e1eb48422
[2017-05-10 10:28:00,597: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5f82b6a3-6ada-48ef-aa9f-093e1eb48422
[2017-05-10 10:28:00,688: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:28:00,688: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:28:00,688: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:28:00,688: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:28:00,689: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 89c56f7b-f3a4-44c8-9aa4-e4ead2b76f9e
[2017-05-10 10:28:00,689: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 89c56f7b-f3a4-44c8-9aa4-e4ead2b76f9e
[2017-05-10 10:28:00,690: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:28:00,690: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:28:00,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:28:00,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:28:00,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1f826223-d953-4ea3-b922-836f9fef4b46
[2017-05-10 10:28:00,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1f826223-d953-4ea3-b922-836f9fef4b46
[2017-05-10 10:28:00,695: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:28:00,695: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:28:00,695: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:28:00,695: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:28:00,695: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5f82b6a3-6ada-48ef-aa9f-093e1eb48422
[2017-05-10 10:28:00,695: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5f82b6a3-6ada-48ef-aa9f-093e1eb48422
[2017-05-10 10:28:00,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 89c56f7b-f3a4-44c8-9aa4-e4ead2b76f9e
[2017-05-10 10:28:00,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 89c56f7b-f3a4-44c8-9aa4-e4ead2b76f9e
[2017-05-10 10:28:00,702: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 10:28:00,710: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1f826223-d953-4ea3-b922-836f9fef4b46
[2017-05-10 10:28:00,710: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1f826223-d953-4ea3-b922-836f9fef4b46
[2017-05-10 10:28:00,710: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 10:28:00,713: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5f82b6a3-6ada-48ef-aa9f-093e1eb48422
[2017-05-10 10:28:00,713: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5f82b6a3-6ada-48ef-aa9f-093e1eb48422
[2017-05-10 10:28:00,713: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 10:28:00,715: INFO/job_scheduler] Job 57 finished 1 steps successfully
[2017-05-10 10:28:00,761: INFO/job_scheduler] Job 57: complete_job: Updating cache
[2017-05-10 10:28:00,761: INFO/job_scheduler] Job 57 completing (errored=False, cancelled=False)
[2017-05-10 10:28:00,826: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:31:49,703: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:31:49,721: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:31:50,405: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:31:50,429: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:31:51,899: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:31:51,915: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:31:55,553: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:31:55,568: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:33:41,245: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:33:41,245: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/04ea8004-2bbe-4673-b81c-9eef6595a6e5
[2017-05-10 10:33:41,246: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 04ea8004-2bbe-4673-b81c-9eef6595a6e5 because session terminated
[2017-05-10 10:33:41,246: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 04ea8004-2bbe-4673-b81c-9eef6595a6e5 because session terminated
[2017-05-10 10:33:41,253: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/c4c20899-8d80-4f1a-a02e-6449f9afeef0
[2017-05-10 10:33:41,253: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/c4c20899-8d80-4f1a-a02e-6449f9afeef0
[2017-05-10 10:33:41,291: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:33:41,291: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/c7969cb3-3257-474f-bf10-a9e2c95e4ee1
[2017-05-10 10:33:41,291: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session c7969cb3-3257-474f-bf10-a9e2c95e4ee1 because session terminated
[2017-05-10 10:33:41,291: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session c7969cb3-3257-474f-bf10-a9e2c95e4ee1 because session terminated
[2017-05-10 10:33:41,294: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/4485f89b-8837-419e-8454-5a311b0e6496
[2017-05-10 10:33:41,294: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/4485f89b-8837-419e-8454-5a311b0e6496
[2017-05-10 10:33:41,336: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:33:41,336: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/37c1f3d3-cbfc-46c4-8ced-1dada7328b4a
[2017-05-10 10:33:41,336: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 37c1f3d3-cbfc-46c4-8ced-1dada7328b4a because session terminated
[2017-05-10 10:33:41,336: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 37c1f3d3-cbfc-46c4-8ced-1dada7328b4a because session terminated
[2017-05-10 10:33:41,338: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/863fad36-d22c-4233-b9a8-2f30efc80c4f
[2017-05-10 10:33:41,338: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/863fad36-d22c-4233-b9a8-2f30efc80c4f
[2017-05-10 10:33:45,285: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:33:45,285: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/aaed99a8-f16d-4619-a9ec-a9c8d9aafafc
[2017-05-10 10:33:45,286: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session aaed99a8-f16d-4619-a9ec-a9c8d9aafafc because session terminated
[2017-05-10 10:33:45,286: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session aaed99a8-f16d-4619-a9ec-a9c8d9aafafc because session terminated
[2017-05-10 10:33:45,287: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/6e5b9046-5240-47b9-8fed-59898e7139e7
[2017-05-10 10:33:45,287: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/6e5b9046-5240-47b9-8fed-59898e7139e7
[2017-05-10 10:33:45,363: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:33:45,379: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:33:45,425: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:33:45,442: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:33:45,486: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:33:45,500: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:33:49,316: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:33:49,335: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:33:51,357: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:33:52,498: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:33:53,119: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:33:56,288: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:40:20,018: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/4485f89b-8837-419e-8454-5a311b0e6496
[2017-05-10 10:40:20,018: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/4485f89b-8837-419e-8454-5a311b0e6496
[2017-05-10 10:40:20,019: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 4485f89b-8837-419e-8454-5a311b0e6496 because session terminated
[2017-05-10 10:40:20,019: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 4485f89b-8837-419e-8454-5a311b0e6496 because session terminated
[2017-05-10 10:40:20,036: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/6e5b9046-5240-47b9-8fed-59898e7139e7
[2017-05-10 10:40:20,036: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/6e5b9046-5240-47b9-8fed-59898e7139e7
[2017-05-10 10:40:20,036: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 6e5b9046-5240-47b9-8fed-59898e7139e7 because session terminated
[2017-05-10 10:40:20,036: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 6e5b9046-5240-47b9-8fed-59898e7139e7 because session terminated
[2017-05-10 10:40:20,091: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/c4c20899-8d80-4f1a-a02e-6449f9afeef0
[2017-05-10 10:40:20,091: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/c4c20899-8d80-4f1a-a02e-6449f9afeef0
[2017-05-10 10:40:20,091: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session c4c20899-8d80-4f1a-a02e-6449f9afeef0 because session terminated
[2017-05-10 10:40:20,091: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session c4c20899-8d80-4f1a-a02e-6449f9afeef0 because session terminated
[2017-05-10 10:40:20,104: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/863fad36-d22c-4233-b9a8-2f30efc80c4f
[2017-05-10 10:40:20,104: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/863fad36-d22c-4233-b9a8-2f30efc80c4f
[2017-05-10 10:40:20,104: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 863fad36-d22c-4233-b9a8-2f30efc80c4f because session terminated
[2017-05-10 10:40:20,104: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 863fad36-d22c-4233-b9a8-2f30efc80c4f because session terminated
[2017-05-10 10:42:26,941: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:42:26,942: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 10:42:26,955: INFO/command_plan] add_jobs: created Job 58 (Test for host connectivity)
[2017-05-10 10:42:26,975: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 10:42:26,978: INFO/job_scheduler] Job 58: RunJobThread.run
[2017-05-10 10:42:27,081: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:42:27,082: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 10:42:27,114: INFO/command_plan] add_jobs: created Job 59 (Test for host connectivity)
[2017-05-10 10:42:27,139: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 10:42:27,141: INFO/job_scheduler] Job 59: RunJobThread.run
[2017-05-10 10:42:27,178: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:42:27,178: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 10:42:27,213: INFO/command_plan] add_jobs: created Job 60 (Test for host connectivity)
[2017-05-10 10:42:27,260: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 10:42:27,263: INFO/job_scheduler] Job 60: RunJobThread.run
[2017-05-10 10:42:27,298: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:42:27,299: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 10:42:27,572: INFO/command_plan] add_jobs: created Job 61 (Test for host connectivity)
[2017-05-10 10:42:27,667: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 10:42:27,699: INFO/job_scheduler] Job 61: RunJobThread.run
[2017-05-10 10:42:30,640: INFO/job_scheduler] Job 58 finished 1 steps successfully
[2017-05-10 10:42:30,671: INFO/job_scheduler] Job 58: complete_job: Updating cache
[2017-05-10 10:42:30,671: INFO/job_scheduler] Job 58 completing (errored=False, cancelled=False)
[2017-05-10 10:42:30,738: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 10:42:31,168: INFO/job_scheduler] Job 59 finished 1 steps successfully
[2017-05-10 10:42:31,196: INFO/job_scheduler] Job 59: complete_job: Updating cache
[2017-05-10 10:42:31,196: INFO/job_scheduler] Job 59 completing (errored=False, cancelled=False)
[2017-05-10 10:42:31,249: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-05-10 10:42:31,339: INFO/job_scheduler] Job 60 finished 1 steps successfully
[2017-05-10 10:42:31,389: INFO/job_scheduler] Job 60: complete_job: Updating cache
[2017-05-10 10:42:31,389: INFO/job_scheduler] Job 60 completing (errored=False, cancelled=False)
[2017-05-10 10:42:31,467: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 10:42:31,953: INFO/job_scheduler] Job 61 finished 1 steps successfully
[2017-05-10 10:42:31,992: INFO/job_scheduler] Job 61: complete_job: Updating cache
[2017-05-10 10:42:31,993: INFO/job_scheduler] Job 61 completing (errored=False, cancelled=False)
[2017-05-10 10:42:32,038: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:42:34,685: ERROR/rpc] RunOneRpc: exception calling create_host_ssh: Traceback (most recent call last):

  File "/usr/share/chroma-manager/chroma_core/services/rpc.py", line 141, in run

  File "/usr/share/chroma-manager/chroma_core/services/rpc.py", line 623, in _local_call

  File "/usr/share/chroma-manager/chroma_core/services/job_scheduler/job_scheduler.py", line 1224, in create_host_ssh

AssertionError

[2017-05-10 10:42:34,694: ERROR/rpc] RunOneRpc: exception calling create_host_ssh: Traceback (most recent call last):

  File "/usr/share/chroma-manager/chroma_core/services/rpc.py", line 141, in run

  File "/usr/share/chroma-manager/chroma_core/services/rpc.py", line 623, in _local_call

  File "/usr/share/chroma-manager/chroma_core/services/job_scheduler/job_scheduler.py", line 1224, in create_host_ssh

AssertionError

[2017-05-10 10:42:34,703: ERROR/rpc] RunOneRpc: exception calling create_host_ssh: Traceback (most recent call last):

  File "/usr/share/chroma-manager/chroma_core/services/rpc.py", line 141, in run

  File "/usr/share/chroma-manager/chroma_core/services/rpc.py", line 623, in _local_call

  File "/usr/share/chroma-manager/chroma_core/services/job_scheduler/job_scheduler.py", line 1224, in create_host_ssh

AssertionError

[2017-05-10 10:42:34,712: ERROR/rpc] RunOneRpc: exception calling create_host_ssh: Traceback (most recent call last):

  File "/usr/share/chroma-manager/chroma_core/services/rpc.py", line 141, in run

  File "/usr/share/chroma-manager/chroma_core/services/rpc.py", line 623, in _local_call

  File "/usr/share/chroma-manager/chroma_core/services/job_scheduler/job_scheduler.py", line 1224, in create_host_ssh

AssertionError

[2017-05-10 10:44:05,857: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:44:05,883: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 10:44:05,918: INFO/command_plan] add_jobs: created Job 62 (Force remove host lotus-32vm5.lotus.hpdd.lab.intel.com from configuration)
[2017-05-10 10:44:05,927: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 10:44:05,942: INFO/job_scheduler] Job 62: RunJobThread.run
[2017-05-10 10:44:05,998: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:44:05,998: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration configured->removed (intentional=False) 140516567117008
[2017-05-10 10:44:06,007: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.ntp.NTPConfiguration'> 1
[2017-05-10 10:44:06,043: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:44:06,043: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration started->removed (intentional=False) 140516566809424
[2017-05-10 10:44:06,071: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-10 10:44:06,116: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.corosync2.Corosync2Configuration'> 1
[2017-05-10 10:44:06,138: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:44:06,138: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration configured->removed (intentional=False) 140516566473424
[2017-05-10 10:44:06,149: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> 1
[2017-05-10 10:44:06,170: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:44:06,170: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_up->removed (intentional=False) 140516566472784
[2017-05-10 10:44:06,187: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 10:44:06,193: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> 1
[2017-05-10 10:44:06,253: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:44:06,253: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration started->removed (intentional=False) 140516567061200
[2017-05-10 10:44:06,285: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-10 10:44:06,340: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> 1
[2017-05-10 10:44:07,610: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.host.ManagedHost'> 1
[2017-05-10 10:44:07,624: INFO/job_scheduler] Job 62 finished 1 steps successfully
[2017-05-10 10:44:07,669: INFO/job_scheduler] Job 62: complete_job: Updating cache
[2017-05-10 10:44:07,669: INFO/job_scheduler] Job 62: locked_item 140516566808272 <class 'chroma_core.models.host.ManagedHost'> True None
[2017-05-10 10:44:07,669: INFO/job_scheduler] Job 62: locked_item 140516566683536 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-10 10:44:07,672: INFO/job_scheduler] Job 62: locked_item 140516567060816 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 10:44:07,686: INFO/job_scheduler] Job 62: locked_item 140516566733520 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-10 10:44:07,689: INFO/job_scheduler] Job 62: locked_item 140516566571472 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 10:44:07,692: INFO/job_scheduler] Job 62: locked_item 140516566683280 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 10:44:07,701: INFO/job_scheduler] Job 62 completing (errored=False, cancelled=False)
[2017-05-10 10:44:07,830: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:44:19,830: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:44:19,849: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 10:44:19,891: INFO/command_plan] add_jobs: created Job 63 (Force remove host lotus-32vm6.lotus.hpdd.lab.intel.com from configuration)
[2017-05-10 10:44:19,907: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 10:44:19,927: INFO/job_scheduler] Job 63: RunJobThread.run
[2017-05-10 10:44:19,973: INFO/job] Deleting dependent lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration for host lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:44:19,973: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration started->removed (intentional=False) 140516566681808
[2017-05-10 10:44:19,992: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-10 10:44:20,017: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> 4
[2017-05-10 10:44:20,055: INFO/job] Deleting dependent lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration for host lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:44:20,055: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration configured->removed (intentional=False) 140516566733520
[2017-05-10 10:44:20,063: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.ntp.NTPConfiguration'> 4
[2017-05-10 10:44:20,080: INFO/job] Deleting dependent lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration for host lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:44:20,080: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration started->removed (intentional=False) 140516567139792
[2017-05-10 10:44:20,095: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-10 10:44:20,118: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.corosync2.Corosync2Configuration'> 4
[2017-05-10 10:44:20,155: INFO/job] Deleting dependent lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration for host lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:44:20,155: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration configured->removed (intentional=False) 140516566683280
[2017-05-10 10:44:20,166: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> 4
[2017-05-10 10:44:20,180: INFO/job] Deleting dependent lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration for host lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:44:20,180: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration lnet_up->removed (intentional=False) 140516566688912
[2017-05-10 10:44:20,195: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 10:44:20,209: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> 4
[2017-05-10 10:44:21,040: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.host.ManagedHost'> 4
[2017-05-10 10:44:21,045: INFO/job_scheduler] Job 63 finished 1 steps successfully
[2017-05-10 10:44:21,120: INFO/job_scheduler] Job 63: complete_job: Updating cache
[2017-05-10 10:44:21,120: INFO/job_scheduler] Job 63: locked_item 140516566629776 <class 'chroma_core.models.host.ManagedHost'> True None
[2017-05-10 10:44:21,120: INFO/job_scheduler] Job 63: locked_item 140516567138768 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 10:44:21,123: INFO/job_scheduler] Job 63: locked_item 140516567140880 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-10 10:44:21,126: INFO/job_scheduler] Job 63: locked_item 140516567139280 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 10:44:21,129: INFO/job_scheduler] Job 63: locked_item 140516567141200 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-10 10:44:21,132: INFO/job_scheduler] Job 63: locked_item 140516566614480 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 10:44:21,134: INFO/job_scheduler] Job 63 completing (errored=False, cancelled=False)
[2017-05-10 10:44:21,213: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:44:28,851: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:44:28,876: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 10:44:28,901: INFO/command_plan] add_jobs: created Job 64 (Force remove host lotus-32vm7.lotus.hpdd.lab.intel.com from configuration)
[2017-05-10 10:44:28,933: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 10:44:28,972: INFO/job_scheduler] Job 64: RunJobThread.run
[2017-05-10 10:44:29,073: INFO/job] Deleting dependent lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration for host lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:44:29,073: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration lnet_up->removed (intentional=False) 140516566538896
[2017-05-10 10:44:29,093: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 10:44:29,117: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> 2
[2017-05-10 10:44:29,145: INFO/job] Deleting dependent lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration for host lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:44:29,145: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration started->removed (intentional=False) 140516566539344
[2017-05-10 10:44:29,180: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-10 10:44:29,193: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.corosync2.Corosync2Configuration'> 2
[2017-05-10 10:44:29,222: INFO/job] Deleting dependent lotus-32vm7.lotus.hpdd.lab.intel.com NTP configuration for host lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:44:29,223: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com NTP configuration configured->removed (intentional=False) 140516297765200
[2017-05-10 10:44:29,235: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.ntp.NTPConfiguration'> 2
[2017-05-10 10:44:29,253: INFO/job] Deleting dependent lotus-32vm7.lotus.hpdd.lab.intel.com RSyslog configuration for host lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:44:29,254: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com RSyslog configuration configured->removed (intentional=False) 140516566610832
[2017-05-10 10:44:29,260: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> 2
[2017-05-10 10:44:29,288: INFO/job] Deleting dependent lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration for host lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:44:29,288: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration started->removed (intentional=False) 140516566610128
[2017-05-10 10:44:29,327: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-10 10:44:29,333: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> 2
[2017-05-10 10:44:30,105: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.host.ManagedHost'> 2
[2017-05-10 10:44:30,110: INFO/job_scheduler] Job 64 finished 1 steps successfully
[2017-05-10 10:44:30,127: INFO/job_scheduler] Job 64: complete_job: Updating cache
[2017-05-10 10:44:30,127: INFO/job_scheduler] Job 64: locked_item 140516567116176 <class 'chroma_core.models.host.ManagedHost'> True None
[2017-05-10 10:44:30,127: INFO/job_scheduler] Job 64: locked_item 140516297614800 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 10:44:30,130: INFO/job_scheduler] Job 64: locked_item 140516567160272 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 10:44:30,134: INFO/job_scheduler] Job 64: locked_item 140516567162320 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-10 10:44:30,137: INFO/job_scheduler] Job 64: locked_item 140516567159056 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-10 10:44:30,140: INFO/job_scheduler] Job 64: locked_item 140516567160080 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 10:44:30,144: INFO/job_scheduler] Job 64 completing (errored=False, cancelled=False)
[2017-05-10 10:44:30,301: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:44:36,642: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:44:36,656: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 10:44:36,683: INFO/command_plan] add_jobs: created Job 65 (Force remove host lotus-32vm8.lotus.hpdd.lab.intel.com from configuration)
[2017-05-10 10:44:36,719: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 10:44:36,737: INFO/job_scheduler] Job 65: RunJobThread.run
[2017-05-10 10:44:36,789: INFO/job] Deleting dependent lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration for host lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:44:36,789: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration started->removed (intentional=False) 140516567139728
[2017-05-10 10:44:36,809: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-10 10:44:36,831: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> 3
[2017-05-10 10:44:36,985: INFO/job] Deleting dependent lotus-32vm8.lotus.hpdd.lab.intel.com RSyslog configuration for host lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:44:36,985: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com RSyslog configuration configured->removed (intentional=False) 140516567142352
[2017-05-10 10:44:36,991: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> 3
[2017-05-10 10:44:37,009: INFO/job] Deleting dependent lotus-32vm8.lotus.hpdd.lab.intel.com NTP configuration for host lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:44:37,009: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com NTP configuration configured->removed (intentional=False) 140516567138384
[2017-05-10 10:44:37,015: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.ntp.NTPConfiguration'> 3
[2017-05-10 10:44:37,033: INFO/job] Deleting dependent lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration for host lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:44:37,033: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration lnet_up->removed (intentional=False) 140516567139792
[2017-05-10 10:44:37,052: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 10:44:37,069: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> 3
[2017-05-10 10:44:37,097: INFO/job] Deleting dependent lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration for host lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:44:37,097: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration started->removed (intentional=False) 140516566681808
[2017-05-10 10:44:37,117: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-10 10:44:37,144: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.corosync2.Corosync2Configuration'> 3
[2017-05-10 10:44:37,689: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.host.ManagedHost'> 3
[2017-05-10 10:44:37,700: INFO/job_scheduler] Job 65 finished 1 steps successfully
[2017-05-10 10:44:37,761: INFO/job_scheduler] Job 65: complete_job: Updating cache
[2017-05-10 10:44:37,761: INFO/job_scheduler] Job 65: locked_item 140516566667088 <class 'chroma_core.models.host.ManagedHost'> True None
[2017-05-10 10:44:37,761: INFO/job_scheduler] Job 65: locked_item 140516499262160 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 10:44:37,768: INFO/job_scheduler] Job 65: locked_item 140516297614800 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-10 10:44:37,773: INFO/job_scheduler] Job 65: locked_item 140516566845840 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-10 10:44:37,779: INFO/job_scheduler] Job 65: locked_item 140516566506448 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 10:44:37,785: INFO/job_scheduler] Job 65: locked_item 140516566809744 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 10:44:37,790: INFO/job_scheduler] Job 65 completing (errored=False, cancelled=False)
[2017-05-10 10:44:37,850: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:44:47,652: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:44:47,652: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 10:44:47,666: INFO/command_plan] add_jobs: created Job 66 (Test for host connectivity)
[2017-05-10 10:44:47,751: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:44:47,751: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 10:44:47,765: INFO/command_plan] add_jobs: created Job 67 (Test for host connectivity)
[2017-05-10 10:44:47,775: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-05-10 10:44:47,790: INFO/job_scheduler] Job 66: RunJobThread.run
[2017-05-10 10:44:47,790: INFO/job_scheduler] Job 67: RunJobThread.run
[2017-05-10 10:44:47,804: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-05-10 10:44:47,891: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:44:47,892: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 10:44:47,922: INFO/command_plan] add_jobs: created Job 68 (Test for host connectivity)
[2017-05-10 10:44:47,974: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 10:44:48,008: INFO/job_scheduler] Job 68: RunJobThread.run
[2017-05-10 10:44:48,057: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:44:48,057: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 10:44:48,082: INFO/command_plan] add_jobs: created Job 69 (Test for host connectivity)
[2017-05-10 10:44:48,114: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 10:44:48,148: INFO/job_scheduler] Job 69: RunJobThread.run
[2017-05-10 10:44:51,557: INFO/job_scheduler] Job 66 finished 1 steps successfully
[2017-05-10 10:44:51,581: INFO/job_scheduler] Job 66: complete_job: Updating cache
[2017-05-10 10:44:51,581: INFO/job_scheduler] Job 66 completing (errored=False, cancelled=False)
[2017-05-10 10:44:51,656: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 10:44:51,712: INFO/job_scheduler] Job 68 finished 1 steps successfully
[2017-05-10 10:44:51,749: INFO/job_scheduler] Job 68: complete_job: Updating cache
[2017-05-10 10:44:51,749: INFO/job_scheduler] Job 68 completing (errored=False, cancelled=False)
[2017-05-10 10:44:51,805: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-05-10 10:44:51,933: INFO/job_scheduler] Job 67 finished 1 steps successfully
[2017-05-10 10:44:51,956: INFO/job_scheduler] Job 67: complete_job: Updating cache
[2017-05-10 10:44:51,956: INFO/job_scheduler] Job 67 completing (errored=False, cancelled=False)
[2017-05-10 10:44:51,989: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 10:44:52,162: INFO/job_scheduler] Job 69 finished 1 steps successfully
[2017-05-10 10:44:52,214: INFO/job_scheduler] Job 69: complete_job: Updating cache
[2017-05-10 10:44:52,214: INFO/job_scheduler] Job 69 completing (errored=False, cancelled=False)
[2017-05-10 10:44:52,264: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:44:54,980: INFO/job_scheduler] Getting FQDN for 'lotus-32vm5': lotus-32vm5
lotus-32vm5.lotus.hpdd.lab.intel.com

[2017-05-10 10:44:55,004: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:44:55,006: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-5 transitioning from undeployed to unconfigured
[2017-05-10 10:44:55,028: INFO/command_plan] Created command 30 (Deploying agent to host lotus-32vm5.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-05-10 10:44:55,041: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 10:44:55,102: INFO/job_scheduler] Job 70: RunJobThread.run
[2017-05-10 10:44:55,651: INFO/job_scheduler] Getting FQDN for 'lotus-32vm7.lotus.hpdd.lab.intel.com': lotus-32vm7
lotus-32vm7.lotus.hpdd.lab.intel.com

[2017-05-10 10:44:55,679: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:44:55,686: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-6 transitioning from undeployed to unconfigured
[2017-05-10 10:44:55,711: INFO/command_plan] Created command 31 (Deploying agent to host lotus-32vm7.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-05-10 10:44:55,755: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 10:44:55,820: INFO/job_scheduler] Job 71: RunJobThread.run
[2017-05-10 10:44:56,396: INFO/job_scheduler] Getting FQDN for 'lotus-32vm8.lotus.hpdd.lab.intel.com': lotus-32vm8
lotus-32vm8.lotus.hpdd.lab.intel.com

[2017-05-10 10:44:56,431: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:44:56,433: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-7 transitioning from undeployed to unconfigured
[2017-05-10 10:44:56,459: INFO/command_plan] Created command 32 (Deploying agent to host lotus-32vm8.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-05-10 10:44:56,487: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 10:44:56,549: INFO/job_scheduler] Job 72: RunJobThread.run
[2017-05-10 10:44:56,588: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 10:44:56,749: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 10:44:57,217: INFO/job_scheduler] Getting FQDN for 'lotus-32vm6.lotus.hpdd.lab.intel.com': lotus-32vm6
lotus-32vm6.lotus.hpdd.lab.intel.com

[2017-05-10 10:44:57,247: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:44:57,255: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-8 transitioning from undeployed to unconfigured
[2017-05-10 10:44:57,274: INFO/command_plan] Created command 33 (Deploying agent to host lotus-32vm6.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-05-10 10:44:57,412: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 10:44:57,450: INFO/job_scheduler] Job 73: RunJobThread.run
[2017-05-10 10:44:57,452: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bf10e52e-a467-4959-9a54-9a34f5bbf877
[2017-05-10 10:44:57,452: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bf10e52e-a467-4959-9a54-9a34f5bbf877
[2017-05-10 10:44:57,467: INFO/job_scheduler] Job 70 finished 1 steps successfully
[2017-05-10 10:44:57,471: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:44:57,471: INFO/job_scheduler]   Job 70 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-10 10:44:57,471: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 10:44:57,501: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 10:44:57,548: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 140516567162192
[2017-05-10 10:44:57,554: INFO/job] Job 70: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:44:57,554: INFO/job] Job 70: not_deleted=True
[2017-05-10 10:44:57,554: INFO/job_scheduler] Job 70: complete_job: Updating cache
[2017-05-10 10:44:57,554: INFO/job_scheduler] Job 70: locked_item 140516567162192 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 10:44:57,570: INFO/job_scheduler] Job 70 completing (errored=False, cancelled=False)
[2017-05-10 10:44:57,634: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-5
[2017-05-10 10:44:57,635: INFO/job_scheduler] _notify: Updating .boot_time of item 5 (lotus-32vm5.lotus.hpdd.lab.intel.com) from None to 2017-05-03 16:42:07+00:00
[2017-05-10 10:44:57,652: INFO/job_scheduler] _notify: Set boot_time=2017-05-03 16:42:07+00:00 on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-5) and saved
[2017-05-10 10:44:57,673: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 10:44:58,042: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-10 10:44:58,042: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm7.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-10 10:44:58,180: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:44:58,180: INFO/job_scheduler]   Job 71 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-10 10:44:58,181: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 10:44:58,195: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/2b8aa948-d199-4d2e-b4f1-63c55dfaffa0
[2017-05-10 10:44:58,195: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/2b8aa948-d199-4d2e-b4f1-63c55dfaffa0
[2017-05-10 10:44:58,470: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 10:44:59,043: INFO/job_scheduler] Job 71 finished 1 steps successfully
[2017-05-10 10:44:59,169: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 140516566812432
[2017-05-10 10:44:59,175: INFO/job] Job 71: StateChangeJob complete, setting state unconfigured on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:44:59,175: INFO/job] Job 71: not_deleted=True
[2017-05-10 10:44:59,175: INFO/job_scheduler] Job 71: complete_job: Updating cache
[2017-05-10 10:44:59,175: INFO/job_scheduler] Job 71: locked_item 140516566812432 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 10:44:59,181: INFO/job_scheduler] Job 71 completing (errored=False, cancelled=False)
[2017-05-10 10:44:59,270: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-6
[2017-05-10 10:44:59,270: INFO/job_scheduler] _notify: Updating .boot_time of item 6 (lotus-32vm7.lotus.hpdd.lab.intel.com) from None to 2017-04-25 07:09:31+00:00
[2017-05-10 10:44:59,286: INFO/job_scheduler] _notify: Set boot_time=2017-04-25 07:09:31+00:00 on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-6) and saved
[2017-05-10 10:44:59,301: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-10 10:44:59,301: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-10 10:44:59,312: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-05-10 10:44:59,371: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/f4d2c53b-c235-4776-b5f0-c63787788b4c
[2017-05-10 10:44:59,371: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/f4d2c53b-c235-4776-b5f0-c63787788b4c
[2017-05-10 10:44:59,382: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:44:59,383: INFO/job_scheduler]   Job 72 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-10 10:44:59,383: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-05-10 10:44:59,930: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:44:59,930: INFO/job_scheduler]   Job 73 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-10 10:44:59,930: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-05-10 10:44:59,955: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-10 10:44:59,955: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-10 10:44:59,969: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2db754e7-e3ad-4f28-bcf7-935c9d33cfd4
[2017-05-10 10:44:59,969: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2db754e7-e3ad-4f28-bcf7-935c9d33cfd4
[2017-05-10 10:45:00,303: INFO/job_scheduler] Job 72 finished 1 steps successfully
[2017-05-10 10:45:00,330: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 140516567184848
[2017-05-10 10:45:00,337: INFO/job] Job 72: StateChangeJob complete, setting state unconfigured on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:00,337: INFO/job] Job 72: not_deleted=True
[2017-05-10 10:45:00,338: INFO/job_scheduler] Job 72: complete_job: Updating cache
[2017-05-10 10:45:00,338: INFO/job_scheduler] Job 72: locked_item 140516567184848 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 10:45:00,341: INFO/job_scheduler] Job 72 completing (errored=False, cancelled=False)
[2017-05-10 10:45:00,460: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-7
[2017-05-10 10:45:00,461: INFO/job_scheduler] _notify: Updating .boot_time of item 7 (lotus-32vm8.lotus.hpdd.lab.intel.com) from None to 2017-04-25 07:09:56+00:00
[2017-05-10 10:45:00,470: INFO/job_scheduler] _notify: Set boot_time=2017-04-25 07:09:56+00:00 on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-7) and saved
[2017-05-10 10:45:00,488: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 10:45:00,956: INFO/job_scheduler] Job 73 finished 1 steps successfully
[2017-05-10 10:45:01,014: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 140516567151248
[2017-05-10 10:45:01,021: INFO/job] Job 73: StateChangeJob complete, setting state unconfigured on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:01,021: INFO/job] Job 73: not_deleted=True
[2017-05-10 10:45:01,021: INFO/job_scheduler] Job 73: complete_job: Updating cache
[2017-05-10 10:45:01,021: INFO/job_scheduler] Job 73: locked_item 140516567151248 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 10:45:01,025: INFO/job_scheduler] Job 73 completing (errored=False, cancelled=False)
[2017-05-10 10:45:01,119: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-8
[2017-05-10 10:45:01,119: INFO/job_scheduler] _notify: Updating .boot_time of item 8 (lotus-32vm6.lotus.hpdd.lab.intel.com) from None to 2017-04-25 07:09:19+00:00
[2017-05-10 10:45:01,155: INFO/job_scheduler] _notify: Set boot_time=2017-04-25 07:09:19+00:00 on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-8) and saved
[2017-05-10 10:45:01,170: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:45:06,978: INFO/job_scheduler] _notify: Updating .properties of item 5 (lotus-32vm5.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-05-10 10:45:06,986: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-5) and saved
[2017-05-10 10:45:06,992: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:45:07,168: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:45:07,665: INFO/job_scheduler] _notify: Updating .properties of item 6 (lotus-32vm7.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-05-10 10:45:07,672: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-6) and saved
[2017-05-10 10:45:07,678: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:45:07,808: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:45:08,453: INFO/job_scheduler] _notify: Updating .properties of item 8 (lotus-32vm6.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-05-10 10:45:08,461: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-8) and saved
[2017-05-10 10:45:08,468: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:45:08,636: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:45:09,823: INFO/job_scheduler] _notify: Updating .properties of item 7 (lotus-32vm8.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-05-10 10:45:09,828: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-7) and saved
[2017-05-10 10:45:09,834: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:45:09,961: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:45:12,069: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:45:12,078: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-5 transitioning from unconfigured to managed
[2017-05-10 10:45:12,337: INFO/command_plan] Created command 34 (Setup managed host lotus-32vm5.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-05-10 10:45:12,499: INFO/job_scheduler] run_next: 1 runnable jobs of (11 pending, 0 tasked)
[2017-05-10 10:45:12,562: INFO/job_scheduler] Job 74: RunJobThread.run
[2017-05-10 10:45:12,563: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-05-10 10:45:12,602: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cbdf9332-5746-4d50-9db5-dd765f482157
[2017-05-10 10:45:12,602: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cbdf9332-5746-4d50-9db5-dd765f482157
[2017-05-10 10:45:12,708: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:45:12,717: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-6 transitioning from unconfigured to managed
[2017-05-10 10:45:12,979: INFO/command_plan] Created command 35 (Setup managed host lotus-32vm7.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-05-10 10:45:13,094: INFO/job_scheduler] run_next: 1 runnable jobs of (21 pending, 1 tasked)
[2017-05-10 10:45:13,116: INFO/job_scheduler] Job 85: RunJobThread.run
[2017-05-10 10:45:13,116: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-05-10 10:45:13,129: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 70dee31f-158b-41c6-80f7-b37555abceab
[2017-05-10 10:45:13,129: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 70dee31f-158b-41c6-80f7-b37555abceab
[2017-05-10 10:45:13,263: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:45:13,266: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-7 transitioning from unconfigured to managed
[2017-05-10 10:45:13,422: INFO/command_plan] Created command 36 (Setup managed host lotus-32vm8.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-05-10 10:45:13,507: INFO/job_scheduler] run_next: 1 runnable jobs of (31 pending, 2 tasked)
[2017-05-10 10:45:13,578: INFO/job_scheduler] Job 96: RunJobThread.run
[2017-05-10 10:45:13,578: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-05-10 10:45:13,603: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e33710ad-38b8-4929-98d6-0a4555f762b8
[2017-05-10 10:45:13,603: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e33710ad-38b8-4929-98d6-0a4555f762b8
[2017-05-10 10:45:13,702: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:45:13,706: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bf10e52e-a467-4959-9a54-9a34f5bbf877
[2017-05-10 10:45:13,706: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bf10e52e-a467-4959-9a54-9a34f5bbf877
[2017-05-10 10:45:13,706: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bf10e52e-a467-4959-9a54-9a34f5bbf877
[2017-05-10 10:45:13,706: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bf10e52e-a467-4959-9a54-9a34f5bbf877
[2017-05-10 10:45:13,706: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cbdf9332-5746-4d50-9db5-dd765f482157
[2017-05-10 10:45:13,706: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cbdf9332-5746-4d50-9db5-dd765f482157
[2017-05-10 10:45:13,714: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-8 transitioning from unconfigured to managed
[2017-05-10 10:45:13,726: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cbdf9332-5746-4d50-9db5-dd765f482157
[2017-05-10 10:45:13,726: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cbdf9332-5746-4d50-9db5-dd765f482157
[2017-05-10 10:45:13,726: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-10 10:45:13.726465+00:00:
[2017-05-10 10:45:13,726: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-10 10:45:13.726465+00:00:
[2017-05-10 10:45:13,727: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 10:45:13,727: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 10:45:13,775: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-10 10:45:13.775072+00:00:
[2017-05-10 10:45:13,775: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-10 10:45:13.775072+00:00:
[2017-05-10 10:45:13,776: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 10:45:13,776: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 10:45:13,821: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com restart_agent {}
[2017-05-10 10:45:13,842: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3cc4dfce-ca4d-4fe7-a53b-5ac586069931
[2017-05-10 10:45:13,842: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3cc4dfce-ca4d-4fe7-a53b-5ac586069931
[2017-05-10 10:45:13,886: INFO/command_plan] Created command 37 (Setup managed host lotus-32vm6.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-05-10 10:45:13,902: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:45:13,902: INFO/job_scheduler]   Job 74 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 10:45:13,902: INFO/job_scheduler]   Job 75 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:13,902: INFO/job_scheduler]   Job 76 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:13,903: INFO/job_scheduler]   Job 77 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:13,903: INFO/job_scheduler]   Job 78 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:13,903: INFO/job_scheduler]   Job 82 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:13,903: INFO/job_scheduler]   Job 84 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:45:13,903: INFO/job_scheduler] run_next: 1 runnable jobs of (41 pending, 3 tasked)
[2017-05-10 10:45:13,934: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bf10e52e-a467-4959-9a54-9a34f5bbf877
[2017-05-10 10:45:13,934: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bf10e52e-a467-4959-9a54-9a34f5bbf877
[2017-05-10 10:45:13,934: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bf10e52e-a467-4959-9a54-9a34f5bbf877
[2017-05-10 10:45:13,934: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bf10e52e-a467-4959-9a54-9a34f5bbf877
[2017-05-10 10:45:13,935: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3cc4dfce-ca4d-4fe7-a53b-5ac586069931
[2017-05-10 10:45:13,935: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3cc4dfce-ca4d-4fe7-a53b-5ac586069931
[2017-05-10 10:45:13,964: INFO/job_scheduler] Job 107: RunJobThread.run
[2017-05-10 10:45:13,964: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-05-10 10:45:13,975: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3cc4dfce-ca4d-4fe7-a53b-5ac586069931
[2017-05-10 10:45:13,975: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3cc4dfce-ca4d-4fe7-a53b-5ac586069931
[2017-05-10 10:45:13,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old bf10e52e-a467-4959-9a54-9a34f5bbf877)
[2017-05-10 10:45:13,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old bf10e52e-a467-4959-9a54-9a34f5bbf877)
[2017-05-10 10:45:14,030: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2652c91c-a9bb-4371-a036-2d8994347791
[2017-05-10 10:45:14,030: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2652c91c-a9bb-4371-a036-2d8994347791
[2017-05-10 10:45:14,052: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 10:45:14,055: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:45:14,055: INFO/job_scheduler]   Job 74 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 10:45:14,055: INFO/job_scheduler]   Job 75 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:14,055: INFO/job_scheduler]   Job 76 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:14,055: INFO/job_scheduler]   Job 77 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:14,056: INFO/job_scheduler]   Job 78 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:14,056: INFO/job_scheduler]   Job 82 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:14,056: INFO/job_scheduler]   Job 84 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:45:14,056: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 10:45:14,219: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/2b8aa948-d199-4d2e-b4f1-63c55dfaffa0
[2017-05-10 10:45:14,219: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/2b8aa948-d199-4d2e-b4f1-63c55dfaffa0
[2017-05-10 10:45:14,219: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/2b8aa948-d199-4d2e-b4f1-63c55dfaffa0
[2017-05-10 10:45:14,219: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/2b8aa948-d199-4d2e-b4f1-63c55dfaffa0
[2017-05-10 10:45:14,219: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 70dee31f-158b-41c6-80f7-b37555abceab
[2017-05-10 10:45:14,219: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 70dee31f-158b-41c6-80f7-b37555abceab
[2017-05-10 10:45:14,265: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 70dee31f-158b-41c6-80f7-b37555abceab
[2017-05-10 10:45:14,265: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 70dee31f-158b-41c6-80f7-b37555abceab
[2017-05-10 10:45:14,265: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-05-10 10:45:14.265464+00:00:
[2017-05-10 10:45:14,265: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-05-10 10:45:14.265464+00:00:
[2017-05-10 10:45:14,265: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 10:45:14,265: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 10:45:14,282: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:45:14,282: INFO/job_scheduler]   Job 85 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 10:45:14,282: INFO/job_scheduler]   Job 86 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:14,283: INFO/job_scheduler]   Job 87 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:14,283: INFO/job_scheduler]   Job 88 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:14,283: INFO/job_scheduler]   Job 89 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:14,283: INFO/job_scheduler]   Job 93 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:14,283: INFO/job_scheduler]   Job 95 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:45:14,283: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 10:45:14,285: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-05-10 10:45:14.285307+00:00:
[2017-05-10 10:45:14,285: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-05-10 10:45:14.285307+00:00:
[2017-05-10 10:45:14,285: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 10:45:14,285: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 10:45:14,313: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:45:14,313: INFO/job_scheduler]   Job 85 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 10:45:14,313: INFO/job_scheduler]   Job 86 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:14,313: INFO/job_scheduler]   Job 87 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:14,314: INFO/job_scheduler]   Job 88 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:14,314: INFO/job_scheduler]   Job 89 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:14,314: INFO/job_scheduler]   Job 93 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:14,314: INFO/job_scheduler]   Job 95 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:45:14,314: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 10:45:14,315: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com restart_agent {}
[2017-05-10 10:45:14,338: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 57aaf0a6-7264-4b8a-8ea4-eff66cc8ce48
[2017-05-10 10:45:14,338: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 57aaf0a6-7264-4b8a-8ea4-eff66cc8ce48
[2017-05-10 10:45:14,398: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/2b8aa948-d199-4d2e-b4f1-63c55dfaffa0
[2017-05-10 10:45:14,398: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/2b8aa948-d199-4d2e-b4f1-63c55dfaffa0
[2017-05-10 10:45:14,398: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/2b8aa948-d199-4d2e-b4f1-63c55dfaffa0
[2017-05-10 10:45:14,398: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/2b8aa948-d199-4d2e-b4f1-63c55dfaffa0
[2017-05-10 10:45:14,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 57aaf0a6-7264-4b8a-8ea4-eff66cc8ce48
[2017-05-10 10:45:14,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 57aaf0a6-7264-4b8a-8ea4-eff66cc8ce48
[2017-05-10 10:45:14,407: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 57aaf0a6-7264-4b8a-8ea4-eff66cc8ce48
[2017-05-10 10:45:14,407: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 57aaf0a6-7264-4b8a-8ea4-eff66cc8ce48
[2017-05-10 10:45:14,407: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm7.lotus.hpdd.lab.intel.com (old 2b8aa948-d199-4d2e-b4f1-63c55dfaffa0)
[2017-05-10 10:45:14,407: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm7.lotus.hpdd.lab.intel.com (old 2b8aa948-d199-4d2e-b4f1-63c55dfaffa0)
[2017-05-10 10:45:14,681: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/f4d2c53b-c235-4776-b5f0-c63787788b4c
[2017-05-10 10:45:14,681: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/f4d2c53b-c235-4776-b5f0-c63787788b4c
[2017-05-10 10:45:14,682: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/f4d2c53b-c235-4776-b5f0-c63787788b4c
[2017-05-10 10:45:14,682: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/f4d2c53b-c235-4776-b5f0-c63787788b4c
[2017-05-10 10:45:14,682: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e33710ad-38b8-4929-98d6-0a4555f762b8
[2017-05-10 10:45:14,682: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e33710ad-38b8-4929-98d6-0a4555f762b8
[2017-05-10 10:45:14,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e33710ad-38b8-4929-98d6-0a4555f762b8
[2017-05-10 10:45:14,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e33710ad-38b8-4929-98d6-0a4555f762b8
[2017-05-10 10:45:14,720: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-05-10 10:45:14.720095+00:00:
[2017-05-10 10:45:14,720: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-05-10 10:45:14.720095+00:00:
[2017-05-10 10:45:14,720: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 10:45:14,720: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 10:45:14,736: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:45:14,736: INFO/job_scheduler]   Job 96 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 10:45:14,736: INFO/job_scheduler]   Job 97 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:14,736: INFO/job_scheduler]   Job 98 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:14,736: INFO/job_scheduler]   Job 99 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:14,736: INFO/job_scheduler]   Job 100 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:14,736: INFO/job_scheduler]   Job 104 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:14,736: INFO/job_scheduler]   Job 106 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:45:14,737: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 10:45:14,738: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-05-10 10:45:14.738344+00:00:
[2017-05-10 10:45:14,738: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-05-10 10:45:14.738344+00:00:
[2017-05-10 10:45:14,738: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 10:45:14,738: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 10:45:14,750: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:45:14,750: INFO/job_scheduler]   Job 96 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 10:45:14,750: INFO/job_scheduler]   Job 97 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:14,750: INFO/job_scheduler]   Job 98 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:14,750: INFO/job_scheduler]   Job 99 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:14,751: INFO/job_scheduler]   Job 100 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:14,751: INFO/job_scheduler]   Job 104 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:14,751: INFO/job_scheduler]   Job 106 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:45:14,751: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 10:45:14,752: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com restart_agent {}
[2017-05-10 10:45:14,764: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d04ba3f7-b76b-4ef6-b5ea-4619b1f12296
[2017-05-10 10:45:14,764: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d04ba3f7-b76b-4ef6-b5ea-4619b1f12296
[2017-05-10 10:45:14,853: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/f4d2c53b-c235-4776-b5f0-c63787788b4c
[2017-05-10 10:45:14,853: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/f4d2c53b-c235-4776-b5f0-c63787788b4c
[2017-05-10 10:45:14,853: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/f4d2c53b-c235-4776-b5f0-c63787788b4c
[2017-05-10 10:45:14,853: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/f4d2c53b-c235-4776-b5f0-c63787788b4c
[2017-05-10 10:45:14,854: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d04ba3f7-b76b-4ef6-b5ea-4619b1f12296
[2017-05-10 10:45:14,854: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d04ba3f7-b76b-4ef6-b5ea-4619b1f12296
[2017-05-10 10:45:14,879: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d04ba3f7-b76b-4ef6-b5ea-4619b1f12296
[2017-05-10 10:45:14,879: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d04ba3f7-b76b-4ef6-b5ea-4619b1f12296
[2017-05-10 10:45:14,879: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm8.lotus.hpdd.lab.intel.com (old f4d2c53b-c235-4776-b5f0-c63787788b4c)
[2017-05-10 10:45:14,879: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm8.lotus.hpdd.lab.intel.com (old f4d2c53b-c235-4776-b5f0-c63787788b4c)
[2017-05-10 10:45:15,127: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2db754e7-e3ad-4f28-bcf7-935c9d33cfd4
[2017-05-10 10:45:15,127: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2db754e7-e3ad-4f28-bcf7-935c9d33cfd4
[2017-05-10 10:45:15,127: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2db754e7-e3ad-4f28-bcf7-935c9d33cfd4
[2017-05-10 10:45:15,127: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2db754e7-e3ad-4f28-bcf7-935c9d33cfd4
[2017-05-10 10:45:15,128: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2652c91c-a9bb-4371-a036-2d8994347791
[2017-05-10 10:45:15,128: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2652c91c-a9bb-4371-a036-2d8994347791
[2017-05-10 10:45:15,144: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2652c91c-a9bb-4371-a036-2d8994347791
[2017-05-10 10:45:15,144: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2652c91c-a9bb-4371-a036-2d8994347791
[2017-05-10 10:45:15,145: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-05-10 10:45:15.145390+00:00:
[2017-05-10 10:45:15,145: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-05-10 10:45:15.145390+00:00:
[2017-05-10 10:45:15,145: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 10:45:15,145: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 10:45:15,161: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:45:15,161: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-05-10 10:45:15.161305+00:00:
[2017-05-10 10:45:15,161: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-05-10 10:45:15.161305+00:00:
[2017-05-10 10:45:15,161: INFO/job_scheduler]   Job 107 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 10:45:15,162: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 10:45:15,162: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 10:45:15,162: INFO/job_scheduler]   Job 108 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:15,162: INFO/job_scheduler]   Job 109 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:15,163: INFO/job_scheduler]   Job 110 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:15,175: INFO/job_scheduler]   Job 111 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:15,176: INFO/job_scheduler]   Job 114 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:15,176: INFO/job_scheduler]   Job 117 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:45:15,177: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 10:45:15,189: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:45:15,189: INFO/job_scheduler]   Job 107 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 10:45:15,189: INFO/job_scheduler]   Job 108 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:15,189: INFO/job_scheduler]   Job 109 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:15,189: INFO/job_scheduler]   Job 110 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:15,189: INFO/job_scheduler]   Job 111 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:15,189: INFO/job_scheduler]   Job 114 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:15,189: INFO/job_scheduler]   Job 117 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:45:15,190: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 10:45:15,191: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com restart_agent {}
[2017-05-10 10:45:15,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d4b0bd67-70c2-4f1a-b132-3de6887dea84
[2017-05-10 10:45:15,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d4b0bd67-70c2-4f1a-b132-3de6887dea84
[2017-05-10 10:45:15,302: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2db754e7-e3ad-4f28-bcf7-935c9d33cfd4
[2017-05-10 10:45:15,302: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2db754e7-e3ad-4f28-bcf7-935c9d33cfd4
[2017-05-10 10:45:15,302: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2db754e7-e3ad-4f28-bcf7-935c9d33cfd4
[2017-05-10 10:45:15,302: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/2db754e7-e3ad-4f28-bcf7-935c9d33cfd4
[2017-05-10 10:45:15,302: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d4b0bd67-70c2-4f1a-b132-3de6887dea84
[2017-05-10 10:45:15,302: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d4b0bd67-70c2-4f1a-b132-3de6887dea84
[2017-05-10 10:45:15,326: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d4b0bd67-70c2-4f1a-b132-3de6887dea84
[2017-05-10 10:45:15,326: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d4b0bd67-70c2-4f1a-b132-3de6887dea84
[2017-05-10 10:45:15,326: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm6.lotus.hpdd.lab.intel.com (old 2db754e7-e3ad-4f28-bcf7-935c9d33cfd4)
[2017-05-10 10:45:15,326: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm6.lotus.hpdd.lab.intel.com (old 2db754e7-e3ad-4f28-bcf7-935c9d33cfd4)
[2017-05-10 10:45:43,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/2b8aa948-d199-4d2e-b4f1-63c55dfaffa0
[2017-05-10 10:45:43,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/2b8aa948-d199-4d2e-b4f1-63c55dfaffa0
[2017-05-10 10:45:43,271: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 2b8aa948-d199-4d2e-b4f1-63c55dfaffa0 because session terminated
[2017-05-10 10:45:43,271: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 2b8aa948-d199-4d2e-b4f1-63c55dfaffa0 because session terminated
[2017-05-10 10:45:43,274: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:45:43,274: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:45:43,441: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm7.lotus.hpdd.lab.intel.com new f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:45:43,441: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm7.lotus.hpdd.lab.intel.com new f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:45:43,442: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 10:45:43,455: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc abdf8b8d-c1b2-411e-822c-e9e3e2475ed2
[2017-05-10 10:45:43,455: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc abdf8b8d-c1b2-411e-822c-e9e3e2475ed2
[2017-05-10 10:45:43,560: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:45:43,560: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:45:43,560: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:45:43,560: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:45:43,560: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc abdf8b8d-c1b2-411e-822c-e9e3e2475ed2
[2017-05-10 10:45:43,560: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc abdf8b8d-c1b2-411e-822c-e9e3e2475ed2
[2017-05-10 10:45:43,570: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc abdf8b8d-c1b2-411e-822c-e9e3e2475ed2
[2017-05-10 10:45:43,570: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc abdf8b8d-c1b2-411e-822c-e9e3e2475ed2
[2017-05-10 10:45:43,571: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-05-10 10:45:43,585: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0a8ab2c4-8074-4e22-8655-c9cfa6eab3f9
[2017-05-10 10:45:43,585: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0a8ab2c4-8074-4e22-8655-c9cfa6eab3f9
[2017-05-10 10:45:43,700: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/f4d2c53b-c235-4776-b5f0-c63787788b4c
[2017-05-10 10:45:43,700: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/f4d2c53b-c235-4776-b5f0-c63787788b4c
[2017-05-10 10:45:43,700: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session f4d2c53b-c235-4776-b5f0-c63787788b4c because session terminated
[2017-05-10 10:45:43,700: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session f4d2c53b-c235-4776-b5f0-c63787788b4c because session terminated
[2017-05-10 10:45:43,703: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:45:43,703: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:45:43,839: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bf10e52e-a467-4959-9a54-9a34f5bbf877
[2017-05-10 10:45:43,839: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bf10e52e-a467-4959-9a54-9a34f5bbf877
[2017-05-10 10:45:43,839: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session bf10e52e-a467-4959-9a54-9a34f5bbf877 because session terminated
[2017-05-10 10:45:43,839: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session bf10e52e-a467-4959-9a54-9a34f5bbf877 because session terminated
[2017-05-10 10:45:43,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:45:43,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:45:43,913: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm8.lotus.hpdd.lab.intel.com new 7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:45:43,913: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm8.lotus.hpdd.lab.intel.com new 7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:45:43,914: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 10:45:43,938: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3cd5e4ef-bd72-417a-8e93-e457f47ba652
[2017-05-10 10:45:43,938: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3cd5e4ef-bd72-417a-8e93-e457f47ba652
[2017-05-10 10:45:44,027: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:45:44,027: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:45:44,028: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:45:44,028: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:45:44,028: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3cd5e4ef-bd72-417a-8e93-e457f47ba652
[2017-05-10 10:45:44,028: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3cd5e4ef-bd72-417a-8e93-e457f47ba652
[2017-05-10 10:45:44,055: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3cd5e4ef-bd72-417a-8e93-e457f47ba652
[2017-05-10 10:45:44,055: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3cd5e4ef-bd72-417a-8e93-e457f47ba652
[2017-05-10 10:45:44,056: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-05-10 10:45:44,076: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0c165d2c-cf80-440e-aa48-b0e48e65687e
[2017-05-10 10:45:44,076: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0c165d2c-cf80-440e-aa48-b0e48e65687e
[2017-05-10 10:45:44,076: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new 6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:45:44,076: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new 6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:45:44,077: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 10:45:44,106: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 75e01704-4140-444d-8f1a-7f6fc3fb637e
[2017-05-10 10:45:44,106: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 75e01704-4140-444d-8f1a-7f6fc3fb637e
[2017-05-10 10:45:45,185: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:45:45,185: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:45:45,185: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:45:45,185: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:45:45,185: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 75e01704-4140-444d-8f1a-7f6fc3fb637e
[2017-05-10 10:45:45,185: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 75e01704-4140-444d-8f1a-7f6fc3fb637e
[2017-05-10 10:45:45,228: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 75e01704-4140-444d-8f1a-7f6fc3fb637e
[2017-05-10 10:45:45,228: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 75e01704-4140-444d-8f1a-7f6fc3fb637e
[2017-05-10 10:45:45,228: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-05-10 10:45:45,241: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 41eec5ec-1309-4d05-ab6e-6472afe9a9a9
[2017-05-10 10:45:45,241: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 41eec5ec-1309-4d05-ab6e-6472afe9a9a9
[2017-05-10 10:45:45,359: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm6.lotus.hpdd.lab.intel.com timeout after 30.0s
[2017-05-10 10:45:45,359: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm6.lotus.hpdd.lab.intel.com timeout after 30.0s
[2017-05-10 10:45:46,106: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2db754e7-e3ad-4f28-bcf7-935c9d33cfd4
[2017-05-10 10:45:46,106: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/2db754e7-e3ad-4f28-bcf7-935c9d33cfd4
[2017-05-10 10:45:46,106: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 2db754e7-e3ad-4f28-bcf7-935c9d33cfd4 because session terminated
[2017-05-10 10:45:46,106: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 2db754e7-e3ad-4f28-bcf7-935c9d33cfd4 because session terminated
[2017-05-10 10:45:46,112: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:45:46,112: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:45:46,361: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm6.lotus.hpdd.lab.intel.com new afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:45:46,361: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm6.lotus.hpdd.lab.intel.com new afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:45:46,362: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 10:45:46,383: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 229d99f8-dca7-4376-b116-617166cbf118
[2017-05-10 10:45:46,383: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 229d99f8-dca7-4376-b116-617166cbf118
[2017-05-10 10:45:46,501: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:45:46,501: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:45:46,501: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:45:46,501: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:45:46,501: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 229d99f8-dca7-4376-b116-617166cbf118
[2017-05-10 10:45:46,501: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 229d99f8-dca7-4376-b116-617166cbf118
[2017-05-10 10:45:46,502: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 229d99f8-dca7-4376-b116-617166cbf118
[2017-05-10 10:45:46,502: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 229d99f8-dca7-4376-b116-617166cbf118
[2017-05-10 10:45:46,502: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-05-10 10:45:46,519: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 952cd58d-1d46-4faf-bb57-b34bff4bb0f6
[2017-05-10 10:45:46,519: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 952cd58d-1d46-4faf-bb57-b34bff4bb0f6
[2017-05-10 10:45:53,626: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:45:53,627: INFO/job_scheduler]   Job 96 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 10:45:53,627: INFO/job_scheduler]   Job 97 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:53,627: INFO/job_scheduler]   Job 98 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:53,627: INFO/job_scheduler]   Job 99 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:53,627: INFO/job_scheduler]   Job 100 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:53,627: INFO/job_scheduler]   Job 104 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:53,627: INFO/job_scheduler]   Job 106 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:45:53,628: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 10:45:54,412: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:45:54,412: INFO/job_scheduler]   Job 85 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 10:45:54,412: INFO/job_scheduler]   Job 86 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:54,412: INFO/job_scheduler]   Job 87 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:54,413: INFO/job_scheduler]   Job 88 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:54,413: INFO/job_scheduler]   Job 89 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:54,413: INFO/job_scheduler]   Job 93 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:54,413: INFO/job_scheduler]   Job 95 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:45:54,413: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 10:45:55,475: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:45:55,475: INFO/job_scheduler]   Job 74 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 10:45:55,475: INFO/job_scheduler]   Job 75 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:55,475: INFO/job_scheduler]   Job 76 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:55,475: INFO/job_scheduler]   Job 77 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:55,475: INFO/job_scheduler]   Job 78 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:55,476: INFO/job_scheduler]   Job 82 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:55,476: INFO/job_scheduler]   Job 84 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:45:55,476: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 10:45:56,623: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:45:56,623: INFO/job_scheduler]   Job 107 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 10:45:56,624: INFO/job_scheduler]   Job 108 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:56,624: INFO/job_scheduler]   Job 109 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:56,624: INFO/job_scheduler]   Job 110 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:56,624: INFO/job_scheduler]   Job 111 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:56,624: INFO/job_scheduler]   Job 114 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:45:56,624: INFO/job_scheduler]   Job 117 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:45:56,625: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 10:46:29,449: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:29,449: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:29,449: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:29,449: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:29,450: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0a8ab2c4-8074-4e22-8655-c9cfa6eab3f9
[2017-05-10 10:46:29,450: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0a8ab2c4-8074-4e22-8655-c9cfa6eab3f9
[2017-05-10 10:46:29,461: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0a8ab2c4-8074-4e22-8655-c9cfa6eab3f9
[2017-05-10 10:46:29,461: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0a8ab2c4-8074-4e22-8655-c9cfa6eab3f9
[2017-05-10 10:46:32,265: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:46:32,265: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:46:32,265: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:46:32,265: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:46:32,265: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 952cd58d-1d46-4faf-bb57-b34bff4bb0f6
[2017-05-10 10:46:32,265: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 952cd58d-1d46-4faf-bb57-b34bff4bb0f6
[2017-05-10 10:46:32,286: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 952cd58d-1d46-4faf-bb57-b34bff4bb0f6
[2017-05-10 10:46:32,286: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 952cd58d-1d46-4faf-bb57-b34bff4bb0f6
[2017-05-10 10:46:32,473: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com install_packages {'repos': [u'lustre', u'e2fsprogs', u'zfs', u'iml-agent'], 'packages': [u'chroma-agent-management', u'kernel-devel-3.10.0-514.10.2.el7_lustre', u'lustre-osd-zfs', u'lustre-osd-ldiskfs', u'lustre-modules', u'lustre', u'zfs']}
[2017-05-10 10:46:32,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc eb9d009f-ff53-455c-bad1-acb9c2011a0c
[2017-05-10 10:46:32,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc eb9d009f-ff53-455c-bad1-acb9c2011a0c
[2017-05-10 10:46:34,951: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com install_packages {'repos': [u'lustre', u'e2fsprogs', u'zfs', u'iml-agent'], 'packages': [u'chroma-agent-management', u'kernel-devel-3.10.0-514.10.2.el7_lustre', u'lustre-osd-zfs', u'lustre-osd-ldiskfs', u'lustre-modules', u'lustre', u'zfs']}
[2017-05-10 10:46:34,968: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4a35b18e-1378-44c8-892c-573665583817
[2017-05-10 10:46:34,968: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4a35b18e-1378-44c8-892c-573665583817
[2017-05-10 10:46:36,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:46:36,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:46:36,550: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:46:36,550: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:46:36,550: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0c165d2c-cf80-440e-aa48-b0e48e65687e
[2017-05-10 10:46:36,550: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0c165d2c-cf80-440e-aa48-b0e48e65687e
[2017-05-10 10:46:36,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0c165d2c-cf80-440e-aa48-b0e48e65687e
[2017-05-10 10:46:36,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0c165d2c-cf80-440e-aa48-b0e48e65687e
[2017-05-10 10:46:36,983: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:46:36,983: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:46:36,983: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:46:36,983: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:46:36,984: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 41eec5ec-1309-4d05-ab6e-6472afe9a9a9
[2017-05-10 10:46:36,984: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 41eec5ec-1309-4d05-ab6e-6472afe9a9a9
[2017-05-10 10:46:37,027: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 41eec5ec-1309-4d05-ab6e-6472afe9a9a9
[2017-05-10 10:46:37,027: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 41eec5ec-1309-4d05-ab6e-6472afe9a9a9
[2017-05-10 10:46:37,735: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com install_packages {'repos': [u'lustre', u'e2fsprogs', u'zfs', u'iml-agent'], 'packages': [u'chroma-agent-management', u'kernel-devel-3.10.0-514.10.2.el7_lustre', u'lustre-osd-zfs', u'lustre-osd-ldiskfs', u'lustre-modules', u'lustre', u'zfs']}
[2017-05-10 10:46:37,752: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e13d33a6-12ab-4365-bf64-bec5c745f875
[2017-05-10 10:46:37,752: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e13d33a6-12ab-4365-bf64-bec5c745f875
[2017-05-10 10:46:38,406: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com install_packages {'repos': [u'lustre', u'e2fsprogs', u'zfs', u'iml-agent'], 'packages': [u'chroma-agent-management', u'kernel-devel-3.10.0-514.10.2.el7_lustre', u'lustre-osd-zfs', u'lustre-osd-ldiskfs', u'lustre-modules', u'lustre', u'zfs']}
[2017-05-10 10:46:38,428: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4aafc5c9-f885-45fb-9540-e812c01668d3
[2017-05-10 10:46:38,428: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4aafc5c9-f885-45fb-9540-e812c01668d3
[2017-05-10 10:46:51,015: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:51,015: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:51,015: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:51,015: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:51,015: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc eb9d009f-ff53-455c-bad1-acb9c2011a0c
[2017-05-10 10:46:51,015: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc eb9d009f-ff53-455c-bad1-acb9c2011a0c
[2017-05-10 10:46:51,060: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc eb9d009f-ff53-455c-bad1-acb9c2011a0c
[2017-05-10 10:46:51,060: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc eb9d009f-ff53-455c-bad1-acb9c2011a0c
[2017-05-10 10:46:51,790: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com kernel_status {}
[2017-05-10 10:46:51,811: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5e284045-93d5-4873-8041-dd6464a28483
[2017-05-10 10:46:51,811: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5e284045-93d5-4873-8041-dd6464a28483
[2017-05-10 10:46:52,492: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:46:52,492: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:46:52,492: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:46:52,492: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:46:52,492: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4a35b18e-1378-44c8-892c-573665583817
[2017-05-10 10:46:52,492: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4a35b18e-1378-44c8-892c-573665583817
[2017-05-10 10:46:52,504: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4a35b18e-1378-44c8-892c-573665583817
[2017-05-10 10:46:52,504: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4a35b18e-1378-44c8-892c-573665583817
[2017-05-10 10:46:53,560: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com kernel_status {}
[2017-05-10 10:46:53,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7b8cdff7-2a9b-43f2-a9ac-874a12cdb656
[2017-05-10 10:46:53,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7b8cdff7-2a9b-43f2-a9ac-874a12cdb656
[2017-05-10 10:46:53,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:53,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:53,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:53,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:53,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5e284045-93d5-4873-8041-dd6464a28483
[2017-05-10 10:46:53,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5e284045-93d5-4873-8041-dd6464a28483
[2017-05-10 10:46:53,984: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5e284045-93d5-4873-8041-dd6464a28483
[2017-05-10 10:46:53,984: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5e284045-93d5-4873-8041-dd6464a28483
[2017-05-10 10:46:53,985: INFO/job_scheduler] Job 85 finished 4 steps successfully
[2017-05-10 10:46:54,069: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 140516567186704
[2017-05-10 10:46:54,074: INFO/job] Job 85: StateChangeJob complete, setting state packages_installed on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:46:54,074: INFO/job] Job 85: not_deleted=True
[2017-05-10 10:46:54,074: INFO/job_scheduler] Job 85: complete_job: Updating cache
[2017-05-10 10:46:54,074: INFO/job_scheduler] Job 85: locked_item 140516567183696 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 10:46:54,077: INFO/job_scheduler] Job 85 completing (errored=False, cancelled=False)
[2017-05-10 10:46:54,101: INFO/job_scheduler] run_next: 4 runnable jobs of (40 pending, 3 tasked)
[2017-05-10 10:46:54,143: INFO/job_scheduler] Job 86: open DB connection during completion
[2017-05-10 10:46:54,167: INFO/job_scheduler] Job 87: RunJobThread.run
[2017-05-10 10:46:54,168: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-05-10 10:46:54,181: INFO/job_scheduler] Job 88: RunJobThread.run
[2017-05-10 10:46:54,183: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc eec0fa19-0eaa-4079-8e56-2ac47c9973f9
[2017-05-10 10:46:54,183: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc eec0fa19-0eaa-4079-8e56-2ac47c9973f9
[2017-05-10 10:46:54,185: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-05-10 10:46:54,197: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3cb77fae-7bc4-436d-937c-68530f707c7e
[2017-05-10 10:46:54,197: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3cb77fae-7bc4-436d-937c-68530f707c7e
[2017-05-10 10:46:54,200: INFO/job_scheduler] Job 89: RunJobThread.run
[2017-05-10 10:46:54,201: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-05-10 10:46:54,210: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 140516567194960
[2017-05-10 10:46:54,229: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 10:46:54,234: INFO/job] Job 86: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 10:46:54,234: INFO/job] Job 86: not_deleted=True
[2017-05-10 10:46:54,234: INFO/job_scheduler] Job 86: complete_job: Updating cache
[2017-05-10 10:46:54,235: INFO/job_scheduler] Job 86: locked_item 140516567194960 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 10:46:54,249: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c86baf6f-6444-4695-9f1a-f84075e37b02
[2017-05-10 10:46:54,249: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c86baf6f-6444-4695-9f1a-f84075e37b02
[2017-05-10 10:46:54,253: INFO/job_scheduler] Job 86 completing (errored=False, cancelled=False)
[2017-05-10 10:46:54,309: INFO/job_scheduler] run_next: 1 runnable jobs of (36 pending, 6 tasked)
[2017-05-10 10:46:54,314: INFO/job_scheduler] Job 90: RunJobThread.run
[2017-05-10 10:46:54,316: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com load_lnet {}
[2017-05-10 10:46:54,336: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 908ebc46-a8fa-445a-a2c3-9bbfde895c15
[2017-05-10 10:46:54,336: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 908ebc46-a8fa-445a-a2c3-9bbfde895c15
[2017-05-10 10:46:54,459: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:54,459: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:54,459: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:54,459: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:54,459: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 908ebc46-a8fa-445a-a2c3-9bbfde895c15
[2017-05-10 10:46:54,459: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 908ebc46-a8fa-445a-a2c3-9bbfde895c15
[2017-05-10 10:46:54,470: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 908ebc46-a8fa-445a-a2c3-9bbfde895c15
[2017-05-10 10:46:54,470: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 908ebc46-a8fa-445a-a2c3-9bbfde895c15
[2017-05-10 10:46:54,471: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 10:46:54,490: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 44dbaee2-9b56-47de-bc64-a579091174c4
[2017-05-10 10:46:54,490: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 44dbaee2-9b56-47de-bc64-a579091174c4
[2017-05-10 10:46:54,593: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:54,593: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:54,594: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:54,594: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:54,594: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 44dbaee2-9b56-47de-bc64-a579091174c4
[2017-05-10 10:46:54,594: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 44dbaee2-9b56-47de-bc64-a579091174c4
[2017-05-10 10:46:54,610: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 44dbaee2-9b56-47de-bc64-a579091174c4
[2017-05-10 10:46:54,610: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 44dbaee2-9b56-47de-bc64-a579091174c4
[2017-05-10 10:46:55,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:46:55,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:46:55,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:46:55,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:46:55,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7b8cdff7-2a9b-43f2-a9ac-874a12cdb656
[2017-05-10 10:46:55,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7b8cdff7-2a9b-43f2-a9ac-874a12cdb656
[2017-05-10 10:46:55,709: INFO/job_scheduler] Job 90 finished 2 steps successfully
[2017-05-10 10:46:55,717: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7b8cdff7-2a9b-43f2-a9ac-874a12cdb656
[2017-05-10 10:46:55,717: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7b8cdff7-2a9b-43f2-a9ac-874a12cdb656
[2017-05-10 10:46:55,718: INFO/job_scheduler] Job 107 finished 4 steps successfully
[2017-05-10 10:46:55,753: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 140516567194960
[2017-05-10 10:46:55,763: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 10:46:55,785: INFO/job] Job 90: StateChangeJob complete, setting state lnet_down on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 10:46:55,785: INFO/job] Job 90: not_deleted=True
[2017-05-10 10:46:55,785: INFO/job_scheduler] Job 90: complete_job: Updating cache
[2017-05-10 10:46:55,785: INFO/job_scheduler] Job 90: locked_item 140516567194960 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 10:46:55,799: INFO/job_scheduler] Job 90 completing (errored=False, cancelled=False)
[2017-05-10 10:46:55,837: INFO/job_scheduler] run_next: 1 runnable jobs of (35 pending, 6 tasked)
[2017-05-10 10:46:55,872: INFO/job_scheduler] Job 92: RunJobThread.run
[2017-05-10 10:46:55,872: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_lnet {}
[2017-05-10 10:46:55,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c0588198-d9d3-47bc-bcd8-168eaee99ee1
[2017-05-10 10:46:55,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c0588198-d9d3-47bc-bcd8-168eaee99ee1
[2017-05-10 10:46:55,939: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 140516567218832
[2017-05-10 10:46:55,943: INFO/job] Job 107: StateChangeJob complete, setting state packages_installed on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:46:55,943: INFO/job] Job 107: not_deleted=True
[2017-05-10 10:46:55,943: INFO/job_scheduler] Job 107: complete_job: Updating cache
[2017-05-10 10:46:55,944: INFO/job_scheduler] Job 107: locked_item 140516566813072 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 10:46:55,958: INFO/job_scheduler] Job 107 completing (errored=False, cancelled=False)
[2017-05-10 10:46:55,994: INFO/job_scheduler] run_next: 4 runnable jobs of (34 pending, 6 tasked)
[2017-05-10 10:46:56,055: INFO/job_scheduler] Job 108: RunJobThread.run
[2017-05-10 10:46:56,056: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-05-10 10:46:56,064: INFO/job_scheduler] Job 109: open DB connection during completion
[2017-05-10 10:46:56,097: INFO/job_scheduler] Job 110: RunJobThread.run
[2017-05-10 10:46:56,098: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-05-10 10:46:56,108: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 92ab9431-3dfe-4098-876e-70e2b22de255
[2017-05-10 10:46:56,108: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 92ab9431-3dfe-4098-876e-70e2b22de255
[2017-05-10 10:46:56,131: INFO/job_scheduler] Job 111: RunJobThread.run
[2017-05-10 10:46:56,138: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-05-10 10:46:56,161: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 140516567194384
[2017-05-10 10:46:56,168: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d6a74bab-6cba-487d-9f4d-19001ed1d95b
[2017-05-10 10:46:56,168: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d6a74bab-6cba-487d-9f4d-19001ed1d95b
[2017-05-10 10:46:56,182: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 10:46:56,195: INFO/job] Job 109: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 10:46:56,195: INFO/job] Job 109: not_deleted=True
[2017-05-10 10:46:56,195: INFO/job_scheduler] Job 109: complete_job: Updating cache
[2017-05-10 10:46:56,195: INFO/job_scheduler] Job 109: locked_item 140516567194384 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 10:46:56,197: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 80578cf1-6923-49bc-b3c7-93489dfafb78
[2017-05-10 10:46:56,197: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 80578cf1-6923-49bc-b3c7-93489dfafb78
[2017-05-10 10:46:56,214: INFO/job_scheduler] Job 109 completing (errored=False, cancelled=False)
[2017-05-10 10:46:56,387: INFO/job_scheduler] run_next: 1 runnable jobs of (30 pending, 9 tasked)
[2017-05-10 10:46:56,424: INFO/job_scheduler] Job 112: RunJobThread.run
[2017-05-10 10:46:56,425: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com load_lnet {}
[2017-05-10 10:46:56,516: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c8df8bb1-39c6-4087-97b8-2aeb18c0d781
[2017-05-10 10:46:56,516: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c8df8bb1-39c6-4087-97b8-2aeb18c0d781
[2017-05-10 10:46:57,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:57,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:57,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:57,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:57,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c0588198-d9d3-47bc-bcd8-168eaee99ee1
[2017-05-10 10:46:57,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c0588198-d9d3-47bc-bcd8-168eaee99ee1
[2017-05-10 10:46:57,026: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c0588198-d9d3-47bc-bcd8-168eaee99ee1
[2017-05-10 10:46:57,026: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c0588198-d9d3-47bc-bcd8-168eaee99ee1
[2017-05-10 10:46:57,027: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 10:46:57,078: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d7abd8bc-9e1d-4a57-a996-576855551c5f
[2017-05-10 10:46:57,078: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d7abd8bc-9e1d-4a57-a996-576855551c5f
[2017-05-10 10:46:57,156: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:57,156: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:57,156: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:57,156: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:57,156: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d7abd8bc-9e1d-4a57-a996-576855551c5f
[2017-05-10 10:46:57,156: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d7abd8bc-9e1d-4a57-a996-576855551c5f
[2017-05-10 10:46:57,205: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d7abd8bc-9e1d-4a57-a996-576855551c5f
[2017-05-10 10:46:57,205: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d7abd8bc-9e1d-4a57-a996-576855551c5f
[2017-05-10 10:46:57,569: INFO/job_scheduler] Job 92 finished 2 steps successfully
[2017-05-10 10:46:57,631: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:46:57,631: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:46:57,632: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:46:57,632: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:46:57,632: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c8df8bb1-39c6-4087-97b8-2aeb18c0d781
[2017-05-10 10:46:57,632: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c8df8bb1-39c6-4087-97b8-2aeb18c0d781
[2017-05-10 10:46:57,642: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c8df8bb1-39c6-4087-97b8-2aeb18c0d781
[2017-05-10 10:46:57,642: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c8df8bb1-39c6-4087-97b8-2aeb18c0d781
[2017-05-10 10:46:57,643: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 10:46:57,679: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c97530c5-e82e-48fa-ab21-dd95f50db38c
[2017-05-10 10:46:57,679: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c97530c5-e82e-48fa-ab21-dd95f50db38c
[2017-05-10 10:46:57,696: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 140516567194960
[2017-05-10 10:46:57,716: INFO/job] Job 92: StateChangeJob complete, setting state lnet_up on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 10:46:57,717: INFO/job] Job 92: not_deleted=True
[2017-05-10 10:46:57,717: INFO/job_scheduler] Job 92: complete_job: Updating cache
[2017-05-10 10:46:57,717: INFO/job_scheduler] Job 92: locked_item 140516567194960 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 10:46:57,730: INFO/job_scheduler] Job 92 completing (errored=False, cancelled=False)
[2017-05-10 10:46:57,798: INFO/job_scheduler] run_next: 0 runnable jobs of (29 pending, 9 tasked)
[2017-05-10 10:46:57,924: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:46:57,924: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:46:57,924: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:46:57,924: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:46:57,924: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4aafc5c9-f885-45fb-9540-e812c01668d3
[2017-05-10 10:46:57,924: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4aafc5c9-f885-45fb-9540-e812c01668d3
[2017-05-10 10:46:57,932: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4aafc5c9-f885-45fb-9540-e812c01668d3
[2017-05-10 10:46:57,932: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4aafc5c9-f885-45fb-9540-e812c01668d3
[2017-05-10 10:46:58,296: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:46:58,296: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:46:58,296: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:46:58,296: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:46:58,296: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e13d33a6-12ab-4365-bf64-bec5c745f875
[2017-05-10 10:46:58,296: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e13d33a6-12ab-4365-bf64-bec5c745f875
[2017-05-10 10:46:58,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e13d33a6-12ab-4365-bf64-bec5c745f875
[2017-05-10 10:46:58,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e13d33a6-12ab-4365-bf64-bec5c745f875
[2017-05-10 10:46:58,787: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:46:58,787: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:46:58,787: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:46:58,787: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:46:58,787: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c97530c5-e82e-48fa-ab21-dd95f50db38c
[2017-05-10 10:46:58,787: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c97530c5-e82e-48fa-ab21-dd95f50db38c
[2017-05-10 10:46:58,802: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c97530c5-e82e-48fa-ab21-dd95f50db38c
[2017-05-10 10:46:58,802: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c97530c5-e82e-48fa-ab21-dd95f50db38c
[2017-05-10 10:46:59,004: INFO/job_scheduler] Job 112 finished 2 steps successfully
[2017-05-10 10:46:59,048: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 140516567194384
[2017-05-10 10:46:59,061: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 10:46:59,077: INFO/job] Job 112: StateChangeJob complete, setting state lnet_down on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 10:46:59,078: INFO/job] Job 112: not_deleted=True
[2017-05-10 10:46:59,083: INFO/job_scheduler] Job 112: complete_job: Updating cache
[2017-05-10 10:46:59,083: INFO/job_scheduler] Job 112: locked_item 140516567194384 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 10:46:59,088: INFO/job_scheduler] Job 112 completing (errored=False, cancelled=False)
[2017-05-10 10:46:59,149: INFO/job_scheduler] run_next: 1 runnable jobs of (29 pending, 8 tasked)
[2017-05-10 10:46:59,189: INFO/job_scheduler] Job 115: RunJobThread.run
[2017-05-10 10:46:59,190: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_lnet {}
[2017-05-10 10:46:59,236: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 66e5d6e1-4dbd-4f56-b2b9-2be188d590c5
[2017-05-10 10:46:59,236: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 66e5d6e1-4dbd-4f56-b2b9-2be188d590c5
[2017-05-10 10:46:59,391: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:46:59,391: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:46:59,391: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:46:59,391: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:46:59,391: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 66e5d6e1-4dbd-4f56-b2b9-2be188d590c5
[2017-05-10 10:46:59,391: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 66e5d6e1-4dbd-4f56-b2b9-2be188d590c5
[2017-05-10 10:46:59,410: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 66e5d6e1-4dbd-4f56-b2b9-2be188d590c5
[2017-05-10 10:46:59,410: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 66e5d6e1-4dbd-4f56-b2b9-2be188d590c5
[2017-05-10 10:46:59,411: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 10:46:59,456: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9265c8fd-6344-4fd9-975f-0b5abaf227f1
[2017-05-10 10:46:59,456: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9265c8fd-6344-4fd9-975f-0b5abaf227f1
[2017-05-10 10:46:59,509: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com kernel_status {}
[2017-05-10 10:46:59,554: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2a68da2c-9063-455c-bfac-5e2c1500c053
[2017-05-10 10:46:59,554: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2a68da2c-9063-455c-bfac-5e2c1500c053
[2017-05-10 10:46:59,673: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:59,673: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:59,673: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:59,673: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:46:59,674: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3cb77fae-7bc4-436d-937c-68530f707c7e
[2017-05-10 10:46:59,674: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3cb77fae-7bc4-436d-937c-68530f707c7e
[2017-05-10 10:46:59,687: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3cb77fae-7bc4-436d-937c-68530f707c7e
[2017-05-10 10:46:59,687: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3cb77fae-7bc4-436d-937c-68530f707c7e
[2017-05-10 10:46:59,687: INFO/job_scheduler] Job 88 finished 1 steps successfully
[2017-05-10 10:46:59,743: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com kernel_status {}
[2017-05-10 10:46:59,772: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 129ba55e-37fa-4f08-b70b-c5497e1155b0
[2017-05-10 10:46:59,772: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 129ba55e-37fa-4f08-b70b-c5497e1155b0
[2017-05-10 10:46:59,822: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 140516566779664
[2017-05-10 10:46:59,830: INFO/job] Job 88: StateChangeJob complete, setting state configured on lotus-32vm7.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-10 10:46:59,830: INFO/job] Job 88: not_deleted=True
[2017-05-10 10:46:59,830: INFO/job_scheduler] Job 88: complete_job: Updating cache
[2017-05-10 10:46:59,830: INFO/job_scheduler] Job 88: locked_item 140516566779664 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-10 10:46:59,835: INFO/job_scheduler] Job 88 completing (errored=False, cancelled=False)
[2017-05-10 10:46:59,886: INFO/job_scheduler] run_next: 0 runnable jobs of (28 pending, 8 tasked)
[2017-05-10 10:47:00,659: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:47:00,659: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:47:00,659: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:47:00,659: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:47:00,660: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9265c8fd-6344-4fd9-975f-0b5abaf227f1
[2017-05-10 10:47:00,660: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9265c8fd-6344-4fd9-975f-0b5abaf227f1
[2017-05-10 10:47:00,685: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9265c8fd-6344-4fd9-975f-0b5abaf227f1
[2017-05-10 10:47:00,685: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9265c8fd-6344-4fd9-975f-0b5abaf227f1
[2017-05-10 10:47:00,783: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:47:00,783: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:47:00,783: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:47:00,783: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:47:00,783: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c86baf6f-6444-4695-9f1a-f84075e37b02
[2017-05-10 10:47:00,783: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c86baf6f-6444-4695-9f1a-f84075e37b02
[2017-05-10 10:47:00,790: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c86baf6f-6444-4695-9f1a-f84075e37b02
[2017-05-10 10:47:00,790: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c86baf6f-6444-4695-9f1a-f84075e37b02
[2017-05-10 10:47:00,791: INFO/job_scheduler] Job 89 finished 1 steps successfully
[2017-05-10 10:47:00,907: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 140516567189904
[2017-05-10 10:47:00,914: INFO/job] Job 89: StateChangeJob complete, setting state configured on lotus-32vm7.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-10 10:47:00,914: INFO/job] Job 89: not_deleted=True
[2017-05-10 10:47:00,914: INFO/job_scheduler] Job 89: complete_job: Updating cache
[2017-05-10 10:47:00,914: INFO/job_scheduler] Job 89: locked_item 140516567189904 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-10 10:47:00,928: INFO/job_scheduler] Job 89 completing (errored=False, cancelled=False)
[2017-05-10 10:47:00,973: INFO/job_scheduler] run_next: 0 runnable jobs of (28 pending, 7 tasked)
[2017-05-10 10:47:01,083: INFO/job_scheduler] Job 115 finished 2 steps successfully
[2017-05-10 10:47:01,115: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 140516567194384
[2017-05-10 10:47:01,132: INFO/job] Job 115: StateChangeJob complete, setting state lnet_up on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 10:47:01,133: INFO/job] Job 115: not_deleted=True
[2017-05-10 10:47:01,133: INFO/job_scheduler] Job 115: complete_job: Updating cache
[2017-05-10 10:47:01,133: INFO/job_scheduler] Job 115: locked_item 140516567194384 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 10:47:01,136: INFO/job_scheduler] Job 115 completing (errored=False, cancelled=False)
[2017-05-10 10:47:01,162: INFO/job_scheduler] run_next: 0 runnable jobs of (28 pending, 6 tasked)
[2017-05-10 10:47:01,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:01,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:01,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:01,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:01,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2a68da2c-9063-455c-bfac-5e2c1500c053
[2017-05-10 10:47:01,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2a68da2c-9063-455c-bfac-5e2c1500c053
[2017-05-10 10:47:01,685: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2a68da2c-9063-455c-bfac-5e2c1500c053
[2017-05-10 10:47:01,685: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2a68da2c-9063-455c-bfac-5e2c1500c053
[2017-05-10 10:47:01,685: INFO/job_scheduler] Job 74 finished 4 steps successfully
[2017-05-10 10:47:01,747: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 140516566628496
[2017-05-10 10:47:01,752: INFO/job] Job 74: StateChangeJob complete, setting state packages_installed on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:47:01,752: INFO/job] Job 74: not_deleted=True
[2017-05-10 10:47:01,752: INFO/job_scheduler] Job 74: complete_job: Updating cache
[2017-05-10 10:47:01,752: INFO/job_scheduler] Job 74: locked_item 140516566536400 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 10:47:01,755: INFO/job_scheduler] Job 74 completing (errored=False, cancelled=False)
[2017-05-10 10:47:01,770: INFO/job_scheduler] run_next: 4 runnable jobs of (28 pending, 5 tasked)
[2017-05-10 10:47:01,787: INFO/job_scheduler] Job 75: RunJobThread.run
[2017-05-10 10:47:01,788: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-05-10 10:47:01,796: INFO/job_scheduler] Job 76: open DB connection during completion
[2017-05-10 10:47:01,812: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ea2606fe-d293-4a74-b10d-88629d8255a9
[2017-05-10 10:47:01,812: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ea2606fe-d293-4a74-b10d-88629d8255a9
[2017-05-10 10:47:01,814: INFO/job_scheduler] Job 77: RunJobThread.run
[2017-05-10 10:47:01,820: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-05-10 10:47:01,858: INFO/job_scheduler] Job 78: RunJobThread.run
[2017-05-10 10:47:01,858: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-05-10 10:47:01,864: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 140516567204688
[2017-05-10 10:47:01,875: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ba913ada-42ea-4513-8c77-e31d0e3ef7e5
[2017-05-10 10:47:01,875: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ba913ada-42ea-4513-8c77-e31d0e3ef7e5
[2017-05-10 10:47:01,896: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 10:47:01,918: INFO/job] Job 76: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 10:47:01,918: INFO/job] Job 76: not_deleted=True
[2017-05-10 10:47:01,918: INFO/job_scheduler] Job 76: complete_job: Updating cache
[2017-05-10 10:47:01,918: INFO/job_scheduler] Job 76: locked_item 140516567204688 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 10:47:01,920: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 561fab06-668e-4f0d-ab6f-18d17d514134
[2017-05-10 10:47:01,920: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 561fab06-668e-4f0d-ab6f-18d17d514134
[2017-05-10 10:47:01,922: INFO/job_scheduler] Job 76 completing (errored=False, cancelled=False)
[2017-05-10 10:47:01,956: INFO/job_scheduler] run_next: 1 runnable jobs of (24 pending, 8 tasked)
[2017-05-10 10:47:01,990: INFO/job_scheduler] Job 80: RunJobThread.run
[2017-05-10 10:47:01,990: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com load_lnet {}
[2017-05-10 10:47:02,031: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ee02ad9e-e0e5-47b8-8736-d8daf4051047
[2017-05-10 10:47:02,031: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ee02ad9e-e0e5-47b8-8736-d8daf4051047
[2017-05-10 10:47:02,464: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:47:02,464: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:47:02,464: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:47:02,464: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:47:02,464: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 80578cf1-6923-49bc-b3c7-93489dfafb78
[2017-05-10 10:47:02,464: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 80578cf1-6923-49bc-b3c7-93489dfafb78
[2017-05-10 10:47:02,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:47:02,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:47:02,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:47:02,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:47:02,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d6a74bab-6cba-487d-9f4d-19001ed1d95b
[2017-05-10 10:47:02,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d6a74bab-6cba-487d-9f4d-19001ed1d95b
[2017-05-10 10:47:02,483: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 80578cf1-6923-49bc-b3c7-93489dfafb78
[2017-05-10 10:47:02,483: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 80578cf1-6923-49bc-b3c7-93489dfafb78
[2017-05-10 10:47:02,483: INFO/job_scheduler] Job 111 finished 1 steps successfully
[2017-05-10 10:47:02,496: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d6a74bab-6cba-487d-9f4d-19001ed1d95b
[2017-05-10 10:47:02,496: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d6a74bab-6cba-487d-9f4d-19001ed1d95b
[2017-05-10 10:47:02,496: INFO/job_scheduler] Job 110 finished 1 steps successfully
[2017-05-10 10:47:02,689: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 140516567195344
[2017-05-10 10:47:02,695: INFO/job] Job 111: StateChangeJob complete, setting state configured on lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-10 10:47:02,695: INFO/job] Job 111: not_deleted=True
[2017-05-10 10:47:02,695: INFO/job_scheduler] Job 111: complete_job: Updating cache
[2017-05-10 10:47:02,695: INFO/job_scheduler] Job 111: locked_item 140516567195344 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-10 10:47:02,697: INFO/job_scheduler] Job 111 completing (errored=False, cancelled=False)
[2017-05-10 10:47:02,748: INFO/job_scheduler] run_next: 0 runnable jobs of (23 pending, 8 tasked)
[2017-05-10 10:47:02,899: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:02,899: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:02,899: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:02,899: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:02,899: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 129ba55e-37fa-4f08-b70b-c5497e1155b0
[2017-05-10 10:47:02,899: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 129ba55e-37fa-4f08-b70b-c5497e1155b0
[2017-05-10 10:47:02,900: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 129ba55e-37fa-4f08-b70b-c5497e1155b0
[2017-05-10 10:47:02,900: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 129ba55e-37fa-4f08-b70b-c5497e1155b0
[2017-05-10 10:47:02,900: INFO/job_scheduler] Job 96 finished 4 steps successfully
[2017-05-10 10:47:02,991: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 140516567137168
[2017-05-10 10:47:02,996: INFO/job] Job 110: StateChangeJob complete, setting state configured on lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-10 10:47:02,996: INFO/job] Job 110: not_deleted=True
[2017-05-10 10:47:02,996: INFO/job_scheduler] Job 110: complete_job: Updating cache
[2017-05-10 10:47:02,996: INFO/job_scheduler] Job 110: locked_item 140516567137168 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-10 10:47:02,999: INFO/job_scheduler] Job 110 completing (errored=False, cancelled=False)
[2017-05-10 10:47:03,040: INFO/job_scheduler] run_next: 0 runnable jobs of (23 pending, 7 tasked)
[2017-05-10 10:47:03,090: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 140516567154640
[2017-05-10 10:47:03,097: INFO/job] Job 96: StateChangeJob complete, setting state packages_installed on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:47:03,097: INFO/job] Job 96: not_deleted=True
[2017-05-10 10:47:03,098: INFO/job_scheduler] Job 96: complete_job: Updating cache
[2017-05-10 10:47:03,098: INFO/job_scheduler] Job 96: locked_item 140516567191888 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 10:47:03,106: INFO/job_scheduler] Job 96 completing (errored=False, cancelled=False)
[2017-05-10 10:47:03,122: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:03,122: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:03,122: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:03,122: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:03,122: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ee02ad9e-e0e5-47b8-8736-d8daf4051047
[2017-05-10 10:47:03,122: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ee02ad9e-e0e5-47b8-8736-d8daf4051047
[2017-05-10 10:47:03,133: INFO/job_scheduler] run_next: 4 runnable jobs of (23 pending, 6 tasked)
[2017-05-10 10:47:03,150: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ee02ad9e-e0e5-47b8-8736-d8daf4051047
[2017-05-10 10:47:03,150: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ee02ad9e-e0e5-47b8-8736-d8daf4051047
[2017-05-10 10:47:03,151: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 10:47:03,171: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a421fd92-803c-4b45-b652-c20fd3eedb77
[2017-05-10 10:47:03,171: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a421fd92-803c-4b45-b652-c20fd3eedb77
[2017-05-10 10:47:03,200: INFO/job_scheduler] Job 97: RunJobThread.run
[2017-05-10 10:47:03,201: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-05-10 10:47:03,227: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 715b5310-6867-47e2-a995-20e75f7172b6
[2017-05-10 10:47:03,227: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 715b5310-6867-47e2-a995-20e75f7172b6
[2017-05-10 10:47:03,233: INFO/job_scheduler] Job 98: RunJobThread.run
[2017-05-10 10:47:03,234: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-05-10 10:47:03,256: INFO/job_scheduler] Job 99: RunJobThread.run
[2017-05-10 10:47:03,257: INFO/job_scheduler] Job 100: open DB connection during completion
[2017-05-10 10:47:03,261: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fd86532b-557a-41b8-8885-34ad23695aa4
[2017-05-10 10:47:03,261: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fd86532b-557a-41b8-8885-34ad23695aa4
[2017-05-10 10:47:03,263: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-05-10 10:47:03,284: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e12ce2b4-6bb1-4414-9a64-ec2b78d69405
[2017-05-10 10:47:03,284: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e12ce2b4-6bb1-4414-9a64-ec2b78d69405
[2017-05-10 10:47:03,411: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 140516567196560
[2017-05-10 10:47:03,422: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 10:47:03,426: INFO/job] Job 100: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 10:47:03,426: INFO/job] Job 100: not_deleted=True
[2017-05-10 10:47:03,426: INFO/job_scheduler] Job 100: complete_job: Updating cache
[2017-05-10 10:47:03,426: INFO/job_scheduler] Job 100: locked_item 140516567196560 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 10:47:03,449: INFO/job_scheduler] Job 100 completing (errored=False, cancelled=False)
[2017-05-10 10:47:03,471: INFO/job_scheduler] run_next: 1 runnable jobs of (19 pending, 9 tasked)
[2017-05-10 10:47:03,478: INFO/job_scheduler] Job 102: RunJobThread.run
[2017-05-10 10:47:03,479: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com load_lnet {}
[2017-05-10 10:47:03,499: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 782c9c23-756a-4bb1-8d22-31d100cf9c25
[2017-05-10 10:47:03,499: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 782c9c23-756a-4bb1-8d22-31d100cf9c25
[2017-05-10 10:47:04,296: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:04,296: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:04,297: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:04,297: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:04,297: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a421fd92-803c-4b45-b652-c20fd3eedb77
[2017-05-10 10:47:04,297: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a421fd92-803c-4b45-b652-c20fd3eedb77
[2017-05-10 10:47:04,297: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a421fd92-803c-4b45-b652-c20fd3eedb77
[2017-05-10 10:47:04,297: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a421fd92-803c-4b45-b652-c20fd3eedb77
[2017-05-10 10:47:04,600: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:04,600: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:04,601: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:04,601: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:04,601: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 782c9c23-756a-4bb1-8d22-31d100cf9c25
[2017-05-10 10:47:04,601: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 782c9c23-756a-4bb1-8d22-31d100cf9c25
[2017-05-10 10:47:04,619: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 782c9c23-756a-4bb1-8d22-31d100cf9c25
[2017-05-10 10:47:04,619: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 782c9c23-756a-4bb1-8d22-31d100cf9c25
[2017-05-10 10:47:04,620: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 10:47:04,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 411110d3-dd45-4a6b-8084-f178eec16b90
[2017-05-10 10:47:04,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 411110d3-dd45-4a6b-8084-f178eec16b90
[2017-05-10 10:47:04,760: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:04,760: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:04,760: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:04,760: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:04,760: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 411110d3-dd45-4a6b-8084-f178eec16b90
[2017-05-10 10:47:04,760: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 411110d3-dd45-4a6b-8084-f178eec16b90
[2017-05-10 10:47:04,761: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 411110d3-dd45-4a6b-8084-f178eec16b90
[2017-05-10 10:47:04,761: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 411110d3-dd45-4a6b-8084-f178eec16b90
[2017-05-10 10:47:04,855: INFO/job_scheduler] Job 80 finished 2 steps successfully
[2017-05-10 10:47:04,915: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 140516567204688
[2017-05-10 10:47:04,939: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 10:47:04,958: INFO/job] Job 80: StateChangeJob complete, setting state lnet_down on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 10:47:04,958: INFO/job] Job 80: not_deleted=True
[2017-05-10 10:47:04,958: INFO/job_scheduler] Job 80: complete_job: Updating cache
[2017-05-10 10:47:04,958: INFO/job_scheduler] Job 80: locked_item 140516567204688 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 10:47:04,962: INFO/job_scheduler] Job 80 completing (errored=False, cancelled=False)
[2017-05-10 10:47:04,998: INFO/job_scheduler] run_next: 1 runnable jobs of (18 pending, 9 tasked)
[2017-05-10 10:47:05,057: INFO/job_scheduler] Job 81: RunJobThread.run
[2017-05-10 10:47:05,057: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_lnet {}
[2017-05-10 10:47:05,099: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc eded2f13-26d5-4688-b726-d43a893cb18d
[2017-05-10 10:47:05,099: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc eded2f13-26d5-4688-b726-d43a893cb18d
[2017-05-10 10:47:05,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:05,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:05,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:05,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:05,199: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc eded2f13-26d5-4688-b726-d43a893cb18d
[2017-05-10 10:47:05,199: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc eded2f13-26d5-4688-b726-d43a893cb18d
[2017-05-10 10:47:05,220: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc eded2f13-26d5-4688-b726-d43a893cb18d
[2017-05-10 10:47:05,220: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc eded2f13-26d5-4688-b726-d43a893cb18d
[2017-05-10 10:47:05,220: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 10:47:05,246: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 60edd589-cb9b-4305-8dec-387335e55d03
[2017-05-10 10:47:05,246: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 60edd589-cb9b-4305-8dec-387335e55d03
[2017-05-10 10:47:05,323: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:05,323: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:05,324: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:05,324: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:05,324: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 60edd589-cb9b-4305-8dec-387335e55d03
[2017-05-10 10:47:05,324: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 60edd589-cb9b-4305-8dec-387335e55d03
[2017-05-10 10:47:05,366: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 60edd589-cb9b-4305-8dec-387335e55d03
[2017-05-10 10:47:05,366: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 60edd589-cb9b-4305-8dec-387335e55d03
[2017-05-10 10:47:05,800: INFO/job_scheduler] Job 102 finished 2 steps successfully
[2017-05-10 10:47:05,833: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 140516567196560
[2017-05-10 10:47:05,839: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 10:47:05,867: INFO/job] Job 102: StateChangeJob complete, setting state lnet_down on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 10:47:05,867: INFO/job] Job 102: not_deleted=True
[2017-05-10 10:47:05,867: INFO/job_scheduler] Job 102: complete_job: Updating cache
[2017-05-10 10:47:05,867: INFO/job_scheduler] Job 102: locked_item 140516567196560 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 10:47:05,873: INFO/job_scheduler] Job 102 completing (errored=False, cancelled=False)
[2017-05-10 10:47:05,932: INFO/job_scheduler] run_next: 1 runnable jobs of (17 pending, 9 tasked)
[2017-05-10 10:47:05,976: INFO/job_scheduler] Job 103: RunJobThread.run
[2017-05-10 10:47:05,979: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_lnet {}
[2017-05-10 10:47:06,030: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 65537dc3-c29a-45e6-a71a-55c18dc40e4b
[2017-05-10 10:47:06,030: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 65537dc3-c29a-45e6-a71a-55c18dc40e4b
[2017-05-10 10:47:06,276: INFO/job_scheduler] Job 81 finished 2 steps successfully
[2017-05-10 10:47:06,291: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 140516567204688
[2017-05-10 10:47:06,296: INFO/job] Job 81: StateChangeJob complete, setting state lnet_up on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 10:47:06,296: INFO/job] Job 81: not_deleted=True
[2017-05-10 10:47:06,296: INFO/job_scheduler] Job 81: complete_job: Updating cache
[2017-05-10 10:47:06,296: INFO/job_scheduler] Job 81: locked_item 140516567204688 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 10:47:06,304: INFO/job_scheduler] Job 81 completing (errored=False, cancelled=False)
[2017-05-10 10:47:06,348: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 9 tasked)
[2017-05-10 10:47:07,058: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:07,058: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:07,059: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:07,059: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:07,059: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ba913ada-42ea-4513-8c77-e31d0e3ef7e5
[2017-05-10 10:47:07,059: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ba913ada-42ea-4513-8c77-e31d0e3ef7e5
[2017-05-10 10:47:07,107: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ba913ada-42ea-4513-8c77-e31d0e3ef7e5
[2017-05-10 10:47:07,107: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ba913ada-42ea-4513-8c77-e31d0e3ef7e5
[2017-05-10 10:47:07,108: INFO/job_scheduler] Job 77 finished 1 steps successfully
[2017-05-10 10:47:07,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:07,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:07,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:07,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:07,190: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 65537dc3-c29a-45e6-a71a-55c18dc40e4b
[2017-05-10 10:47:07,190: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 65537dc3-c29a-45e6-a71a-55c18dc40e4b
[2017-05-10 10:47:07,206: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 65537dc3-c29a-45e6-a71a-55c18dc40e4b
[2017-05-10 10:47:07,206: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 65537dc3-c29a-45e6-a71a-55c18dc40e4b
[2017-05-10 10:47:07,206: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 10:47:07,215: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 140516567204496
[2017-05-10 10:47:07,220: INFO/job] Job 77: StateChangeJob complete, setting state configured on lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-10 10:47:07,220: INFO/job] Job 77: not_deleted=True
[2017-05-10 10:47:07,220: INFO/job_scheduler] Job 77: complete_job: Updating cache
[2017-05-10 10:47:07,220: INFO/job_scheduler] Job 77: locked_item 140516567204496 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-10 10:47:07,223: INFO/job_scheduler] Job 77 completing (errored=False, cancelled=False)
[2017-05-10 10:47:07,237: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a0b2f9aa-c22d-4521-8999-77fd111b9062
[2017-05-10 10:47:07,237: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a0b2f9aa-c22d-4521-8999-77fd111b9062
[2017-05-10 10:47:07,255: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 8 tasked)
[2017-05-10 10:47:07,432: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:07,432: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:07,432: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:07,432: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:07,432: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a0b2f9aa-c22d-4521-8999-77fd111b9062
[2017-05-10 10:47:07,432: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a0b2f9aa-c22d-4521-8999-77fd111b9062
[2017-05-10 10:47:07,459: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a0b2f9aa-c22d-4521-8999-77fd111b9062
[2017-05-10 10:47:07,459: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a0b2f9aa-c22d-4521-8999-77fd111b9062
[2017-05-10 10:47:07,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:07,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:07,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:07,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:07,567: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e12ce2b4-6bb1-4414-9a64-ec2b78d69405
[2017-05-10 10:47:07,567: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e12ce2b4-6bb1-4414-9a64-ec2b78d69405
[2017-05-10 10:47:07,617: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e12ce2b4-6bb1-4414-9a64-ec2b78d69405
[2017-05-10 10:47:07,617: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e12ce2b4-6bb1-4414-9a64-ec2b78d69405
[2017-05-10 10:47:07,617: INFO/job_scheduler] Job 99 finished 1 steps successfully
[2017-05-10 10:47:07,707: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 140516566766416
[2017-05-10 10:47:07,724: INFO/job] Job 99: StateChangeJob complete, setting state configured on lotus-32vm8.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-10 10:47:07,724: INFO/job] Job 99: not_deleted=True
[2017-05-10 10:47:07,724: INFO/job_scheduler] Job 99: complete_job: Updating cache
[2017-05-10 10:47:07,724: INFO/job_scheduler] Job 99: locked_item 140516566766416 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-10 10:47:07,727: INFO/job_scheduler] Job 99 completing (errored=False, cancelled=False)
[2017-05-10 10:47:07,798: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 7 tasked)
[2017-05-10 10:47:07,837: INFO/job_scheduler] Job 103 finished 2 steps successfully
[2017-05-10 10:47:07,967: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 140516567196560
[2017-05-10 10:47:07,980: INFO/job] Job 103: StateChangeJob complete, setting state lnet_up on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 10:47:07,980: INFO/job] Job 103: not_deleted=True
[2017-05-10 10:47:07,980: INFO/job_scheduler] Job 103: complete_job: Updating cache
[2017-05-10 10:47:07,980: INFO/job_scheduler] Job 103: locked_item 140516567196560 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 10:47:07,987: INFO/job_scheduler] Job 103 completing (errored=False, cancelled=False)
[2017-05-10 10:47:08,062: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:08,062: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:08,062: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:08,062: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:08,063: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ea2606fe-d293-4a74-b10d-88629d8255a9
[2017-05-10 10:47:08,063: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ea2606fe-d293-4a74-b10d-88629d8255a9
[2017-05-10 10:47:08,104: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ea2606fe-d293-4a74-b10d-88629d8255a9
[2017-05-10 10:47:08,104: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ea2606fe-d293-4a74-b10d-88629d8255a9
[2017-05-10 10:47:08,105: INFO/job_scheduler] Job 75 finished 1 steps successfully
[2017-05-10 10:47:08,107: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 6 tasked)
[2017-05-10 10:47:08,768: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 140516567195408
[2017-05-10 10:47:08,775: INFO/job] Job 75: StateChangeJob complete, setting state configured on lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-10 10:47:08,775: INFO/job] Job 75: not_deleted=True
[2017-05-10 10:47:08,776: INFO/job_scheduler] Job 75: complete_job: Updating cache
[2017-05-10 10:47:08,776: INFO/job_scheduler] Job 75: locked_item 140516567195408 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-10 10:47:08,791: INFO/job_scheduler] Job 75 completing (errored=False, cancelled=False)
[2017-05-10 10:47:08,885: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 5 tasked)
[2017-05-10 10:47:10,657: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:10,657: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:10,657: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:10,657: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:10,658: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 715b5310-6867-47e2-a995-20e75f7172b6
[2017-05-10 10:47:10,658: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 715b5310-6867-47e2-a995-20e75f7172b6
[2017-05-10 10:47:10,662: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 715b5310-6867-47e2-a995-20e75f7172b6
[2017-05-10 10:47:10,662: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 715b5310-6867-47e2-a995-20e75f7172b6
[2017-05-10 10:47:10,663: INFO/job_scheduler] Job 97 finished 1 steps successfully
[2017-05-10 10:47:10,757: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 140516566767568
[2017-05-10 10:47:10,765: INFO/job] Job 97: StateChangeJob complete, setting state configured on lotus-32vm8.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-10 10:47:10,765: INFO/job] Job 97: not_deleted=True
[2017-05-10 10:47:10,765: INFO/job_scheduler] Job 97: complete_job: Updating cache
[2017-05-10 10:47:10,765: INFO/job_scheduler] Job 97: locked_item 140516566767568 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-10 10:47:10,776: INFO/job_scheduler] Job 97 completing (errored=False, cancelled=False)
[2017-05-10 10:47:10,834: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 4 tasked)
[2017-05-10 10:47:18,151: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:18,151: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:18,151: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:18,151: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:18,152: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fd86532b-557a-41b8-8885-34ad23695aa4
[2017-05-10 10:47:18,152: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fd86532b-557a-41b8-8885-34ad23695aa4
[2017-05-10 10:47:18,161: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fd86532b-557a-41b8-8885-34ad23695aa4
[2017-05-10 10:47:18,161: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fd86532b-557a-41b8-8885-34ad23695aa4
[2017-05-10 10:47:18,161: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.227', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-05-10 10:47:18,177: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc eba5ca63-d4f4-4709-b919-fb0c11394eba
[2017-05-10 10:47:18,177: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc eba5ca63-d4f4-4709-b919-fb0c11394eba
[2017-05-10 10:47:18,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:18,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:18,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:18,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:18,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc eba5ca63-d4f4-4709-b919-fb0c11394eba
[2017-05-10 10:47:18,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc eba5ca63-d4f4-4709-b919-fb0c11394eba
[2017-05-10 10:47:18,295: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc eba5ca63-d4f4-4709-b919-fb0c11394eba
[2017-05-10 10:47:18,295: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc eba5ca63-d4f4-4709-b919-fb0c11394eba
[2017-05-10 10:47:18,313: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 52671, 'pcs_password': 'ZK2MA4NKp980fRZxWNr2'}
[2017-05-10 10:47:18,325: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc df44d018-b71b-4313-8596-cda2f7cb2056
[2017-05-10 10:47:18,325: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc df44d018-b71b-4313-8596-cda2f7cb2056
[2017-05-10 10:47:18,762: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:18,762: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:18,762: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:18,762: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:18,762: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 561fab06-668e-4f0d-ab6f-18d17d514134
[2017-05-10 10:47:18,762: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 561fab06-668e-4f0d-ab6f-18d17d514134
[2017-05-10 10:47:18,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 561fab06-668e-4f0d-ab6f-18d17d514134
[2017-05-10 10:47:18,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 561fab06-668e-4f0d-ab6f-18d17d514134
[2017-05-10 10:47:18,781: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.224', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-05-10 10:47:18,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f990978f-6c2e-4395-a8c5-0b8687944bbb
[2017-05-10 10:47:18,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f990978f-6c2e-4395-a8c5-0b8687944bbb
[2017-05-10 10:47:18,942: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:18,942: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:18,942: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:18,942: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:18,942: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f990978f-6c2e-4395-a8c5-0b8687944bbb
[2017-05-10 10:47:18,942: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f990978f-6c2e-4395-a8c5-0b8687944bbb
[2017-05-10 10:47:18,967: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f990978f-6c2e-4395-a8c5-0b8687944bbb
[2017-05-10 10:47:18,967: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f990978f-6c2e-4395-a8c5-0b8687944bbb
[2017-05-10 10:47:19,007: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 48953, 'pcs_password': 'YzUaPBsE2IDFWW8N8pBD'}
[2017-05-10 10:47:19,030: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 365dff3a-40b7-4b1c-bb97-63e21cda7da4
[2017-05-10 10:47:19,030: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 365dff3a-40b7-4b1c-bb97-63e21cda7da4
[2017-05-10 10:47:30,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:30,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:30,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:30,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:30,452: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc df44d018-b71b-4313-8596-cda2f7cb2056
[2017-05-10 10:47:30,452: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc df44d018-b71b-4313-8596-cda2f7cb2056
[2017-05-10 10:47:30,496: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc df44d018-b71b-4313-8596-cda2f7cb2056
[2017-05-10 10:47:30,496: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc df44d018-b71b-4313-8596-cda2f7cb2056
[2017-05-10 10:47:30,497: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': True, 'mcast_port': 52671, 'new_node_fqdn': u'lotus-32vm8.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'ZK2MA4NKp980fRZxWNr2'}
[2017-05-10 10:47:30,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d8c7947c-98aa-4275-b0f3-2e6ced1f101a
[2017-05-10 10:47:30,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d8c7947c-98aa-4275-b0f3-2e6ced1f101a
[2017-05-10 10:47:31,178: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:31,178: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:31,178: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:31,178: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:31,179: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 365dff3a-40b7-4b1c-bb97-63e21cda7da4
[2017-05-10 10:47:31,179: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 365dff3a-40b7-4b1c-bb97-63e21cda7da4
[2017-05-10 10:47:31,213: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 365dff3a-40b7-4b1c-bb97-63e21cda7da4
[2017-05-10 10:47:31,213: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 365dff3a-40b7-4b1c-bb97-63e21cda7da4
[2017-05-10 10:47:31,214: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': True, 'mcast_port': 48953, 'new_node_fqdn': u'lotus-32vm5.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'YzUaPBsE2IDFWW8N8pBD'}
[2017-05-10 10:47:31,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 69b39008-ac3a-45a6-ac1e-d3b8f1cfe0a5
[2017-05-10 10:47:31,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 69b39008-ac3a-45a6-ac1e-d3b8f1cfe0a5
[2017-05-10 10:47:36,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:36,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:36,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:36,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:36,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d8c7947c-98aa-4275-b0f3-2e6ced1f101a
[2017-05-10 10:47:36,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d8c7947c-98aa-4275-b0f3-2e6ced1f101a
[2017-05-10 10:47:36,657: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d8c7947c-98aa-4275-b0f3-2e6ced1f101a
[2017-05-10 10:47:36,657: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d8c7947c-98aa-4275-b0f3-2e6ced1f101a
[2017-05-10 10:47:36,658: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 10:47:36.658192+00:00:
[2017-05-10 10:47:36,658: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 10:47:36.658192+00:00:
[2017-05-10 10:47:36,658: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->52671
[2017-05-10 10:47:36,658: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->52671
[2017-05-10 10:47:36,660: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 10:47:36,660: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 10:47:36,676: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:47:36,677: INFO/job_scheduler]   Job 98 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-05-10 10:47:36,677: INFO/job_scheduler]   Job 101 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-05-10 10:47:36,677: INFO/job_scheduler]   Job 104 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:47:36,677: INFO/job_scheduler]   Job 105 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:47:36,678: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 4 tasked)
[2017-05-10 10:47:36,679: INFO/job_scheduler] Job 98 finished 1 steps successfully
[2017-05-10 10:47:36,733: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 140516566814032
[2017-05-10 10:47:36,795: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-10 10:47:36,812: INFO/job] Job 98: StateChangeJob complete, setting state stopped on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:47:36,812: INFO/job] Job 98: not_deleted=True
[2017-05-10 10:47:36,812: INFO/job_scheduler] Job 98: complete_job: Updating cache
[2017-05-10 10:47:36,812: INFO/job_scheduler] Job 98: locked_item 140516566814032 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 10:47:36,817: INFO/job_scheduler] Job 98 completing (errored=False, cancelled=False)
[2017-05-10 10:47:36,860: INFO/job_scheduler] run_next: 1 runnable jobs of (16 pending, 3 tasked)
[2017-05-10 10:47:36,889: INFO/job_scheduler] Job 101: RunJobThread.run
[2017-05-10 10:47:36,890: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-05-10 10:47:36,944: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 34ca036a-939e-4caf-a096-b22e83dd3324
[2017-05-10 10:47:36,944: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 34ca036a-939e-4caf-a096-b22e83dd3324
[2017-05-10 10:47:37,395: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:37,395: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:37,395: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:37,395: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:37,395: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 69b39008-ac3a-45a6-ac1e-d3b8f1cfe0a5
[2017-05-10 10:47:37,395: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 69b39008-ac3a-45a6-ac1e-d3b8f1cfe0a5
[2017-05-10 10:47:37,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 69b39008-ac3a-45a6-ac1e-d3b8f1cfe0a5
[2017-05-10 10:47:37,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 69b39008-ac3a-45a6-ac1e-d3b8f1cfe0a5
[2017-05-10 10:47:37,422: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 10:47:37.421988+00:00:
[2017-05-10 10:47:37,422: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 10:47:37.421988+00:00:
[2017-05-10 10:47:37,422: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->48953
[2017-05-10 10:47:37,422: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->48953
[2017-05-10 10:47:37,425: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 10:47:37,425: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 10:47:37,453: INFO/job_scheduler] Job 78 finished 1 steps successfully
[2017-05-10 10:47:37,460: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:47:37,460: INFO/job_scheduler]   Job 78 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-05-10 10:47:37,460: INFO/job_scheduler]   Job 79 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-05-10 10:47:37,460: INFO/job_scheduler]   Job 82 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:47:37,460: INFO/job_scheduler]   Job 83 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:47:37,461: INFO/job_scheduler] run_next: 0 runnable jobs of (15 pending, 4 tasked)
[2017-05-10 10:47:37,499: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 140516567197840
[2017-05-10 10:47:37,506: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-10 10:47:37,528: INFO/job] Job 78: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:47:37,528: INFO/job] Job 78: not_deleted=True
[2017-05-10 10:47:37,528: INFO/job_scheduler] Job 78: complete_job: Updating cache
[2017-05-10 10:47:37,529: INFO/job_scheduler] Job 78: locked_item 140516567197840 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 10:47:37,540: INFO/job_scheduler] Job 78 completing (errored=False, cancelled=False)
[2017-05-10 10:47:37,675: INFO/job_scheduler] run_next: 1 runnable jobs of (15 pending, 3 tasked)
[2017-05-10 10:47:37,707: INFO/job_scheduler] Job 79: RunJobThread.run
[2017-05-10 10:47:37,708: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-05-10 10:47:37,745: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 65fba84d-b35a-4bba-88ba-53912a3cf32f
[2017-05-10 10:47:37,745: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 65fba84d-b35a-4bba-88ba-53912a3cf32f
[2017-05-10 10:47:38,234: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:47:38,234: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:47:38,235: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:47:38,235: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:47:38,235: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc eec0fa19-0eaa-4079-8e56-2ac47c9973f9
[2017-05-10 10:47:38,235: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc eec0fa19-0eaa-4079-8e56-2ac47c9973f9
[2017-05-10 10:47:38,235: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc eec0fa19-0eaa-4079-8e56-2ac47c9973f9
[2017-05-10 10:47:38,235: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc eec0fa19-0eaa-4079-8e56-2ac47c9973f9
[2017-05-10 10:47:38,236: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.226', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-05-10 10:47:38,259: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a33a9ef5-bf08-42d5-9b9c-f1c03cdd3c82
[2017-05-10 10:47:38,259: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a33a9ef5-bf08-42d5-9b9c-f1c03cdd3c82
[2017-05-10 10:47:38,358: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:47:38,358: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:47:38,360: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:47:38,360: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:47:38,361: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a33a9ef5-bf08-42d5-9b9c-f1c03cdd3c82
[2017-05-10 10:47:38,361: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a33a9ef5-bf08-42d5-9b9c-f1c03cdd3c82
[2017-05-10 10:47:38,388: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a33a9ef5-bf08-42d5-9b9c-f1c03cdd3c82
[2017-05-10 10:47:38,388: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a33a9ef5-bf08-42d5-9b9c-f1c03cdd3c82
[2017-05-10 10:47:38,433: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 52671, 'pcs_password': 'Ib7eIU04xCZEOhSBaWIo'}
[2017-05-10 10:47:38,462: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cca3b1e6-b9f6-45f1-bfad-3a60cbc0f7dd
[2017-05-10 10:47:38,462: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cca3b1e6-b9f6-45f1-bfad-3a60cbc0f7dd
[2017-05-10 10:47:39,036: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:47:39,036: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:47:39,036: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:47:39,036: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:47:39,037: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 92ab9431-3dfe-4098-876e-70e2b22de255
[2017-05-10 10:47:39,037: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 92ab9431-3dfe-4098-876e-70e2b22de255
[2017-05-10 10:47:39,083: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 92ab9431-3dfe-4098-876e-70e2b22de255
[2017-05-10 10:47:39,083: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 92ab9431-3dfe-4098-876e-70e2b22de255
[2017-05-10 10:47:39,084: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.225', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-05-10 10:47:39,099: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8d9258ef-09c7-4051-a5d7-94cc430188cd
[2017-05-10 10:47:39,099: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8d9258ef-09c7-4051-a5d7-94cc430188cd
[2017-05-10 10:47:39,158: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:47:39,158: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:47:39,158: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:47:39,158: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:47:39,159: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8d9258ef-09c7-4051-a5d7-94cc430188cd
[2017-05-10 10:47:39,159: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8d9258ef-09c7-4051-a5d7-94cc430188cd
[2017-05-10 10:47:39,166: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8d9258ef-09c7-4051-a5d7-94cc430188cd
[2017-05-10 10:47:39,166: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8d9258ef-09c7-4051-a5d7-94cc430188cd
[2017-05-10 10:47:39,200: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 48953, 'pcs_password': 'vPq3uf4eDRJTNZ8X7WI8'}
[2017-05-10 10:47:39,224: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7c782141-813c-4457-bf03-dbfda89c0369
[2017-05-10 10:47:39,224: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7c782141-813c-4457-bf03-dbfda89c0369
[2017-05-10 10:47:45,026: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:45,026: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:45,026: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:45,026: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:45,027: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 34ca036a-939e-4caf-a096-b22e83dd3324
[2017-05-10 10:47:45,027: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 34ca036a-939e-4caf-a096-b22e83dd3324
[2017-05-10 10:47:45,073: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 34ca036a-939e-4caf-a096-b22e83dd3324
[2017-05-10 10:47:45,073: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 34ca036a-939e-4caf-a096-b22e83dd3324
[2017-05-10 10:47:45,073: INFO/job_scheduler] Job 101 finished 1 steps successfully
[2017-05-10 10:47:45,147: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 140516566814032
[2017-05-10 10:47:45,205: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm8.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 10:47:45,234: INFO/job] Job 101: StateChangeJob complete, setting state started on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:47:45,234: INFO/job] Job 101: not_deleted=True
[2017-05-10 10:47:45,234: INFO/job_scheduler] Job 101: complete_job: Updating cache
[2017-05-10 10:47:45,234: INFO/job_scheduler] Job 101: locked_item 140516566814032 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 10:47:45,253: INFO/job_scheduler] Job 101 completing (errored=False, cancelled=False)
[2017-05-10 10:47:45,303: INFO/job_scheduler] run_next: 1 runnable jobs of (14 pending, 3 tasked)
[2017-05-10 10:47:45,332: INFO/job_scheduler] Job 104: RunJobThread.run
[2017-05-10 10:47:45,333: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 10:47:45,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d16d289f-e9d7-4520-ae79-b3935da4ab2e
[2017-05-10 10:47:45,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d16d289f-e9d7-4520-ae79-b3935da4ab2e
[2017-05-10 10:47:45,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:45,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:45,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:45,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:45,857: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 65fba84d-b35a-4bba-88ba-53912a3cf32f
[2017-05-10 10:47:45,857: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 65fba84d-b35a-4bba-88ba-53912a3cf32f
[2017-05-10 10:47:45,866: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 65fba84d-b35a-4bba-88ba-53912a3cf32f
[2017-05-10 10:47:45,866: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 65fba84d-b35a-4bba-88ba-53912a3cf32f
[2017-05-10 10:47:45,866: INFO/job_scheduler] Job 79 finished 1 steps successfully
[2017-05-10 10:47:45,926: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 140516567197840
[2017-05-10 10:47:45,967: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 10:47:45,974: INFO/job] Job 79: StateChangeJob complete, setting state started on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:47:45,974: INFO/job] Job 79: not_deleted=True
[2017-05-10 10:47:45,974: INFO/job_scheduler] Job 79: complete_job: Updating cache
[2017-05-10 10:47:45,975: INFO/job_scheduler] Job 79: locked_item 140516567197840 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 10:47:45,980: INFO/job_scheduler] Job 79 completing (errored=False, cancelled=False)
[2017-05-10 10:47:46,020: INFO/job_scheduler] run_next: 1 runnable jobs of (13 pending, 3 tasked)
[2017-05-10 10:47:46,033: INFO/job_scheduler] Job 82: RunJobThread.run
[2017-05-10 10:47:46,034: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 10:47:46,050: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 41122164-7344-4884-b8a0-b89542f68786
[2017-05-10 10:47:46,050: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 41122164-7344-4884-b8a0-b89542f68786
[2017-05-10 10:47:48,381: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-10 10:47:48,432: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-10 10:47:48,489: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-10 10:47:48,535: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-10 10:47:50,662: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:47:50,662: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:47:50,663: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:47:50,663: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:47:50,663: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cca3b1e6-b9f6-45f1-bfad-3a60cbc0f7dd
[2017-05-10 10:47:50,663: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cca3b1e6-b9f6-45f1-bfad-3a60cbc0f7dd
[2017-05-10 10:47:50,689: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cca3b1e6-b9f6-45f1-bfad-3a60cbc0f7dd
[2017-05-10 10:47:50,689: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cca3b1e6-b9f6-45f1-bfad-3a60cbc0f7dd
[2017-05-10 10:47:50,690: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': False, 'mcast_port': 52671, 'new_node_fqdn': u'lotus-32vm7.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'Ib7eIU04xCZEOhSBaWIo'}
[2017-05-10 10:47:50,712: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2f333c3c-1b68-4274-adfc-47790e12d76d
[2017-05-10 10:47:50,712: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2f333c3c-1b68-4274-adfc-47790e12d76d
[2017-05-10 10:47:51,369: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:47:51,369: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:47:51,370: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:47:51,370: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:47:51,370: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7c782141-813c-4457-bf03-dbfda89c0369
[2017-05-10 10:47:51,370: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7c782141-813c-4457-bf03-dbfda89c0369
[2017-05-10 10:47:51,403: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7c782141-813c-4457-bf03-dbfda89c0369
[2017-05-10 10:47:51,403: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7c782141-813c-4457-bf03-dbfda89c0369
[2017-05-10 10:47:51,404: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': False, 'mcast_port': 48953, 'new_node_fqdn': u'lotus-32vm6.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'vPq3uf4eDRJTNZ8X7WI8'}
[2017-05-10 10:47:51,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 52f9d879-1a41-447f-9c91-cb10bb8c1188
[2017-05-10 10:47:51,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 52f9d879-1a41-447f-9c91-cb10bb8c1188
[2017-05-10 10:47:52,488: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:52,488: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:52,489: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:52,489: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:52,489: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d16d289f-e9d7-4520-ae79-b3935da4ab2e
[2017-05-10 10:47:52,489: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d16d289f-e9d7-4520-ae79-b3935da4ab2e
[2017-05-10 10:47:52,521: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d16d289f-e9d7-4520-ae79-b3935da4ab2e
[2017-05-10 10:47:52,521: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d16d289f-e9d7-4520-ae79-b3935da4ab2e
[2017-05-10 10:47:52,522: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-05-10 10:47:52,540: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 37b93509-e3fb-4069-afc9-7ebfe40323d0
[2017-05-10 10:47:52,540: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 37b93509-e3fb-4069-afc9-7ebfe40323d0
[2017-05-10 10:47:53,179: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:53,179: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:53,179: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:53,179: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:53,179: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 41122164-7344-4884-b8a0-b89542f68786
[2017-05-10 10:47:53,179: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 41122164-7344-4884-b8a0-b89542f68786
[2017-05-10 10:47:53,223: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 41122164-7344-4884-b8a0-b89542f68786
[2017-05-10 10:47:53,223: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 41122164-7344-4884-b8a0-b89542f68786
[2017-05-10 10:47:53,224: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-05-10 10:47:53,242: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b930eeb1-a7d3-466c-bd2a-12548866880c
[2017-05-10 10:47:53,242: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b930eeb1-a7d3-466c-bd2a-12548866880c
[2017-05-10 10:47:56,853: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:56,853: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:56,853: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:56,853: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:47:56,853: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2f333c3c-1b68-4274-adfc-47790e12d76d
[2017-05-10 10:47:56,853: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2f333c3c-1b68-4274-adfc-47790e12d76d
[2017-05-10 10:47:56,889: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2f333c3c-1b68-4274-adfc-47790e12d76d
[2017-05-10 10:47:56,889: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2f333c3c-1b68-4274-adfc-47790e12d76d
[2017-05-10 10:47:56,890: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 10:47:56.889939+00:00:
[2017-05-10 10:47:56,890: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 10:47:56.889939+00:00:
[2017-05-10 10:47:56,890: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->52671
[2017-05-10 10:47:56,890: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->52671
[2017-05-10 10:47:56,892: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 10:47:56,892: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 10:47:56,921: INFO/job_scheduler] Job 87 finished 1 steps successfully
[2017-05-10 10:47:56,923: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:47:56,923: INFO/job_scheduler]   Job 87 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-05-10 10:47:56,923: INFO/job_scheduler]   Job 91 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-05-10 10:47:56,923: INFO/job_scheduler]   Job 93 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:47:56,923: INFO/job_scheduler]   Job 94 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:47:56,924: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-10 10:47:56,942: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 140516567187600
[2017-05-10 10:47:56,950: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-10 10:47:56,971: INFO/job] Job 87: StateChangeJob complete, setting state stopped on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:47:56,971: INFO/job] Job 87: not_deleted=True
[2017-05-10 10:47:56,971: INFO/job_scheduler] Job 87: complete_job: Updating cache
[2017-05-10 10:47:56,971: INFO/job_scheduler] Job 87: locked_item 140516567187600 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 10:47:56,981: INFO/job_scheduler] Job 87 completing (errored=False, cancelled=False)
[2017-05-10 10:47:57,031: INFO/job_scheduler] run_next: 1 runnable jobs of (12 pending, 3 tasked)
[2017-05-10 10:47:57,071: INFO/job_scheduler] Job 91: RunJobThread.run
[2017-05-10 10:47:57,071: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-05-10 10:47:57,131: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 45058db2-d9c6-4e69-b55d-f7b1d29d0849
[2017-05-10 10:47:57,131: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 45058db2-d9c6-4e69-b55d-f7b1d29d0849
[2017-05-10 10:47:58,388: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 4 tasked)
[2017-05-10 10:47:58,437: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 4 tasked)
[2017-05-10 10:47:58,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:58,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:58,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:58,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:47:58,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 52f9d879-1a41-447f-9c91-cb10bb8c1188
[2017-05-10 10:47:58,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 52f9d879-1a41-447f-9c91-cb10bb8c1188
[2017-05-10 10:47:58,551: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 52f9d879-1a41-447f-9c91-cb10bb8c1188
[2017-05-10 10:47:58,551: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 52f9d879-1a41-447f-9c91-cb10bb8c1188
[2017-05-10 10:47:58,551: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 10:47:58.551485+00:00:
[2017-05-10 10:47:58,551: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 10:47:58.551485+00:00:
[2017-05-10 10:47:58,551: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->48953
[2017-05-10 10:47:58,551: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->48953
[2017-05-10 10:47:58,553: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 10:47:58,553: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 10:47:58,593: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:47:58,593: INFO/job_scheduler]   Job 108 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-05-10 10:47:58,593: INFO/job_scheduler]   Job 113 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-05-10 10:47:58,593: INFO/job_scheduler]   Job 114 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:47:58,593: INFO/job_scheduler]   Job 116 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:47:58,593: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 4 tasked)
[2017-05-10 10:47:58,597: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 4 tasked)
[2017-05-10 10:47:58,601: INFO/job_scheduler] Job 108 finished 1 steps successfully
[2017-05-10 10:47:58,627: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 4 tasked)
[2017-05-10 10:47:58,691: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 140516567162000
[2017-05-10 10:47:58,707: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-10 10:47:58,748: INFO/job] Job 108: StateChangeJob complete, setting state stopped on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:47:58,748: INFO/job] Job 108: not_deleted=True
[2017-05-10 10:47:58,749: INFO/job_scheduler] Job 108: complete_job: Updating cache
[2017-05-10 10:47:58,749: INFO/job_scheduler] Job 108: locked_item 140516567162000 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 10:47:58,764: INFO/job_scheduler] Job 108 completing (errored=False, cancelled=False)
[2017-05-10 10:47:58,840: INFO/job_scheduler] run_next: 1 runnable jobs of (11 pending, 3 tasked)
[2017-05-10 10:47:58,868: INFO/job_scheduler] Job 113: RunJobThread.run
[2017-05-10 10:47:58,869: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-05-10 10:47:58,917: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc aa0df37c-5b65-4b3d-b133-31573d778e52
[2017-05-10 10:47:58,917: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc aa0df37c-5b65-4b3d-b133-31573d778e52
[2017-05-10 10:48:05,251: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:48:05,251: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:48:05,251: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:48:05,251: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:48:05,252: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 45058db2-d9c6-4e69-b55d-f7b1d29d0849
[2017-05-10 10:48:05,252: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 45058db2-d9c6-4e69-b55d-f7b1d29d0849
[2017-05-10 10:48:05,256: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 45058db2-d9c6-4e69-b55d-f7b1d29d0849
[2017-05-10 10:48:05,256: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 45058db2-d9c6-4e69-b55d-f7b1d29d0849
[2017-05-10 10:48:05,257: INFO/job_scheduler] Job 91 finished 1 steps successfully
[2017-05-10 10:48:05,450: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 140516567187600
[2017-05-10 10:48:05,500: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 10:48:05,530: INFO/job] Job 91: StateChangeJob complete, setting state started on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:48:05,530: INFO/job] Job 91: not_deleted=True
[2017-05-10 10:48:05,530: INFO/job_scheduler] Job 91: complete_job: Updating cache
[2017-05-10 10:48:05,530: INFO/job_scheduler] Job 91: locked_item 140516567187600 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 10:48:05,546: INFO/job_scheduler] Job 91 completing (errored=False, cancelled=False)
[2017-05-10 10:48:05,622: INFO/job_scheduler] run_next: 1 runnable jobs of (10 pending, 3 tasked)
[2017-05-10 10:48:05,664: INFO/job_scheduler] Job 93: RunJobThread.run
[2017-05-10 10:48:05,665: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 10:48:05,712: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0827419c-e64a-4f0e-9d62-4e76eea50b4c
[2017-05-10 10:48:05,712: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0827419c-e64a-4f0e-9d62-4e76eea50b4c
[2017-05-10 10:48:07,009: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:48:07,009: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:48:07,009: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:48:07,009: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:48:07,010: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc aa0df37c-5b65-4b3d-b133-31573d778e52
[2017-05-10 10:48:07,010: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc aa0df37c-5b65-4b3d-b133-31573d778e52
[2017-05-10 10:48:07,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc aa0df37c-5b65-4b3d-b133-31573d778e52
[2017-05-10 10:48:07,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc aa0df37c-5b65-4b3d-b133-31573d778e52
[2017-05-10 10:48:07,044: INFO/job_scheduler] Job 113 finished 1 steps successfully
[2017-05-10 10:48:07,183: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 140516567162000
[2017-05-10 10:48:07,243: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm6.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 10:48:07,289: INFO/job] Job 113: StateChangeJob complete, setting state started on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:48:07,290: INFO/job] Job 113: not_deleted=True
[2017-05-10 10:48:07,294: INFO/job_scheduler] Job 113: complete_job: Updating cache
[2017-05-10 10:48:07,294: INFO/job_scheduler] Job 113: locked_item 140516567162000 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 10:48:07,317: INFO/job_scheduler] Job 113 completing (errored=False, cancelled=False)
[2017-05-10 10:48:07,375: INFO/job_scheduler] run_next: 1 runnable jobs of (9 pending, 3 tasked)
[2017-05-10 10:48:07,425: INFO/job_scheduler] Job 114: RunJobThread.run
[2017-05-10 10:48:07,426: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 10:48:07,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 489ce6ee-377b-4180-bba4-68522e25b61c
[2017-05-10 10:48:07,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 489ce6ee-377b-4180-bba4-68522e25b61c
[2017-05-10 10:48:08,031: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:08,071: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:11,027: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:11,054: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:12,828: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:48:12,828: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:48:12,829: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:48:12,829: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:48:12,829: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0827419c-e64a-4f0e-9d62-4e76eea50b4c
[2017-05-10 10:48:12,829: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0827419c-e64a-4f0e-9d62-4e76eea50b4c
[2017-05-10 10:48:12,839: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0827419c-e64a-4f0e-9d62-4e76eea50b4c
[2017-05-10 10:48:12,839: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0827419c-e64a-4f0e-9d62-4e76eea50b4c
[2017-05-10 10:48:12,841: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-05-10 10:48:12,863: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 46c4f1c3-57e5-4ff6-ba0c-a3ee45c8b708
[2017-05-10 10:48:12,863: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 46c4f1c3-57e5-4ff6-ba0c-a3ee45c8b708
[2017-05-10 10:48:14,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:48:14,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:48:14,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:48:14,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:48:14,590: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 489ce6ee-377b-4180-bba4-68522e25b61c
[2017-05-10 10:48:14,590: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 489ce6ee-377b-4180-bba4-68522e25b61c
[2017-05-10 10:48:14,597: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 489ce6ee-377b-4180-bba4-68522e25b61c
[2017-05-10 10:48:14,597: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 489ce6ee-377b-4180-bba4-68522e25b61c
[2017-05-10 10:48:14,598: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-05-10 10:48:14,615: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c5befa15-5ebc-405a-9930-77e7943c91f5
[2017-05-10 10:48:14,615: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c5befa15-5ebc-405a-9930-77e7943c91f5
[2017-05-10 10:48:18,103: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:18,133: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:18,192: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:48:18,193: INFO/job_scheduler]   Job 93 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:48:18,193: INFO/job_scheduler]   Job 94 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:48:18,193: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:18,230: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:48:18,231: INFO/job_scheduler]   Job 93 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:48:18,231: INFO/job_scheduler]   Job 95 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:48:18,231: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:18,282: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:48:18,283: INFO/job_scheduler]   Job 104 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:48:18,283: INFO/job_scheduler]   Job 105 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:48:18,283: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:18,300: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:48:18,301: INFO/job_scheduler]   Job 104 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:48:18,301: INFO/job_scheduler]   Job 106 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:48:18,301: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:18,427: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:18,442: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:21,053: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:21,090: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:28,082: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:28,101: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:28,138: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:48:28,138: INFO/job_scheduler]   Job 93 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:48:28,138: INFO/job_scheduler]   Job 95 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:48:28,138: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:28,165: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:48:28,166: INFO/job_scheduler]   Job 104 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:48:28,166: INFO/job_scheduler]   Job 106 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:48:28,166: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:28,568: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:28,636: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:28,776: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:48:28,776: INFO/job_scheduler]   Job 82 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:48:28,776: INFO/job_scheduler]   Job 84 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:48:28,777: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:28,853: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:48:28,853: INFO/job_scheduler]   Job 114 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:48:28,853: INFO/job_scheduler]   Job 117 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:48:28,854: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:31,122: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:31,142: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:31,171: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:48:31,172: INFO/job_scheduler]   Job 82 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:48:31,172: INFO/job_scheduler]   Job 84 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:48:31,172: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:31,193: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:48:31,193: INFO/job_scheduler]   Job 114 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:48:31,193: INFO/job_scheduler]   Job 117 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:48:31,193: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:38,164: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:48:38,165: INFO/job_scheduler]   Job 93 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:48:38,165: INFO/job_scheduler]   Job 95 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:48:38,165: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:38,192: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:48:38,192: INFO/job_scheduler]   Job 104 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:48:38,192: INFO/job_scheduler]   Job 106 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:48:38,193: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:38,587: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:48:38,587: INFO/job_scheduler]   Job 82 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:48:38,587: INFO/job_scheduler]   Job 84 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:48:38,588: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:38,605: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:48:38,605: INFO/job_scheduler]   Job 114 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:48:38,605: INFO/job_scheduler]   Job 117 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:48:38,605: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:41,207: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:48:41,207: INFO/job_scheduler]   Job 82 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:48:41,207: INFO/job_scheduler]   Job 84 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:48:41,208: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:41,243: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:48:41,243: INFO/job_scheduler]   Job 114 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:48:41,244: INFO/job_scheduler]   Job 117 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:48:41,244: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:48,183: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:48:48,183: INFO/job_scheduler]   Job 93 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:48:48,183: INFO/job_scheduler]   Job 94 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:48:48,183: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:48,198: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:48:48,198: INFO/job_scheduler]   Job 93 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:48:48,199: INFO/job_scheduler]   Job 95 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:48:48,199: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:48,232: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:48:48,232: INFO/job_scheduler]   Job 104 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:48:48,232: INFO/job_scheduler]   Job 105 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:48:48,233: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:48,247: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:48:48,247: INFO/job_scheduler]   Job 104 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:48:48,248: INFO/job_scheduler]   Job 106 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:48:48,248: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:48,632: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:48:48,633: INFO/job_scheduler]   Job 82 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:48:48,633: INFO/job_scheduler]   Job 83 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:48:48,633: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:48,648: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:48:48,649: INFO/job_scheduler]   Job 82 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:48:48,649: INFO/job_scheduler]   Job 84 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:48:48,649: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:48,684: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:48:48,685: INFO/job_scheduler]   Job 114 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:48:48,685: INFO/job_scheduler]   Job 116 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:48:48,685: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:48,701: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:48:48,701: INFO/job_scheduler]   Job 114 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:48:48,701: INFO/job_scheduler]   Job 117 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:48:48,701: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:51,211: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:48:51,211: INFO/job_scheduler]   Job 82 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:48:51,211: INFO/job_scheduler]   Job 83 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:48:51,211: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:51,232: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:48:51,232: INFO/job_scheduler]   Job 82 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:48:51,232: INFO/job_scheduler]   Job 84 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:48:51,232: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:51,260: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:48:51,260: INFO/job_scheduler]   Job 114 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:48:51,260: INFO/job_scheduler]   Job 116 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:48:51,261: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:51,276: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:48:51,276: INFO/job_scheduler]   Job 114 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:48:51,276: INFO/job_scheduler]   Job 117 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:48:51,277: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:53,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:48:53,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:48:53,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:48:53,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:48:53,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 37b93509-e3fb-4069-afc9-7ebfe40323d0
[2017-05-10 10:48:53,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 37b93509-e3fb-4069-afc9-7ebfe40323d0
[2017-05-10 10:48:53,905: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 37b93509-e3fb-4069-afc9-7ebfe40323d0
[2017-05-10 10:48:53,905: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 37b93509-e3fb-4069-afc9-7ebfe40323d0
[2017-05-10 10:48:53,906: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-10 10:48:53,924: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9114a148-1530-426f-ac54-f74a938be2dc
[2017-05-10 10:48:53,924: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9114a148-1530-426f-ac54-f74a938be2dc
[2017-05-10 10:48:58,218: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:48:58,218: INFO/job_scheduler]   Job 93 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:48:58,219: INFO/job_scheduler]   Job 94 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:48:58,219: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:58,239: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:48:58,239: INFO/job_scheduler]   Job 93 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:48:58,239: INFO/job_scheduler]   Job 95 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:48:58,239: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:58,272: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:48:58,272: INFO/job_scheduler]   Job 104 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:48:58,272: INFO/job_scheduler]   Job 105 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:48:58,272: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:58,302: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:48:58,302: INFO/job_scheduler]   Job 104 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:48:58,302: INFO/job_scheduler]   Job 106 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:48:58,303: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:58,574: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:58,592: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:58,624: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:48:58,624: INFO/job_scheduler]   Job 82 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:48:58,624: INFO/job_scheduler]   Job 83 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:48:58,625: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:58,637: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:48:58,637: INFO/job_scheduler]   Job 82 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:48:58,638: INFO/job_scheduler]   Job 84 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:48:58,638: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:58,656: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:48:58,656: INFO/job_scheduler]   Job 114 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:48:58,656: INFO/job_scheduler]   Job 116 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:48:58,656: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:48:58,671: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:48:58,671: INFO/job_scheduler]   Job 114 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:48:58,671: INFO/job_scheduler]   Job 117 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:48:58,671: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 10:49:00,997: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:49:00,997: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:49:01,017: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:49:01,017: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:49:01,017: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9114a148-1530-426f-ac54-f74a938be2dc
[2017-05-10 10:49:01,017: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9114a148-1530-426f-ac54-f74a938be2dc
[2017-05-10 10:49:01,046: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9114a148-1530-426f-ac54-f74a938be2dc
[2017-05-10 10:49:01,046: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9114a148-1530-426f-ac54-f74a938be2dc
[2017-05-10 10:49:01,046: INFO/job_scheduler] Job 104 finished 3 steps successfully
[2017-05-10 10:49:01,098: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 140516567064080
[2017-05-10 10:49:01,108: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-10 10:49:01,135: INFO/job] Job 104: StateChangeJob complete, setting state stopped on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 10:49:01,135: INFO/job] Job 104: not_deleted=True
[2017-05-10 10:49:01,136: INFO/job_scheduler] Job 104: complete_job: Updating cache
[2017-05-10 10:49:01,136: INFO/job_scheduler] Job 104: locked_item 140516567064080 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 10:49:01,138: INFO/job_scheduler] Job 104 completing (errored=False, cancelled=False)
[2017-05-10 10:49:01,314: INFO/job_scheduler] run_next: 1 runnable jobs of (8 pending, 3 tasked)
[2017-05-10 10:49:01,376: INFO/job_scheduler] Job 105: RunJobThread.run
[2017-05-10 10:49:01,377: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 10:49:01,399: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:49:01,400: INFO/job_scheduler]   Job 82 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:49:01,400: INFO/job_scheduler]   Job 83 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:49:01,400: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-05-10 10:49:01,402: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:49:01,402: INFO/job_scheduler]   Job 82 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:49:01,402: INFO/job_scheduler]   Job 84 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:49:01,402: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-05-10 10:49:01,406: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:49:01,406: INFO/job_scheduler]   Job 114 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:49:01,406: INFO/job_scheduler]   Job 116 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:49:01,407: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-05-10 10:49:01,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1768d977-d229-4578-9569-5cbc149fcc40
[2017-05-10 10:49:01,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1768d977-d229-4578-9569-5cbc149fcc40
[2017-05-10 10:49:01,470: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:49:01,470: INFO/job_scheduler]   Job 114 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:49:01,471: INFO/job_scheduler]   Job 117 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:49:01,471: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-05-10 10:49:01,911: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:49:01,911: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:49:01,911: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:49:01,911: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:49:01,912: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c5befa15-5ebc-405a-9930-77e7943c91f5
[2017-05-10 10:49:01,912: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c5befa15-5ebc-405a-9930-77e7943c91f5
[2017-05-10 10:49:01,934: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c5befa15-5ebc-405a-9930-77e7943c91f5
[2017-05-10 10:49:01,934: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c5befa15-5ebc-405a-9930-77e7943c91f5
[2017-05-10 10:49:01,936: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-10 10:49:01,960: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e790cb61-f8d4-4c08-9ea3-2d14f7585d07
[2017-05-10 10:49:01,960: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e790cb61-f8d4-4c08-9ea3-2d14f7585d07
[2017-05-10 10:49:07,211: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:49:07,211: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:49:07,211: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:49:07,211: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:49:07,213: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 46c4f1c3-57e5-4ff6-ba0c-a3ee45c8b708
[2017-05-10 10:49:07,213: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 46c4f1c3-57e5-4ff6-ba0c-a3ee45c8b708
[2017-05-10 10:49:07,261: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 46c4f1c3-57e5-4ff6-ba0c-a3ee45c8b708
[2017-05-10 10:49:07,261: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 46c4f1c3-57e5-4ff6-ba0c-a3ee45c8b708
[2017-05-10 10:49:07,264: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-10 10:49:07,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f16b3b9d-4203-4559-a163-337ee0aa39e9
[2017-05-10 10:49:07,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f16b3b9d-4203-4559-a163-337ee0aa39e9
[2017-05-10 10:49:07,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:49:07,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:49:07,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:49:07,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:49:07,600: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1768d977-d229-4578-9569-5cbc149fcc40
[2017-05-10 10:49:07,600: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1768d977-d229-4578-9569-5cbc149fcc40
[2017-05-10 10:49:07,627: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1768d977-d229-4578-9569-5cbc149fcc40
[2017-05-10 10:49:07,627: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1768d977-d229-4578-9569-5cbc149fcc40
[2017-05-10 10:49:07,628: INFO/job_scheduler] Job 105 finished 1 steps successfully
[2017-05-10 10:49:07,817: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 140516567064080
[2017-05-10 10:49:07,894: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm8.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 10:49:07,919: INFO/job] Job 105: StateChangeJob complete, setting state started on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 10:49:07,919: INFO/job] Job 105: not_deleted=True
[2017-05-10 10:49:07,919: INFO/job_scheduler] Job 105: complete_job: Updating cache
[2017-05-10 10:49:07,920: INFO/job_scheduler] Job 105: locked_item 140516567064080 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 10:49:07,934: INFO/job_scheduler] Job 105 completing (errored=False, cancelled=False)
[2017-05-10 10:49:08,038: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-7
[2017-05-10 10:49:08,038: INFO/job_scheduler] _notify: Updating .mcast_port of item 7 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 52671
[2017-05-10 10:49:08,050: INFO/job_scheduler] _notify: Set mcast_port=52671 on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-7) and saved
[2017-05-10 10:49:08,051: INFO/job_scheduler] _notify: Updating .network_interfaces of item 7 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-05-10 10:49:08,075: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-7) and saved
[2017-05-10 10:49:08,089: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 7 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-10 10:49:08,103: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-7) and saved
[2017-05-10 10:49:08,127: INFO/job_scheduler] run_next: 1 runnable jobs of (7 pending, 3 tasked)
[2017-05-10 10:49:08,213: INFO/job_scheduler] Job 106: RunJobThread.run
[2017-05-10 10:49:08,217: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-05-10 10:49:08,293: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:49:08,293: INFO/job_scheduler]   Job 93 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:49:08,293: INFO/job_scheduler]   Job 94 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:49:08,294: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-05-10 10:49:08,309: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9ac81a49-18d3-49ef-a369-1249045ba6f3
[2017-05-10 10:49:08,309: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9ac81a49-18d3-49ef-a369-1249045ba6f3
[2017-05-10 10:49:08,341: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:49:08,341: INFO/job_scheduler]   Job 93 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:49:08,341: INFO/job_scheduler]   Job 95 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:49:08,341: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-05-10 10:49:08,387: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:49:08,387: INFO/job_scheduler]   Job 106 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:49:08,388: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-05-10 10:49:08,685: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-05-10 10:49:08,726: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-05-10 10:49:08,829: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:49:08,829: INFO/job_scheduler]   Job 82 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:49:08,829: INFO/job_scheduler]   Job 83 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:49:08,830: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-05-10 10:49:08,867: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:49:08,867: INFO/job_scheduler]   Job 82 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:49:08,867: INFO/job_scheduler]   Job 84 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:49:08,867: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-05-10 10:49:08,954: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:49:08,955: INFO/job_scheduler]   Job 114 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:49:08,955: INFO/job_scheduler]   Job 116 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:49:08,955: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-05-10 10:49:09,003: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:49:09,003: INFO/job_scheduler]   Job 114 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:49:09,003: INFO/job_scheduler]   Job 117 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:49:09,004: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-05-10 10:49:09,182: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:49:09,182: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:49:09,182: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:49:09,182: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:49:09,182: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e790cb61-f8d4-4c08-9ea3-2d14f7585d07
[2017-05-10 10:49:09,182: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e790cb61-f8d4-4c08-9ea3-2d14f7585d07
[2017-05-10 10:49:09,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e790cb61-f8d4-4c08-9ea3-2d14f7585d07
[2017-05-10 10:49:09,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e790cb61-f8d4-4c08-9ea3-2d14f7585d07
[2017-05-10 10:49:09,208: INFO/job_scheduler] Job 114 finished 3 steps successfully
[2017-05-10 10:49:09,259: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 140516567193040
[2017-05-10 10:49:09,266: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-10 10:49:09,276: INFO/job] Job 114: StateChangeJob complete, setting state stopped on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 10:49:09,277: INFO/job] Job 114: not_deleted=True
[2017-05-10 10:49:09,281: INFO/job_scheduler] Job 114: complete_job: Updating cache
[2017-05-10 10:49:09,281: INFO/job_scheduler] Job 114: locked_item 140516567193040 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 10:49:09,299: INFO/job_scheduler] Job 114 completing (errored=False, cancelled=False)
[2017-05-10 10:49:09,337: INFO/job_scheduler] run_next: 1 runnable jobs of (6 pending, 3 tasked)
[2017-05-10 10:49:09,367: INFO/job_scheduler] Job 116: RunJobThread.run
[2017-05-10 10:49:09,368: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 10:49:09,392: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e3c2b665-a3ba-4f67-a8dd-d9771f256a50
[2017-05-10 10:49:09,392: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e3c2b665-a3ba-4f67-a8dd-d9771f256a50
[2017-05-10 10:49:15,425: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:49:15,425: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:49:15,425: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:49:15,425: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:49:15,426: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f16b3b9d-4203-4559-a163-337ee0aa39e9
[2017-05-10 10:49:15,426: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f16b3b9d-4203-4559-a163-337ee0aa39e9
[2017-05-10 10:49:15,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f16b3b9d-4203-4559-a163-337ee0aa39e9
[2017-05-10 10:49:15,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f16b3b9d-4203-4559-a163-337ee0aa39e9
[2017-05-10 10:49:15,467: INFO/job_scheduler] Job 93 finished 3 steps successfully
[2017-05-10 10:49:15,509: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 140516566778320
[2017-05-10 10:49:15,530: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-10 10:49:15,559: INFO/job] Job 93: StateChangeJob complete, setting state stopped on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 10:49:15,559: INFO/job] Job 93: not_deleted=True
[2017-05-10 10:49:15,559: INFO/job_scheduler] Job 93: complete_job: Updating cache
[2017-05-10 10:49:15,560: INFO/job_scheduler] Job 93: locked_item 140516566778320 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 10:49:15,575: INFO/job_scheduler] Job 93 completing (errored=False, cancelled=False)
[2017-05-10 10:49:15,630: INFO/job_scheduler] run_next: 1 runnable jobs of (5 pending, 3 tasked)
[2017-05-10 10:49:15,673: INFO/job_scheduler] Job 94: RunJobThread.run
[2017-05-10 10:49:15,674: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 10:49:15,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 31f9f856-3769-4c2e-9b58-c9accec62c89
[2017-05-10 10:49:15,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 31f9f856-3769-4c2e-9b58-c9accec62c89
[2017-05-10 10:49:16,497: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:49:16,497: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:49:16,497: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:49:16,497: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:49:16,497: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e3c2b665-a3ba-4f67-a8dd-d9771f256a50
[2017-05-10 10:49:16,497: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e3c2b665-a3ba-4f67-a8dd-d9771f256a50
[2017-05-10 10:49:16,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e3c2b665-a3ba-4f67-a8dd-d9771f256a50
[2017-05-10 10:49:16,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e3c2b665-a3ba-4f67-a8dd-d9771f256a50
[2017-05-10 10:49:16,525: INFO/job_scheduler] Job 116 finished 1 steps successfully
[2017-05-10 10:49:16,574: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 140516567193040
[2017-05-10 10:49:16,606: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm6.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 10:49:16,615: INFO/job] Job 116: StateChangeJob complete, setting state started on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 10:49:16,615: INFO/job] Job 116: not_deleted=True
[2017-05-10 10:49:16,615: INFO/job_scheduler] Job 116: complete_job: Updating cache
[2017-05-10 10:49:16,616: INFO/job_scheduler] Job 116: locked_item 140516567193040 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 10:49:16,636: INFO/job_scheduler] Job 116 completing (errored=False, cancelled=False)
[2017-05-10 10:49:16,680: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:49:16,680: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:49:16,680: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:49:16,680: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:49:16,680: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b930eeb1-a7d3-466c-bd2a-12548866880c
[2017-05-10 10:49:16,680: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b930eeb1-a7d3-466c-bd2a-12548866880c
[2017-05-10 10:49:16,685: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-8
[2017-05-10 10:49:16,685: INFO/job_scheduler] _notify: Updating .mcast_port of item 8 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 48953
[2017-05-10 10:49:16,696: INFO/job_scheduler] _notify: Set mcast_port=48953 on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-8) and saved
[2017-05-10 10:49:16,699: INFO/job_scheduler] _notify: Updating .network_interfaces of item 8 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-05-10 10:49:16,724: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b930eeb1-a7d3-466c-bd2a-12548866880c
[2017-05-10 10:49:16,724: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b930eeb1-a7d3-466c-bd2a-12548866880c
[2017-05-10 10:49:16,726: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-10 10:49:16,752: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-8) and saved
[2017-05-10 10:49:16,760: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 8 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-10 10:49:16,767: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 54332e23-8731-4365-85ff-5ec3ee6a1916
[2017-05-10 10:49:16,767: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 54332e23-8731-4365-85ff-5ec3ee6a1916
[2017-05-10 10:49:16,779: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-8) and saved
[2017-05-10 10:49:16,821: INFO/job_scheduler] run_next: 1 runnable jobs of (4 pending, 3 tasked)
[2017-05-10 10:49:16,936: INFO/job_scheduler] Job 117: RunJobThread.run
[2017-05-10 10:49:16,938: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-05-10 10:49:17,000: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0ba7f9b3-d076-4fac-aa6b-768dfc2fbd53
[2017-05-10 10:49:17,000: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0ba7f9b3-d076-4fac-aa6b-768dfc2fbd53
[2017-05-10 10:49:18,639: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:49:18,639: INFO/job_scheduler]   Job 94 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:49:18,639: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 4 tasked)
[2017-05-10 10:49:18,656: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:49:18,657: INFO/job_scheduler]   Job 95 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:49:18,657: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 4 tasked)
[2017-05-10 10:49:18,683: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:49:18,683: INFO/job_scheduler]   Job 106 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:49:18,684: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 4 tasked)
[2017-05-10 10:49:18,704: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 4 tasked)
[2017-05-10 10:49:18,746: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 4 tasked)
[2017-05-10 10:49:18,782: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:49:18,782: INFO/job_scheduler]   Job 82 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:49:18,782: INFO/job_scheduler]   Job 83 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:49:18,782: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 4 tasked)
[2017-05-10 10:49:18,798: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:49:18,799: INFO/job_scheduler]   Job 82 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:49:18,799: INFO/job_scheduler]   Job 84 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:49:18,799: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 4 tasked)
[2017-05-10 10:49:18,818: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:49:18,819: INFO/job_scheduler]   Job 117 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:49:18,820: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 4 tasked)
[2017-05-10 10:49:21,318: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:49:21,318: INFO/job_scheduler]   Job 82 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:49:21,318: INFO/job_scheduler]   Job 83 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:49:21,318: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 4 tasked)
[2017-05-10 10:49:21,336: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:49:21,336: INFO/job_scheduler]   Job 82 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:49:21,336: INFO/job_scheduler]   Job 84 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:49:21,336: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 4 tasked)
[2017-05-10 10:49:21,358: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:49:21,359: INFO/job_scheduler]   Job 117 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:49:21,359: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 4 tasked)
[2017-05-10 10:49:22,809: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:49:22,809: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:49:22,810: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:49:22,810: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:49:22,810: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 31f9f856-3769-4c2e-9b58-c9accec62c89
[2017-05-10 10:49:22,810: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 31f9f856-3769-4c2e-9b58-c9accec62c89
[2017-05-10 10:49:22,831: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 31f9f856-3769-4c2e-9b58-c9accec62c89
[2017-05-10 10:49:22,831: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 31f9f856-3769-4c2e-9b58-c9accec62c89
[2017-05-10 10:49:22,831: INFO/job_scheduler] Job 94 finished 1 steps successfully
[2017-05-10 10:49:22,892: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 140516566778320
[2017-05-10 10:49:22,969: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 10:49:22,983: INFO/job] Job 94: StateChangeJob complete, setting state started on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 10:49:22,983: INFO/job] Job 94: not_deleted=True
[2017-05-10 10:49:22,983: INFO/job_scheduler] Job 94: complete_job: Updating cache
[2017-05-10 10:49:22,984: INFO/job_scheduler] Job 94: locked_item 140516566778320 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 10:49:23,016: INFO/job_scheduler] Job 94 completing (errored=False, cancelled=False)
[2017-05-10 10:49:23,044: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-6
[2017-05-10 10:49:23,044: INFO/job_scheduler] _notify: Updating .mcast_port of item 6 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 52671
[2017-05-10 10:49:23,051: INFO/job_scheduler] _notify: Set mcast_port=52671 on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-6) and saved
[2017-05-10 10:49:23,053: INFO/job_scheduler] _notify: Updating .network_interfaces of item 6 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-05-10 10:49:23,062: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-6) and saved
[2017-05-10 10:49:23,069: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 6 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-10 10:49:23,073: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-6) and saved
[2017-05-10 10:49:23,082: INFO/job_scheduler] run_next: 1 runnable jobs of (3 pending, 3 tasked)
[2017-05-10 10:49:23,124: INFO/job_scheduler] Job 95: RunJobThread.run
[2017-05-10 10:49:23,126: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-05-10 10:49:23,155: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 95c3bf43-0074-4cda-a24d-e1e78c5c5f59
[2017-05-10 10:49:23,155: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 95c3bf43-0074-4cda-a24d-e1e78c5c5f59
[2017-05-10 10:49:24,874: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:49:24,874: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:49:24,874: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:49:24,874: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:49:24,875: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 54332e23-8731-4365-85ff-5ec3ee6a1916
[2017-05-10 10:49:24,875: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 54332e23-8731-4365-85ff-5ec3ee6a1916
[2017-05-10 10:49:24,900: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 54332e23-8731-4365-85ff-5ec3ee6a1916
[2017-05-10 10:49:24,900: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 54332e23-8731-4365-85ff-5ec3ee6a1916
[2017-05-10 10:49:24,900: INFO/job_scheduler] Job 82 finished 3 steps successfully
[2017-05-10 10:49:24,941: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 140516567206992
[2017-05-10 10:49:24,950: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-10 10:49:24,968: INFO/job] Job 82: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 10:49:24,969: INFO/job] Job 82: not_deleted=True
[2017-05-10 10:49:24,969: INFO/job_scheduler] Job 82: complete_job: Updating cache
[2017-05-10 10:49:24,969: INFO/job_scheduler] Job 82: locked_item 140516567206992 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 10:49:24,972: INFO/job_scheduler] Job 82 completing (errored=False, cancelled=False)
[2017-05-10 10:49:24,992: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 3 tasked)
[2017-05-10 10:49:25,039: INFO/job_scheduler] Job 83: RunJobThread.run
[2017-05-10 10:49:25,040: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 10:49:25,072: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 36ed3220-9472-4bd0-9171-971c1b69c3d5
[2017-05-10 10:49:25,072: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 36ed3220-9472-4bd0-9171-971c1b69c3d5
[2017-05-10 10:49:28,297: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:49:28,298: INFO/job_scheduler]   Job 95 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:49:28,298: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 4 tasked)
[2017-05-10 10:49:28,320: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:49:28,321: INFO/job_scheduler]   Job 106 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:49:28,321: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 4 tasked)
[2017-05-10 10:49:28,733: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 4 tasked)
[2017-05-10 10:49:28,764: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 4 tasked)
[2017-05-10 10:49:28,845: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:49:28,845: INFO/job_scheduler]   Job 84 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:49:28,845: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 4 tasked)
[2017-05-10 10:49:28,901: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 8 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from True to False
[2017-05-10 10:49:28,914: INFO/job_scheduler] _notify: Set corosync_reported_up=False on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-8) and saved
[2017-05-10 10:49:28,933: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 4 tasked)
[2017-05-10 10:49:28,952: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:49:28,952: INFO/job_scheduler]   Job 117 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:49:28,952: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 4 tasked)
[2017-05-10 10:49:31,370: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 10:49:31,370: INFO/job_scheduler]   Job 83 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 10:49:31,371: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 4 tasked)
[2017-05-10 10:49:31,379: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:49:31,379: INFO/job_scheduler]   Job 84 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:49:31,379: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 4 tasked)
[2017-05-10 10:49:31,418: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 8 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-10 10:49:31,423: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-8) and saved
[2017-05-10 10:49:31,441: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 4 tasked)
[2017-05-10 10:49:31,450: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:49:31,450: INFO/job_scheduler]   Job 117 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:49:31,450: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 4 tasked)
[2017-05-10 10:49:32,205: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:49:32,205: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:49:32,206: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:49:32,206: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:49:32,206: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 36ed3220-9472-4bd0-9171-971c1b69c3d5
[2017-05-10 10:49:32,206: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 36ed3220-9472-4bd0-9171-971c1b69c3d5
[2017-05-10 10:49:32,207: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 36ed3220-9472-4bd0-9171-971c1b69c3d5
[2017-05-10 10:49:32,207: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 36ed3220-9472-4bd0-9171-971c1b69c3d5
[2017-05-10 10:49:32,208: INFO/job_scheduler] Job 83 finished 1 steps successfully
[2017-05-10 10:49:32,258: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 140516567206992
[2017-05-10 10:49:32,278: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 10:49:32,285: INFO/job] Job 83: StateChangeJob complete, setting state started on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 10:49:32,286: INFO/job] Job 83: not_deleted=True
[2017-05-10 10:49:32,286: INFO/job_scheduler] Job 83: complete_job: Updating cache
[2017-05-10 10:49:32,286: INFO/job_scheduler] Job 83: locked_item 140516567206992 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 10:49:32,289: INFO/job_scheduler] Job 83 completing (errored=False, cancelled=False)
[2017-05-10 10:49:32,335: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-5
[2017-05-10 10:49:32,336: INFO/job_scheduler] _notify: Updating .mcast_port of item 5 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 48953
[2017-05-10 10:49:32,359: INFO/job_scheduler] _notify: Set mcast_port=48953 on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-5) and saved
[2017-05-10 10:49:32,360: INFO/job_scheduler] _notify: Updating .network_interfaces of item 5 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-05-10 10:49:32,377: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-5) and saved
[2017-05-10 10:49:32,384: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 5 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-10 10:49:32,391: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-5) and saved
[2017-05-10 10:49:32,408: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 10:49:32,444: INFO/job_scheduler] Job 84: RunJobThread.run
[2017-05-10 10:49:32,446: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-05-10 10:49:32,464: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c45e9781-67b0-4fb1-baa4-33f2399d52fc
[2017-05-10 10:49:32,464: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c45e9781-67b0-4fb1-baa4-33f2399d52fc
[2017-05-10 10:49:33,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:49:33,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:49:33,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:49:33,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:49:33,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9ac81a49-18d3-49ef-a369-1249045ba6f3
[2017-05-10 10:49:33,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9ac81a49-18d3-49ef-a369-1249045ba6f3
[2017-05-10 10:49:33,659: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9ac81a49-18d3-49ef-a369-1249045ba6f3
[2017-05-10 10:49:33,659: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9ac81a49-18d3-49ef-a369-1249045ba6f3
[2017-05-10 10:49:33,661: INFO/job_scheduler] Job 106 finished 1 steps successfully
[2017-05-10 10:49:34,226: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 140516567191888
[2017-05-10 10:49:34,231: INFO/job] Job 106: StateChangeJob complete, setting state managed on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 10:49:34,231: INFO/job] Job 106: not_deleted=True
[2017-05-10 10:49:34,231: INFO/job_scheduler] Job 106: complete_job: Updating cache
[2017-05-10 10:49:34,231: INFO/job_scheduler] Job 106: locked_item 140516567191888 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 10:49:34,239: INFO/job_scheduler] Job 106 completing (errored=False, cancelled=False)
[2017-05-10 10:49:34,325: INFO/job_scheduler] Replaying 4 buffered notifications for ManagedHost-7
[2017-05-10 10:49:34,351: INFO/job_scheduler] _notify: Updating .immutable_state of item 7 (lotus-32vm8.lotus.hpdd.lab.intel.com) from True to False
[2017-05-10 10:49:34,367: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-7) and saved
[2017-05-10 10:49:34,425: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 7 (lotus-32vm8.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7fcc9011f590> to [6]
[2017-05-10 10:49:34,483: INFO/job_scheduler] _notify: Set ha_cluster_peers=[6] on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-7) and saved
[2017-05-10 10:49:34,672: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 10:49:38,766: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:49:38,766: INFO/job_scheduler]   Job 84 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:49:38,767: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 10:49:38,784: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:49:38,784: INFO/job_scheduler]   Job 117 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:49:38,784: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 10:49:41,384: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:49:41,385: INFO/job_scheduler]   Job 84 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:49:41,385: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 10:49:41,412: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 10:49:41,412: INFO/job_scheduler]   Job 117 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 10:49:41,412: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 10:49:42,419: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:49:42,419: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:49:42,419: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:49:42,419: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:49:42,420: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0ba7f9b3-d076-4fac-aa6b-768dfc2fbd53
[2017-05-10 10:49:42,420: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0ba7f9b3-d076-4fac-aa6b-768dfc2fbd53
[2017-05-10 10:49:42,459: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0ba7f9b3-d076-4fac-aa6b-768dfc2fbd53
[2017-05-10 10:49:42,459: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0ba7f9b3-d076-4fac-aa6b-768dfc2fbd53
[2017-05-10 10:49:42,461: INFO/job_scheduler] Job 117 finished 1 steps successfully
[2017-05-10 10:49:43,036: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 140516566813072
[2017-05-10 10:49:43,042: INFO/job] Job 117: StateChangeJob complete, setting state managed on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 10:49:43,042: INFO/job] Job 117: not_deleted=True
[2017-05-10 10:49:43,043: INFO/job_scheduler] Job 117: complete_job: Updating cache
[2017-05-10 10:49:43,043: INFO/job_scheduler] Job 117: locked_item 140516566813072 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 10:49:43,046: INFO/job_scheduler] Job 117 completing (errored=False, cancelled=False)
[2017-05-10 10:49:43,157: INFO/job_scheduler] Replaying 4 buffered notifications for ManagedHost-8
[2017-05-10 10:49:43,175: INFO/job_scheduler] _notify: Updating .immutable_state of item 8 (lotus-32vm6.lotus.hpdd.lab.intel.com) from True to False
[2017-05-10 10:49:43,195: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-8) and saved
[2017-05-10 10:49:43,284: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 8 (lotus-32vm6.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7fcc9c12ff50> to [5]
[2017-05-10 10:49:43,320: INFO/job_scheduler] _notify: Set ha_cluster_peers=[5] on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-8) and saved
[2017-05-10 10:49:43,566: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-05-10 10:49:49,555: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:49:49,555: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:49:49,555: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:49:49,555: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:49:49,555: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 95c3bf43-0074-4cda-a24d-e1e78c5c5f59
[2017-05-10 10:49:49,555: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 95c3bf43-0074-4cda-a24d-e1e78c5c5f59
[2017-05-10 10:49:49,596: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 95c3bf43-0074-4cda-a24d-e1e78c5c5f59
[2017-05-10 10:49:49,596: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 95c3bf43-0074-4cda-a24d-e1e78c5c5f59
[2017-05-10 10:49:49,598: INFO/job_scheduler] Job 95 finished 1 steps successfully
[2017-05-10 10:49:50,264: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 140516567183696
[2017-05-10 10:49:50,270: INFO/job] Job 95: StateChangeJob complete, setting state managed on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 10:49:50,270: INFO/job] Job 95: not_deleted=True
[2017-05-10 10:49:50,270: INFO/job_scheduler] Job 95: complete_job: Updating cache
[2017-05-10 10:49:50,270: INFO/job_scheduler] Job 95: locked_item 140516567183696 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 10:49:50,273: INFO/job_scheduler] Job 95 completing (errored=False, cancelled=False)
[2017-05-10 10:49:50,351: INFO/job_scheduler] Replaying 4 buffered notifications for ManagedHost-6
[2017-05-10 10:49:50,359: INFO/job_scheduler] _notify: Updating .immutable_state of item 6 (lotus-32vm7.lotus.hpdd.lab.intel.com) from True to False
[2017-05-10 10:49:50,366: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-6) and saved
[2017-05-10 10:49:50,406: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 6 (lotus-32vm7.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7fcc9011bd10> to [7]
[2017-05-10 10:49:50,418: INFO/job_scheduler] _notify: Set ha_cluster_peers=[7] on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-6) and saved
[2017-05-10 10:49:50,644: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 10:49:57,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:49:57,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:49:57,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:49:57,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:49:57,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c45e9781-67b0-4fb1-baa4-33f2399d52fc
[2017-05-10 10:49:57,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c45e9781-67b0-4fb1-baa4-33f2399d52fc
[2017-05-10 10:49:57,857: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c45e9781-67b0-4fb1-baa4-33f2399d52fc
[2017-05-10 10:49:57,857: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c45e9781-67b0-4fb1-baa4-33f2399d52fc
[2017-05-10 10:49:57,859: INFO/job_scheduler] Job 84 finished 1 steps successfully
[2017-05-10 10:49:58,588: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 140516566536400
[2017-05-10 10:49:58,600: INFO/job] Job 84: StateChangeJob complete, setting state managed on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 10:49:58,600: INFO/job] Job 84: not_deleted=True
[2017-05-10 10:49:58,600: INFO/job_scheduler] Job 84: complete_job: Updating cache
[2017-05-10 10:49:58,600: INFO/job_scheduler] Job 84: locked_item 140516566536400 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 10:49:58,611: INFO/job_scheduler] Job 84 completing (errored=False, cancelled=False)
[2017-05-10 10:49:58,720: INFO/job_scheduler] Replaying 4 buffered notifications for ManagedHost-5
[2017-05-10 10:49:58,735: INFO/job_scheduler] _notify: Updating .immutable_state of item 5 (lotus-32vm5.lotus.hpdd.lab.intel.com) from True to False
[2017-05-10 10:49:58,747: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-5) and saved
[2017-05-10 10:49:58,806: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 5 (lotus-32vm5.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7fcc9011bd10> to [8]
[2017-05-10 10:49:58,841: INFO/job_scheduler] _notify: Set ha_cluster_peers=[8] on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-5) and saved
[2017-05-10 10:49:58,999: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:51:50,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:51:50,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/afd17982-f810-4349-ac55-02ec76dd1ac7
[2017-05-10 10:51:50,124: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session afd17982-f810-4349-ac55-02ec76dd1ac7 because session terminated
[2017-05-10 10:51:50,124: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session afd17982-f810-4349-ac55-02ec76dd1ac7 because session terminated
[2017-05-10 10:52:00,146: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:52:00,146: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f02d14fc-0801-4cbd-aa98-a52c70b8da72
[2017-05-10 10:52:00,146: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session f02d14fc-0801-4cbd-aa98-a52c70b8da72 because session terminated
[2017-05-10 10:52:00,146: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session f02d14fc-0801-4cbd-aa98-a52c70b8da72 because session terminated
[2017-05-10 10:52:10,168: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:52:10,168: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/7b095c13-d1e8-43fc-b262-cfbaf4cbc360
[2017-05-10 10:52:10,168: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 7b095c13-d1e8-43fc-b262-cfbaf4cbc360 because session terminated
[2017-05-10 10:52:10,168: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 7b095c13-d1e8-43fc-b262-cfbaf4cbc360 because session terminated
[2017-05-10 10:52:10,196: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:52:10,196: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6d6bfc9a-0635-4f17-9b7e-404a263ecd6e
[2017-05-10 10:52:10,196: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 6d6bfc9a-0635-4f17-9b7e-404a263ecd6e because session terminated
[2017-05-10 10:52:10,196: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 6d6bfc9a-0635-4f17-9b7e-404a263ecd6e because session terminated
[2017-05-10 10:54:22,752: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:54:22,753: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 10:54:22,788: INFO/command_plan] add_jobs: created Job 118 (Test for host connectivity)
[2017-05-10 10:54:22,842: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 10:54:22,908: INFO/job_scheduler] Job 118: RunJobThread.run
[2017-05-10 10:54:22,964: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:54:22,965: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 10:54:22,980: INFO/command_plan] add_jobs: created Job 119 (Test for host connectivity)
[2017-05-10 10:54:23,010: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 10:54:23,045: INFO/job_scheduler] Job 119: RunJobThread.run
[2017-05-10 10:54:23,709: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:54:23,709: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 10:54:23,745: INFO/command_plan] add_jobs: created Job 120 (Test for host connectivity)
[2017-05-10 10:54:23,762: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 10:54:23,799: INFO/job_scheduler] Job 120: RunJobThread.run
[2017-05-10 10:54:23,993: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 10:54:23,993: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 10:54:24,045: INFO/command_plan] add_jobs: created Job 121 (Test for host connectivity)
[2017-05-10 10:54:24,099: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 10:54:24,111: INFO/job_scheduler] Job 121: RunJobThread.run
[2017-05-10 10:54:27,284: INFO/job_scheduler] Job 118 finished 1 steps successfully
[2017-05-10 10:54:27,345: INFO/job_scheduler] Job 118: complete_job: Updating cache
[2017-05-10 10:54:27,345: INFO/job_scheduler] Job 118 completing (errored=False, cancelled=False)
[2017-05-10 10:54:27,380: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 10:54:27,457: INFO/job_scheduler] Job 119 finished 1 steps successfully
[2017-05-10 10:54:27,520: INFO/job_scheduler] Job 119: complete_job: Updating cache
[2017-05-10 10:54:27,520: INFO/job_scheduler] Job 119 completing (errored=False, cancelled=False)
[2017-05-10 10:54:27,625: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-05-10 10:54:27,921: INFO/job_scheduler] Job 120 finished 1 steps successfully
[2017-05-10 10:54:27,954: INFO/job_scheduler] Job 120: complete_job: Updating cache
[2017-05-10 10:54:27,954: INFO/job_scheduler] Job 120 completing (errored=False, cancelled=False)
[2017-05-10 10:54:28,086: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 10:54:28,714: INFO/job_scheduler] Job 121 finished 1 steps successfully
[2017-05-10 10:54:28,755: INFO/job_scheduler] Job 121: complete_job: Updating cache
[2017-05-10 10:54:28,756: INFO/job_scheduler] Job 121 completing (errored=False, cancelled=False)
[2017-05-10 10:54:28,833: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 10:54:30,602: ERROR/rpc] RunOneRpc: exception calling create_host_ssh: Traceback (most recent call last):

  File "/usr/share/chroma-manager/chroma_core/services/rpc.py", line 141, in run

  File "/usr/share/chroma-manager/chroma_core/services/rpc.py", line 623, in _local_call

  File "/usr/share/chroma-manager/chroma_core/services/job_scheduler/job_scheduler.py", line 1224, in create_host_ssh

AssertionError

[2017-05-10 10:54:30,617: ERROR/rpc] RunOneRpc: exception calling create_host_ssh: Traceback (most recent call last):

  File "/usr/share/chroma-manager/chroma_core/services/rpc.py", line 141, in run

  File "/usr/share/chroma-manager/chroma_core/services/rpc.py", line 623, in _local_call

  File "/usr/share/chroma-manager/chroma_core/services/job_scheduler/job_scheduler.py", line 1224, in create_host_ssh

AssertionError

[2017-05-10 10:54:30,642: ERROR/rpc] RunOneRpc: exception calling create_host_ssh: Traceback (most recent call last):

  File "/usr/share/chroma-manager/chroma_core/services/rpc.py", line 141, in run

  File "/usr/share/chroma-manager/chroma_core/services/rpc.py", line 623, in _local_call

  File "/usr/share/chroma-manager/chroma_core/services/job_scheduler/job_scheduler.py", line 1224, in create_host_ssh

AssertionError

[2017-05-10 10:54:30,651: ERROR/rpc] RunOneRpc: exception calling create_host_ssh: Traceback (most recent call last):

  File "/usr/share/chroma-manager/chroma_core/services/rpc.py", line 141, in run

  File "/usr/share/chroma-manager/chroma_core/services/rpc.py", line 623, in _local_call

  File "/usr/share/chroma-manager/chroma_core/services/job_scheduler/job_scheduler.py", line 1224, in create_host_ssh

AssertionError

[2017-05-10 11:09:26,138: INFO/chroma_service] Stopping job_scheduler
[2017-05-10 11:09:26,138: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-05-10 11:09:26,138: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-05-10 11:09:26,138: INFO/queue] Stopping ServiceQueue agent_action_runner_rx
[2017-05-10 11:09:26,450: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-05-10 11:09:26,450: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-05-10 11:09:26,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-05-10 11:09:26,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-05-10 11:09:26,453: INFO/job_scheduler] Stopping...
[2017-05-10 11:09:26,453: INFO/queue] Stopping ServiceQueue job_scheduler_notifications
[2017-05-10 11:09:26,455: INFO/job_scheduler] Joining...
[2017-05-10 11:09:27,133: INFO/job_scheduler] Complete.
[2017-05-10 11:09:27,134: INFO/chroma_service] Joining job_scheduler
[2017-05-10 11:09:27,134: INFO/job_scheduler] Cancelling outstanding jobs...
[2017-05-10 11:09:27,137: ERROR/chroma_service] Rogue thread still running, exiting hard
[2017-05-10 11:09:27,137: ERROR/chroma_service] ['Thread-17311', 'Thread-17523', 'Thread-17720', 'Thread-17711', 'Thread-17709', 'Thread-17519', 'Thread-17694', 'Thread-17317', 'Thread-17520', 'Thread-17745', 'MainThread', 'Thread-17729', 'Thread-17318', 'Thread-17721', 'Thread-17314']
[2017-05-10 11:09:51,444: INFO/queue] Purged 0 messages from 'job_scheduler_notifications' queue
[2017-05-10 11:09:51,457: INFO/queue] Purged 1 messages from 'agent_action_runner_rx' queue
[2017-05-10 11:23:50,827: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 11:23:50,834: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 11:23:50,890: INFO/command_plan] add_jobs: created Job 122 (Force remove host lotus-32vm5.lotus.hpdd.lab.intel.com from configuration)
[2017-05-10 11:23:50,917: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 11:23:50,987: INFO/job_scheduler] Job 122: RunJobThread.run
[2017-05-10 11:23:51,039: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:23:51,039: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_up->removed (intentional=False) 139768973880848
[2017-05-10 11:23:51,058: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 11:23:51,087: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> 5
[2017-05-10 11:23:51,100: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:23:51,100: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration started->removed (intentional=False) 139768973927952
[2017-05-10 11:23:51,126: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-10 11:23:51,150: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.corosync2.Corosync2Configuration'> 5
[2017-05-10 11:23:51,176: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:23:51,177: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration configured->removed (intentional=False) 139768973880592
[2017-05-10 11:23:51,185: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> 5
[2017-05-10 11:23:51,200: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:23:51,200: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration configured->removed (intentional=False) 139768973927504
[2017-05-10 11:23:51,223: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.ntp.NTPConfiguration'> 5
[2017-05-10 11:23:51,249: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:23:51,249: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration started->removed (intentional=False) 139768973926992
[2017-05-10 11:23:51,274: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-10 11:23:51,294: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> 5
[2017-05-10 11:23:52,442: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.host.ManagedHost'> 5
[2017-05-10 11:23:52,482: INFO/job_scheduler] Job 122 finished 1 steps successfully
[2017-05-10 11:23:52,511: INFO/job_scheduler] Job 122: complete_job: Updating cache
[2017-05-10 11:23:52,511: INFO/job_scheduler] Job 122: locked_item 139768973945872 <class 'chroma_core.models.host.ManagedHost'> True None
[2017-05-10 11:23:52,511: INFO/job_scheduler] Job 122: locked_item 139768973934352 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 11:23:52,520: INFO/job_scheduler] Job 122: locked_item 139768973918928 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 11:23:52,528: INFO/job_scheduler] Job 122: locked_item 139768973931344 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-10 11:23:52,531: INFO/job_scheduler] Job 122: locked_item 139768973932944 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-10 11:23:52,534: INFO/job_scheduler] Job 122: locked_item 139768973930704 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 11:23:52,540: INFO/job_scheduler] Job 122 completing (errored=False, cancelled=False)
[2017-05-10 11:23:52,642: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 11:27:38,431: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 11:27:38,431: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 11:27:38,461: INFO/command_plan] add_jobs: created Job 123 (Test for host connectivity)
[2017-05-10 11:27:38,513: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 11:27:38,570: INFO/job_scheduler] Job 123: RunJobThread.run
[2017-05-10 11:27:42,312: INFO/job_scheduler] Job 123 finished 1 steps successfully
[2017-05-10 11:27:42,334: INFO/job_scheduler] Job 123: complete_job: Updating cache
[2017-05-10 11:27:42,335: INFO/job_scheduler] Job 123 completing (errored=False, cancelled=False)
[2017-05-10 11:27:42,394: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 11:27:42,606: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 11:27:42,607: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 11:27:42,634: INFO/command_plan] add_jobs: created Job 124 (Test for host connectivity)
[2017-05-10 11:27:42,672: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 11:27:42,678: INFO/job_scheduler] Job 124: RunJobThread.run
[2017-05-10 11:27:46,299: INFO/job_scheduler] Job 124 finished 1 steps successfully
[2017-05-10 11:27:46,324: INFO/job_scheduler] Job 124: complete_job: Updating cache
[2017-05-10 11:27:46,325: INFO/job_scheduler] Job 124 completing (errored=False, cancelled=False)
[2017-05-10 11:27:46,375: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 11:27:46,858: INFO/job_scheduler] Getting FQDN for 'lotus-32vm5': lotus-32vm5
lotus-32vm5.lotus.hpdd.lab.intel.com

[2017-05-10 11:27:46,883: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 11:27:46,887: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-9 transitioning from undeployed to unconfigured
[2017-05-10 11:27:46,968: INFO/command_plan] Created command 45 (Deploying agent to host lotus-32vm5.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-05-10 11:27:47,016: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 11:27:47,091: INFO/job_scheduler] Job 125: RunJobThread.run
[2017-05-10 11:27:48,122: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 11:27:49,885: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7584d6d1-262d-464a-8b46-624549f4733c
[2017-05-10 11:27:49,885: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7584d6d1-262d-464a-8b46-624549f4733c
[2017-05-10 11:27:49,892: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:27:49,892: INFO/job_scheduler]   Job 125 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-10 11:27:49,892: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 11:27:49,894: INFO/job_scheduler] Job 125 finished 1 steps successfully
[2017-05-10 11:27:49,957: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 139768570854416
[2017-05-10 11:27:49,964: INFO/job] Job 125: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:27:49,964: INFO/job] Job 125: not_deleted=True
[2017-05-10 11:27:49,964: INFO/job_scheduler] Job 125: complete_job: Updating cache
[2017-05-10 11:27:49,964: INFO/job_scheduler] Job 125: locked_item 139768570854416 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 11:27:49,967: INFO/job_scheduler] Job 125 completing (errored=False, cancelled=False)
[2017-05-10 11:27:50,043: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-9
[2017-05-10 11:27:50,043: INFO/job_scheduler] _notify: Updating .boot_time of item 9 (lotus-32vm5.lotus.hpdd.lab.intel.com) from None to 2017-05-03 16:42:07+00:00
[2017-05-10 11:27:50,051: INFO/job_scheduler] _notify: Set boot_time=2017-05-03 16:42:07+00:00 on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-9) and saved
[2017-05-10 11:27:50,101: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 11:27:58,414: INFO/job_scheduler] _notify: Updating .properties of item 9 (lotus-32vm5.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-05-10 11:27:58,424: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-9) and saved
[2017-05-10 11:27:58,429: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 11:27:58,581: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 11:28:41,007: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 11:28:41,023: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-9 transitioning from unconfigured to managed
[2017-05-10 11:28:41,243: INFO/command_plan] Created command 46 (Setup managed host lotus-32vm5.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-05-10 11:28:41,311: INFO/job_scheduler] run_next: 1 runnable jobs of (11 pending, 0 tasked)
[2017-05-10 11:28:41,334: INFO/job_scheduler] Job 126: RunJobThread.run
[2017-05-10 11:28:41,334: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-05-10 11:28:41,353: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3ee606ad-4de6-4e98-a8ce-98ba5f51329c
[2017-05-10 11:28:41,353: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3ee606ad-4de6-4e98-a8ce-98ba5f51329c
[2017-05-10 11:28:42,493: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7584d6d1-262d-464a-8b46-624549f4733c
[2017-05-10 11:28:42,493: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7584d6d1-262d-464a-8b46-624549f4733c
[2017-05-10 11:28:42,493: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/7584d6d1-262d-464a-8b46-624549f4733c
[2017-05-10 11:28:42,493: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/7584d6d1-262d-464a-8b46-624549f4733c
[2017-05-10 11:28:42,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3ee606ad-4de6-4e98-a8ce-98ba5f51329c
[2017-05-10 11:28:42,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3ee606ad-4de6-4e98-a8ce-98ba5f51329c
[2017-05-10 11:28:42,522: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3ee606ad-4de6-4e98-a8ce-98ba5f51329c
[2017-05-10 11:28:42,522: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3ee606ad-4de6-4e98-a8ce-98ba5f51329c
[2017-05-10 11:28:42,523: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-10 11:28:42.523057+00:00:
[2017-05-10 11:28:42,523: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-10 11:28:42.523057+00:00:
[2017-05-10 11:28:42,523: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 11:28:42,523: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 11:28:42,542: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:28:42,542: INFO/job_scheduler]   Job 126 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 11:28:42,543: INFO/job_scheduler]   Job 127 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:28:42,543: INFO/job_scheduler]   Job 128 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:28:42,543: INFO/job_scheduler]   Job 129 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:28:42,543: INFO/job_scheduler]   Job 130 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:28:42,543: INFO/job_scheduler]   Job 134 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:28:42,543: INFO/job_scheduler]   Job 136 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:28:42,544: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 1 tasked)
[2017-05-10 11:28:42,545: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-10 11:28:42.545860+00:00:
[2017-05-10 11:28:42,545: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-10 11:28:42.545860+00:00:
[2017-05-10 11:28:42,546: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 11:28:42,546: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 11:28:42,560: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:28:42,560: INFO/job_scheduler]   Job 126 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 11:28:42,560: INFO/job_scheduler]   Job 127 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:28:42,560: INFO/job_scheduler]   Job 128 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:28:42,560: INFO/job_scheduler]   Job 129 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:28:42,561: INFO/job_scheduler]   Job 130 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:28:42,561: INFO/job_scheduler]   Job 134 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:28:42,561: INFO/job_scheduler]   Job 136 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:28:42,561: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 1 tasked)
[2017-05-10 11:28:42,562: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com restart_agent {}
[2017-05-10 11:28:42,576: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d0ab4a1a-32a6-43d1-acd7-9f7187ff2105
[2017-05-10 11:28:42,576: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d0ab4a1a-32a6-43d1-acd7-9f7187ff2105
[2017-05-10 11:28:42,614: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7584d6d1-262d-464a-8b46-624549f4733c
[2017-05-10 11:28:42,614: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7584d6d1-262d-464a-8b46-624549f4733c
[2017-05-10 11:28:42,615: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/7584d6d1-262d-464a-8b46-624549f4733c
[2017-05-10 11:28:42,615: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/7584d6d1-262d-464a-8b46-624549f4733c
[2017-05-10 11:28:42,615: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d0ab4a1a-32a6-43d1-acd7-9f7187ff2105
[2017-05-10 11:28:42,615: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d0ab4a1a-32a6-43d1-acd7-9f7187ff2105
[2017-05-10 11:28:42,640: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d0ab4a1a-32a6-43d1-acd7-9f7187ff2105
[2017-05-10 11:28:42,640: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d0ab4a1a-32a6-43d1-acd7-9f7187ff2105
[2017-05-10 11:28:42,641: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old 7584d6d1-262d-464a-8b46-624549f4733c)
[2017-05-10 11:28:42,641: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old 7584d6d1-262d-464a-8b46-624549f4733c)
[2017-05-10 11:29:10,616: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7584d6d1-262d-464a-8b46-624549f4733c
[2017-05-10 11:29:10,616: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7584d6d1-262d-464a-8b46-624549f4733c
[2017-05-10 11:29:10,616: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 7584d6d1-262d-464a-8b46-624549f4733c because session terminated
[2017-05-10 11:29:10,616: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 7584d6d1-262d-464a-8b46-624549f4733c because session terminated
[2017-05-10 11:29:10,621: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:29:10,621: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:29:10,665: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new 7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:29:10,665: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new 7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:29:10,719: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 11:29:10,736: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 270017d2-df96-4695-b486-6150bcbcff2f
[2017-05-10 11:29:10,736: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 270017d2-df96-4695-b486-6150bcbcff2f
[2017-05-10 11:29:10,862: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:29:10,862: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:29:10,862: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:29:10,862: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:29:10,863: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 270017d2-df96-4695-b486-6150bcbcff2f
[2017-05-10 11:29:10,863: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 270017d2-df96-4695-b486-6150bcbcff2f
[2017-05-10 11:29:10,901: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 270017d2-df96-4695-b486-6150bcbcff2f
[2017-05-10 11:29:10,901: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 270017d2-df96-4695-b486-6150bcbcff2f
[2017-05-10 11:29:10,901: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-05-10 11:29:10,917: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 82abe1bc-dd23-4932-9ba3-a7ca70b41a01
[2017-05-10 11:29:10,917: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 82abe1bc-dd23-4932-9ba3-a7ca70b41a01
[2017-05-10 11:29:20,544: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:29:20,544: INFO/job_scheduler]   Job 126 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 11:29:20,544: INFO/job_scheduler]   Job 127 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:29:20,545: INFO/job_scheduler]   Job 128 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:29:20,545: INFO/job_scheduler]   Job 129 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:29:20,545: INFO/job_scheduler]   Job 130 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:29:20,545: INFO/job_scheduler]   Job 134 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:29:20,545: INFO/job_scheduler]   Job 136 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:29:20,545: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 1 tasked)
[2017-05-10 11:29:43,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:29:43,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:29:43,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:29:43,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:29:43,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 82abe1bc-dd23-4932-9ba3-a7ca70b41a01
[2017-05-10 11:29:43,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 82abe1bc-dd23-4932-9ba3-a7ca70b41a01
[2017-05-10 11:29:43,356: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 82abe1bc-dd23-4932-9ba3-a7ca70b41a01
[2017-05-10 11:29:43,356: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 82abe1bc-dd23-4932-9ba3-a7ca70b41a01
[2017-05-10 11:29:43,359: INFO/job] Job 126 LearnDevicesStep: No data for plugin linux from host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:29:43,710: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com install_packages {'repos': [u'lustre', u'e2fsprogs', u'zfs', u'iml-agent'], 'packages': [u'chroma-agent-management', u'kernel-devel-3.10.0-514.10.2.el7_lustre', u'lustre-osd-zfs', u'lustre-osd-ldiskfs', u'lustre-modules', u'lustre', u'zfs']}
[2017-05-10 11:29:43,726: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7dd45e74-f9b9-48d4-9490-26a942208203
[2017-05-10 11:29:43,726: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7dd45e74-f9b9-48d4-9490-26a942208203
[2017-05-10 11:30:02,234: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:02,234: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:02,234: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:02,234: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:02,234: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7dd45e74-f9b9-48d4-9490-26a942208203
[2017-05-10 11:30:02,234: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7dd45e74-f9b9-48d4-9490-26a942208203
[2017-05-10 11:30:02,249: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7dd45e74-f9b9-48d4-9490-26a942208203
[2017-05-10 11:30:02,249: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7dd45e74-f9b9-48d4-9490-26a942208203
[2017-05-10 11:30:02,666: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com kernel_status {}
[2017-05-10 11:30:02,692: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fc786b30-9b39-4712-a438-f65eb96d14dc
[2017-05-10 11:30:02,692: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fc786b30-9b39-4712-a438-f65eb96d14dc
[2017-05-10 11:30:04,785: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:04,785: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:04,786: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:04,786: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:04,786: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fc786b30-9b39-4712-a438-f65eb96d14dc
[2017-05-10 11:30:04,786: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fc786b30-9b39-4712-a438-f65eb96d14dc
[2017-05-10 11:30:04,808: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fc786b30-9b39-4712-a438-f65eb96d14dc
[2017-05-10 11:30:04,808: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fc786b30-9b39-4712-a438-f65eb96d14dc
[2017-05-10 11:30:04,809: INFO/job_scheduler] Job 126 finished 4 steps successfully
[2017-05-10 11:30:04,853: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 139768571017424
[2017-05-10 11:30:04,857: INFO/job] Job 126: StateChangeJob complete, setting state packages_installed on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:30:04,858: INFO/job] Job 126: not_deleted=True
[2017-05-10 11:30:04,858: INFO/job_scheduler] Job 126: complete_job: Updating cache
[2017-05-10 11:30:04,858: INFO/job_scheduler] Job 126: locked_item 139768570987216 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 11:30:04,861: INFO/job_scheduler] Job 126 completing (errored=False, cancelled=False)
[2017-05-10 11:30:04,980: INFO/job_scheduler] run_next: 4 runnable jobs of (10 pending, 0 tasked)
[2017-05-10 11:30:05,033: INFO/job_scheduler] Job 128: open DB connection during completion
[2017-05-10 11:30:05,078: INFO/job_scheduler] Job 129: RunJobThread.run
[2017-05-10 11:30:05,079: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-05-10 11:30:05,090: INFO/job_scheduler] Job 130: RunJobThread.run
[2017-05-10 11:30:05,091: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-05-10 11:30:05,092: INFO/job_scheduler] Job 127: RunJobThread.run
[2017-05-10 11:30:05,093: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 139768571018768
[2017-05-10 11:30:05,101: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-05-10 11:30:05,102: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 11:30:05,123: INFO/job] Job 128: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 11:30:05,123: INFO/job] Job 128: not_deleted=True
[2017-05-10 11:30:05,123: INFO/job_scheduler] Job 128: complete_job: Updating cache
[2017-05-10 11:30:05,123: INFO/job_scheduler] Job 128: locked_item 139768571018768 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 11:30:05,126: INFO/job_scheduler] Job 128 completing (errored=False, cancelled=False)
[2017-05-10 11:30:05,136: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 171d11ac-e9e7-4f43-bcea-5cec167b8d69
[2017-05-10 11:30:05,136: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 171d11ac-e9e7-4f43-bcea-5cec167b8d69
[2017-05-10 11:30:05,148: INFO/job_scheduler] run_next: 1 runnable jobs of (6 pending, 3 tasked)
[2017-05-10 11:30:05,162: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d3a8ce2e-ea37-4a0e-b556-6c7ec521c09b
[2017-05-10 11:30:05,162: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d3a8ce2e-ea37-4a0e-b556-6c7ec521c09b
[2017-05-10 11:30:05,172: INFO/job_scheduler] Job 131: RunJobThread.run
[2017-05-10 11:30:05,173: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com load_lnet {}
[2017-05-10 11:30:05,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 692b1c0c-6691-4418-8447-15dda2de0487
[2017-05-10 11:30:05,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 692b1c0c-6691-4418-8447-15dda2de0487
[2017-05-10 11:30:05,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f76367b9-cd2f-4464-90a0-d9e7d21e10e0
[2017-05-10 11:30:05,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f76367b9-cd2f-4464-90a0-d9e7d21e10e0
[2017-05-10 11:30:06,318: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:06,318: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:06,318: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:06,318: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:06,318: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f76367b9-cd2f-4464-90a0-d9e7d21e10e0
[2017-05-10 11:30:06,318: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f76367b9-cd2f-4464-90a0-d9e7d21e10e0
[2017-05-10 11:30:06,327: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f76367b9-cd2f-4464-90a0-d9e7d21e10e0
[2017-05-10 11:30:06,327: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f76367b9-cd2f-4464-90a0-d9e7d21e10e0
[2017-05-10 11:30:06,328: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 11:30:06,359: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4c802ba0-4583-4ae3-aae5-719a6bfc0f5d
[2017-05-10 11:30:06,359: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4c802ba0-4583-4ae3-aae5-719a6bfc0f5d
[2017-05-10 11:30:07,488: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:07,488: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:07,489: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:07,489: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:07,489: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4c802ba0-4583-4ae3-aae5-719a6bfc0f5d
[2017-05-10 11:30:07,489: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4c802ba0-4583-4ae3-aae5-719a6bfc0f5d
[2017-05-10 11:30:07,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4c802ba0-4583-4ae3-aae5-719a6bfc0f5d
[2017-05-10 11:30:07,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4c802ba0-4583-4ae3-aae5-719a6bfc0f5d
[2017-05-10 11:30:08,013: INFO/job_scheduler] Job 131 finished 2 steps successfully
[2017-05-10 11:30:08,029: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 139768571018768
[2017-05-10 11:30:08,035: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 11:30:08,038: INFO/job] Job 131: StateChangeJob complete, setting state lnet_down on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 11:30:08,039: INFO/job] Job 131: not_deleted=True
[2017-05-10 11:30:08,039: INFO/job_scheduler] Job 131: complete_job: Updating cache
[2017-05-10 11:30:08,039: INFO/job_scheduler] Job 131: locked_item 139768571018768 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 11:30:08,044: INFO/job_scheduler] Job 131 completing (errored=False, cancelled=False)
[2017-05-10 11:30:08,077: INFO/job_scheduler] run_next: 1 runnable jobs of (5 pending, 3 tasked)
[2017-05-10 11:30:08,106: INFO/job_scheduler] Job 133: RunJobThread.run
[2017-05-10 11:30:08,107: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_lnet {}
[2017-05-10 11:30:08,147: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 85eecdae-e7d8-4686-9171-4399074312f2
[2017-05-10 11:30:08,147: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 85eecdae-e7d8-4686-9171-4399074312f2
[2017-05-10 11:30:09,274: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:09,274: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:09,274: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:09,274: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:09,274: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 85eecdae-e7d8-4686-9171-4399074312f2
[2017-05-10 11:30:09,274: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 85eecdae-e7d8-4686-9171-4399074312f2
[2017-05-10 11:30:09,312: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 85eecdae-e7d8-4686-9171-4399074312f2
[2017-05-10 11:30:09,312: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 85eecdae-e7d8-4686-9171-4399074312f2
[2017-05-10 11:30:09,313: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 11:30:09,331: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f873f575-ec75-41d4-ac6a-e28200312278
[2017-05-10 11:30:09,331: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f873f575-ec75-41d4-ac6a-e28200312278
[2017-05-10 11:30:10,442: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:10,442: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:10,442: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:10,442: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:10,442: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f873f575-ec75-41d4-ac6a-e28200312278
[2017-05-10 11:30:10,442: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f873f575-ec75-41d4-ac6a-e28200312278
[2017-05-10 11:30:10,445: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f873f575-ec75-41d4-ac6a-e28200312278
[2017-05-10 11:30:10,445: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f873f575-ec75-41d4-ac6a-e28200312278
[2017-05-10 11:30:10,604: INFO/job_scheduler] Job 133 finished 2 steps successfully
[2017-05-10 11:30:10,621: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 139768571018768
[2017-05-10 11:30:10,627: INFO/job] Job 133: StateChangeJob complete, setting state lnet_up on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 11:30:10,628: INFO/job] Job 133: not_deleted=True
[2017-05-10 11:30:10,628: INFO/job_scheduler] Job 133: complete_job: Updating cache
[2017-05-10 11:30:10,628: INFO/job_scheduler] Job 133: locked_item 139768571018768 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 11:30:10,637: INFO/job_scheduler] Job 133 completing (errored=False, cancelled=False)
[2017-05-10 11:30:10,664: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 3 tasked)
[2017-05-10 11:30:11,339: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:11,339: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:11,339: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:11,339: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:11,340: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 692b1c0c-6691-4418-8447-15dda2de0487
[2017-05-10 11:30:11,340: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 692b1c0c-6691-4418-8447-15dda2de0487
[2017-05-10 11:30:11,371: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 692b1c0c-6691-4418-8447-15dda2de0487
[2017-05-10 11:30:11,371: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 692b1c0c-6691-4418-8447-15dda2de0487
[2017-05-10 11:30:11,371: INFO/job_scheduler] Job 127 finished 1 steps successfully
[2017-05-10 11:30:11,495: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 139768571017488
[2017-05-10 11:30:11,508: INFO/job] Job 127: StateChangeJob complete, setting state configured on lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-10 11:30:11,508: INFO/job] Job 127: not_deleted=True
[2017-05-10 11:30:11,508: INFO/job_scheduler] Job 127: complete_job: Updating cache
[2017-05-10 11:30:11,508: INFO/job_scheduler] Job 127: locked_item 139768571017488 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-10 11:30:11,511: INFO/job_scheduler] Job 127 completing (errored=False, cancelled=False)
[2017-05-10 11:30:11,640: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-05-10 11:30:12,340: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:12,340: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:12,340: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:12,340: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:30:12,341: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 171d11ac-e9e7-4f43-bcea-5cec167b8d69
[2017-05-10 11:30:12,341: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 171d11ac-e9e7-4f43-bcea-5cec167b8d69
[2017-05-10 11:30:12,357: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 171d11ac-e9e7-4f43-bcea-5cec167b8d69
[2017-05-10 11:30:12,357: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 171d11ac-e9e7-4f43-bcea-5cec167b8d69
[2017-05-10 11:30:12,358: INFO/job_scheduler] Job 129 finished 1 steps successfully
[2017-05-10 11:30:12,585: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 139768571018384
[2017-05-10 11:30:12,589: INFO/job] Job 129: StateChangeJob complete, setting state configured on lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-10 11:30:12,590: INFO/job] Job 129: not_deleted=True
[2017-05-10 11:30:12,590: INFO/job_scheduler] Job 129: complete_job: Updating cache
[2017-05-10 11:30:12,590: INFO/job_scheduler] Job 129: locked_item 139768571018384 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-10 11:30:12,592: INFO/job_scheduler] Job 129 completing (errored=False, cancelled=False)
[2017-05-10 11:30:12,603: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 1 tasked)
[2017-05-10 11:31:21,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:31:21,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/7647c1e6-c7e4-4964-beda-88157befe4ae
[2017-05-10 11:31:21,553: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 7647c1e6-c7e4-4964-beda-88157befe4ae because session terminated
[2017-05-10 11:31:21,553: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 7647c1e6-c7e4-4964-beda-88157befe4ae because session terminated
[2017-05-10 11:31:21,562: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d3a8ce2e-ea37-4a0e-b556-6c7ec521c09b
[2017-05-10 11:31:21,562: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d3a8ce2e-ea37-4a0e-b556-6c7ec521c09b
[2017-05-10 11:31:21,562: ERROR/job_scheduler] Job 130 step 0 encountered an agent error: Communications error with lotus-32vm5.lotus.hpdd.lab.intel.com because session terminated
[2017-05-10 11:31:21,613: INFO/job_scheduler] Job 130: complete_job: Updating cache
[2017-05-10 11:31:21,613: INFO/job_scheduler] Job 130: locked_item 139768973949840 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 11:31:21,621: INFO/job_scheduler] Job 130 completing (errored=True, cancelled=False)
[2017-05-10 11:31:21,647: INFO/job_scheduler] run_next: 1 runnable jobs of (4 pending, 0 tasked)
[2017-05-10 11:31:21,654: WARNING/job] DependOn not satisfied: lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration in state unconfigured, not one of ['stopped'] (preferred stopped)
[2017-05-10 11:31:21,659: WARNING/job] DependOn not satisfied: lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration in state unconfigured, not one of ['stopped'] (preferred stopped)
[2017-05-10 11:31:21,659: WARNING/job_scheduler] Job 132: cancelling because of failed dependency
[2017-05-10 11:31:21,659: INFO/job_scheduler] Job 132 completing (errored=False, cancelled=True)
[2017-05-10 11:31:21,690: INFO/job_scheduler] run_next: 1 runnable jobs of (3 pending, 0 tasked)
[2017-05-10 11:31:21,719: WARNING/job] DependOn not satisfied: lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration in state unconfigured, not one of ['started'] (preferred started)
[2017-05-10 11:31:21,733: WARNING/job] DependOn not satisfied: lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration in state unconfigured, not one of ['started'] (preferred started)
[2017-05-10 11:31:21,735: WARNING/job_scheduler] Job 134: cancelling because of failed dependency
[2017-05-10 11:31:21,735: INFO/job_scheduler] Job 134 completing (errored=False, cancelled=True)
[2017-05-10 11:31:21,745: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 0 tasked)
[2017-05-10 11:31:21,765: WARNING/job] DependOn not satisfied: lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration in state unconfigured, not one of ['started'] (preferred started)
[2017-05-10 11:31:21,785: WARNING/job] DependOn not satisfied: lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration in state unconfigured, not one of ['started'] (preferred started)
[2017-05-10 11:31:21,788: WARNING/job] DependOn not satisfied: lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration in state unconfigured, not one of ['stopped'] (preferred stopped)
[2017-05-10 11:31:21,788: WARNING/job_scheduler] Job 135: cancelling because of failed dependency
[2017-05-10 11:31:21,788: INFO/job_scheduler] Job 135 completing (errored=False, cancelled=True)
[2017-05-10 11:31:21,802: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 11:31:21,836: WARNING/job] DependOn not satisfied: lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration in state unconfigured, not one of ['started'] (preferred started)
[2017-05-10 11:31:21,871: WARNING/job] DependOn not satisfied: lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration in state unconfigured, not one of ['started'] (preferred started)
[2017-05-10 11:31:21,877: WARNING/job_scheduler] Job 136: cancelling because of failed dependency
[2017-05-10 11:31:21,877: INFO/job_scheduler] Job 136 completing (errored=False, cancelled=True)
[2017-05-10 11:31:21,939: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 11:34:14,156: INFO/chroma_service] Stopping job_scheduler
[2017-05-10 11:34:14,157: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-05-10 11:34:14,157: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-05-10 11:34:14,157: INFO/queue] Stopping ServiceQueue agent_action_runner_rx
[2017-05-10 11:34:14,738: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-05-10 11:34:14,738: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-05-10 11:34:14,739: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-05-10 11:34:14,739: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-05-10 11:34:14,739: INFO/job_scheduler] Stopping...
[2017-05-10 11:34:14,739: INFO/queue] Stopping ServiceQueue job_scheduler_notifications
[2017-05-10 11:34:14,739: INFO/job_scheduler] Joining...
[2017-05-10 11:34:14,989: INFO/job_scheduler] Complete.
[2017-05-10 11:34:14,989: INFO/chroma_service] Joining job_scheduler
[2017-05-10 11:34:14,989: INFO/job_scheduler] Cancelling outstanding jobs...
[2017-05-10 11:34:14,992: ERROR/chroma_service] Rogue thread still running, exiting hard
[2017-05-10 11:34:14,992: ERROR/chroma_service] ['Thread-8215', 'Thread-8184', 'Thread-8183', 'Thread-8185', 'Thread-8189', 'Thread-8216', 'Thread-8190', 'Thread-8211', 'MainThread']
[2017-05-10 11:36:03,689: INFO/queue] Purged 0 messages from 'job_scheduler_notifications' queue
[2017-05-10 11:36:03,721: INFO/queue] Purged 1 messages from 'agent_action_runner_rx' queue
[2017-05-10 11:39:35,365: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 11:39:35,366: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 11:39:35,385: INFO/command_plan] add_jobs: created Job 1 (Test for host connectivity)
[2017-05-10 11:39:35,401: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 11:39:35,407: INFO/job_scheduler] Job 1: RunJobThread.run
[2017-05-10 11:39:35,472: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 11:39:35,472: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 11:39:35,484: INFO/command_plan] add_jobs: created Job 2 (Test for host connectivity)
[2017-05-10 11:39:35,501: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 11:39:35,517: INFO/job_scheduler] Job 2: RunJobThread.run
[2017-05-10 11:39:35,545: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 11:39:35,546: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 11:39:35,568: INFO/command_plan] add_jobs: created Job 3 (Test for host connectivity)
[2017-05-10 11:39:35,615: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 11:39:35,617: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 11:39:35,626: INFO/command_plan] add_jobs: created Job 4 (Test for host connectivity)
[2017-05-10 11:39:35,635: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-05-10 11:39:35,637: INFO/job_scheduler] Job 3: RunJobThread.run
[2017-05-10 11:39:35,637: INFO/job_scheduler] Job 4: RunJobThread.run
[2017-05-10 11:39:35,659: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 11:39:39,184: INFO/job_scheduler] Job 1 finished 1 steps successfully
[2017-05-10 11:39:39,268: INFO/job_scheduler] Job 1: complete_job: Updating cache
[2017-05-10 11:39:39,268: INFO/job_scheduler] Job 1 completing (errored=False, cancelled=False)
[2017-05-10 11:39:39,441: INFO/job_scheduler] Job 2 finished 1 steps successfully
[2017-05-10 11:39:39,459: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 11:39:39,476: INFO/job_scheduler] Job 2: complete_job: Updating cache
[2017-05-10 11:39:39,476: INFO/job_scheduler] Job 2 completing (errored=False, cancelled=False)
[2017-05-10 11:39:39,509: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-05-10 11:39:39,511: INFO/job_scheduler] Job 3 finished 1 steps successfully
[2017-05-10 11:39:39,518: INFO/job_scheduler] Job 3: complete_job: Updating cache
[2017-05-10 11:39:39,518: INFO/job_scheduler] Job 3 completing (errored=False, cancelled=False)
[2017-05-10 11:39:39,551: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 11:39:39,616: INFO/job_scheduler] Job 4 finished 1 steps successfully
[2017-05-10 11:39:39,676: INFO/job_scheduler] Job 4: complete_job: Updating cache
[2017-05-10 11:39:39,676: INFO/job_scheduler] Job 4 completing (errored=False, cancelled=False)
[2017-05-10 11:39:39,709: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 11:39:43,289: INFO/job_scheduler] Getting FQDN for 'lotus-32vm5': lotus-32vm5
lotus-32vm5.lotus.hpdd.lab.intel.com

[2017-05-10 11:39:43,333: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 11:39:43,338: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-1 transitioning from undeployed to unconfigured
[2017-05-10 11:39:43,359: INFO/command_plan] Created command 5 (Deploying agent to host lotus-32vm5.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-05-10 11:39:43,398: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 11:39:43,427: INFO/job_scheduler] Job 5: RunJobThread.run
[2017-05-10 11:39:43,888: INFO/job_scheduler] Getting FQDN for 'lotus-32vm6.lotus.hpdd.lab.intel.com': lotus-32vm6
lotus-32vm6.lotus.hpdd.lab.intel.com

[2017-05-10 11:39:43,934: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 11:39:43,936: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-2 transitioning from undeployed to unconfigured
[2017-05-10 11:39:43,959: INFO/command_plan] Created command 6 (Deploying agent to host lotus-32vm6.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-05-10 11:39:44,008: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 11:39:44,044: INFO/job_scheduler] Job 6: RunJobThread.run
[2017-05-10 11:39:44,542: INFO/job_scheduler] Getting FQDN for 'lotus-32vm7.lotus.hpdd.lab.intel.com': lotus-32vm7
lotus-32vm7.lotus.hpdd.lab.intel.com

[2017-05-10 11:39:44,575: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 11:39:44,579: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-3 transitioning from undeployed to unconfigured
[2017-05-10 11:39:44,603: INFO/command_plan] Created command 7 (Deploying agent to host lotus-32vm7.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-05-10 11:39:44,649: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 11:39:44,687: INFO/job_scheduler] Job 7: RunJobThread.run
[2017-05-10 11:39:44,699: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 11:39:44,736: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 11:39:45,268: INFO/job_scheduler] Getting FQDN for 'lotus-32vm8.lotus.hpdd.lab.intel.com': lotus-32vm8
lotus-32vm8.lotus.hpdd.lab.intel.com

[2017-05-10 11:39:45,301: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 11:39:45,305: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-4 transitioning from undeployed to unconfigured
[2017-05-10 11:39:45,327: INFO/command_plan] Created command 8 (Deploying agent to host lotus-32vm8.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-05-10 11:39:45,367: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 11:39:45,412: INFO/job_scheduler] Job 8: RunJobThread.run
[2017-05-10 11:39:45,498: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 11:39:46,032: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 11:39:46,433: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:39:46,433: INFO/job_scheduler]   Job 6 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-10 11:39:46,433: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 11:39:46,463: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-10 11:39:46,463: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-10 11:39:46,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/f5b67c69-2b19-49db-b10b-eb8e48cfebfa
[2017-05-10 11:39:46,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/f5b67c69-2b19-49db-b10b-eb8e48cfebfa
[2017-05-10 11:39:47,120: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:39:47,120: INFO/job_scheduler]   Job 7 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-10 11:39:47,120: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 11:39:47,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/29af1680-4488-4ada-84be-651b007b2840
[2017-05-10 11:39:47,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/29af1680-4488-4ada-84be-651b007b2840
[2017-05-10 11:39:47,129: INFO/job_scheduler] Job 7 finished 1 steps successfully
[2017-05-10 11:39:47,183: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 139877219955728
[2017-05-10 11:39:47,190: INFO/job] Job 7: StateChangeJob complete, setting state unconfigured on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 11:39:47,190: INFO/job] Job 7: not_deleted=True
[2017-05-10 11:39:47,190: INFO/job_scheduler] Job 7: complete_job: Updating cache
[2017-05-10 11:39:47,190: INFO/job_scheduler] Job 7: locked_item 139877219955728 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 11:39:47,193: INFO/job_scheduler] Job 7 completing (errored=False, cancelled=False)
[2017-05-10 11:39:47,235: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-3
[2017-05-10 11:39:47,235: INFO/job_scheduler] _notify: Updating .boot_time of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com) from None to 2017-04-25 07:09:31+00:00
[2017-05-10 11:39:47,240: INFO/job_scheduler] _notify: Set boot_time=2017-04-25 07:09:31+00:00 on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-3) and saved
[2017-05-10 11:39:47,248: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 11:39:47,440: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:39:47,440: INFO/job_scheduler]   Job 8 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-10 11:39:47,441: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 11:39:47,460: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-10 11:39:47,460: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm8.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-10 11:39:47,464: INFO/job_scheduler] Job 6 finished 1 steps successfully
[2017-05-10 11:39:47,468: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/b09fa0eb-e6af-4b39-82b2-b5a1a56266c9
[2017-05-10 11:39:47,468: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/b09fa0eb-e6af-4b39-82b2-b5a1a56266c9
[2017-05-10 11:39:47,609: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 139877219960848
[2017-05-10 11:39:47,614: INFO/job] Job 6: StateChangeJob complete, setting state unconfigured on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 11:39:47,615: INFO/job] Job 6: not_deleted=True
[2017-05-10 11:39:47,615: INFO/job_scheduler] Job 6: complete_job: Updating cache
[2017-05-10 11:39:47,615: INFO/job_scheduler] Job 6: locked_item 139877219960848 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 11:39:47,618: INFO/job_scheduler] Job 6 completing (errored=False, cancelled=False)
[2017-05-10 11:39:47,678: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-2
[2017-05-10 11:39:47,678: INFO/job_scheduler] _notify: Updating .boot_time of item 2 (lotus-32vm6.lotus.hpdd.lab.intel.com) from None to 2017-04-25 07:09:19+00:00
[2017-05-10 11:39:47,684: INFO/job_scheduler] _notify: Set boot_time=2017-04-25 07:09:19+00:00 on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-2) and saved
[2017-05-10 11:39:47,695: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-05-10 11:39:48,460: INFO/job_scheduler] Job 8 finished 1 steps successfully
[2017-05-10 11:39:48,534: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 139877220130576
[2017-05-10 11:39:48,540: INFO/job] Job 8: StateChangeJob complete, setting state unconfigured on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 11:39:48,540: INFO/job] Job 8: not_deleted=True
[2017-05-10 11:39:48,540: INFO/job_scheduler] Job 8: complete_job: Updating cache
[2017-05-10 11:39:48,540: INFO/job_scheduler] Job 8: locked_item 139877220130576 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 11:39:48,544: INFO/job_scheduler] Job 8 completing (errored=False, cancelled=False)
[2017-05-10 11:39:48,586: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-4
[2017-05-10 11:39:48,586: INFO/job_scheduler] _notify: Updating .boot_time of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com) from None to 2017-04-25 07:09:56+00:00
[2017-05-10 11:39:48,592: INFO/job_scheduler] _notify: Set boot_time=2017-04-25 07:09:56+00:00 on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-4) and saved
[2017-05-10 11:39:48,599: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 11:39:49,015: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:39:49,016: INFO/job_scheduler]   Job 5 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-10 11:39:49,016: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 11:39:49,063: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/587f5840-0a36-43b8-85ab-da19a13c3973
[2017-05-10 11:39:49,063: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/587f5840-0a36-43b8-85ab-da19a13c3973
[2017-05-10 11:39:49,122: INFO/job_scheduler] Job 5 finished 1 steps successfully
[2017-05-10 11:39:49,142: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 139877220104016
[2017-05-10 11:39:49,149: INFO/job] Job 5: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:39:49,149: INFO/job] Job 5: not_deleted=True
[2017-05-10 11:39:49,149: INFO/job_scheduler] Job 5: complete_job: Updating cache
[2017-05-10 11:39:49,149: INFO/job_scheduler] Job 5: locked_item 139877220104016 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 11:39:49,153: INFO/job_scheduler] Job 5 completing (errored=False, cancelled=False)
[2017-05-10 11:39:49,194: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-1
[2017-05-10 11:39:49,194: INFO/job_scheduler] _notify: Updating .boot_time of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com) from None to 2017-05-03 16:42:07+00:00
[2017-05-10 11:39:49,200: INFO/job_scheduler] _notify: Set boot_time=2017-05-03 16:42:07+00:00 on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-1) and saved
[2017-05-10 11:39:49,206: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 11:39:56,715: INFO/job_scheduler] _notify: Updating .properties of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-05-10 11:39:56,727: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-3) and saved
[2017-05-10 11:39:56,733: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 11:39:56,857: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 11:39:57,021: INFO/job_scheduler] _notify: Updating .properties of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-05-10 11:39:57,026: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-4) and saved
[2017-05-10 11:39:57,034: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 11:39:57,137: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 11:39:57,220: INFO/job_scheduler] _notify: Updating .properties of item 2 (lotus-32vm6.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-05-10 11:39:57,226: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-2) and saved
[2017-05-10 11:39:57,234: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 11:39:57,323: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 11:39:58,593: INFO/job_scheduler] _notify: Updating .properties of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-05-10 11:39:58,601: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-1) and saved
[2017-05-10 11:39:58,606: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 11:39:58,711: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 11:40:01,918: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 11:40:01,921: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-1 transitioning from unconfigured to managed
[2017-05-10 11:40:02,094: INFO/command_plan] Created command 9 (Setup managed host lotus-32vm5.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-05-10 11:40:02,140: INFO/job_scheduler] run_next: 1 runnable jobs of (11 pending, 0 tasked)
[2017-05-10 11:40:02,168: INFO/job_scheduler] Job 9: RunJobThread.run
[2017-05-10 11:40:02,168: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-05-10 11:40:02,186: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8dc497ee-e931-4f4a-9a7a-2bf4a061baf1
[2017-05-10 11:40:02,186: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8dc497ee-e931-4f4a-9a7a-2bf4a061baf1
[2017-05-10 11:40:02,248: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 11:40:02,254: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-2 transitioning from unconfigured to managed
[2017-05-10 11:40:02,357: INFO/command_plan] Created command 10 (Setup managed host lotus-32vm6.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-05-10 11:40:02,429: INFO/job_scheduler] run_next: 1 runnable jobs of (21 pending, 1 tasked)
[2017-05-10 11:40:02,449: INFO/job_scheduler] Job 20: RunJobThread.run
[2017-05-10 11:40:02,451: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-05-10 11:40:02,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bdd4db62-6833-4011-a7f2-e66a55f49d83
[2017-05-10 11:40:02,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bdd4db62-6833-4011-a7f2-e66a55f49d83
[2017-05-10 11:40:02,476: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 11:40:02,482: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-3 transitioning from unconfigured to managed
[2017-05-10 11:40:02,584: INFO/command_plan] Created command 11 (Setup managed host lotus-32vm7.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-05-10 11:40:02,654: INFO/job_scheduler] run_next: 1 runnable jobs of (31 pending, 2 tasked)
[2017-05-10 11:40:02,675: INFO/job_scheduler] Job 31: RunJobThread.run
[2017-05-10 11:40:02,677: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-05-10 11:40:02,697: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d3057b3a-4c75-4553-9828-fd2af0509d72
[2017-05-10 11:40:02,697: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d3057b3a-4c75-4553-9828-fd2af0509d72
[2017-05-10 11:40:02,698: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 11:40:02,704: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-4 transitioning from unconfigured to managed
[2017-05-10 11:40:02,808: INFO/command_plan] Created command 12 (Setup managed host lotus-32vm8.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-05-10 11:40:02,822: INFO/job_scheduler] run_next: 1 runnable jobs of (41 pending, 3 tasked)
[2017-05-10 11:40:02,840: INFO/job_scheduler] Job 42: RunJobThread.run
[2017-05-10 11:40:02,841: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-05-10 11:40:02,854: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 89984a6d-c7fe-4bc8-8494-2176ef3111d2
[2017-05-10 11:40:02,854: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 89984a6d-c7fe-4bc8-8494-2176ef3111d2
[2017-05-10 11:40:03,548: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/f5b67c69-2b19-49db-b10b-eb8e48cfebfa
[2017-05-10 11:40:03,548: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/f5b67c69-2b19-49db-b10b-eb8e48cfebfa
[2017-05-10 11:40:03,548: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/f5b67c69-2b19-49db-b10b-eb8e48cfebfa
[2017-05-10 11:40:03,548: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/f5b67c69-2b19-49db-b10b-eb8e48cfebfa
[2017-05-10 11:40:03,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bdd4db62-6833-4011-a7f2-e66a55f49d83
[2017-05-10 11:40:03,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bdd4db62-6833-4011-a7f2-e66a55f49d83
[2017-05-10 11:40:03,581: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bdd4db62-6833-4011-a7f2-e66a55f49d83
[2017-05-10 11:40:03,581: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bdd4db62-6833-4011-a7f2-e66a55f49d83
[2017-05-10 11:40:03,581: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-05-10 11:40:03.581509+00:00:
[2017-05-10 11:40:03,581: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-05-10 11:40:03.581509+00:00:
[2017-05-10 11:40:03,581: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 11:40:03,581: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 11:40:03,602: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:40:03,602: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-05-10 11:40:03.602718+00:00:
[2017-05-10 11:40:03,603: INFO/job_scheduler]   Job 20 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 11:40:03,602: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-05-10 11:40:03.602718+00:00:
[2017-05-10 11:40:03,603: INFO/job_scheduler]   Job 21 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:03,604: INFO/job_scheduler]   Job 22 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:03,603: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 11:40:03,603: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 11:40:03,604: INFO/job_scheduler]   Job 23 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:03,613: INFO/job_scheduler]   Job 24 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:03,614: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:03,614: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:40:03,615: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 11:40:03,622: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:40:03,622: INFO/job_scheduler]   Job 20 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 11:40:03,622: INFO/job_scheduler]   Job 21 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:03,622: INFO/job_scheduler]   Job 22 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:03,622: INFO/job_scheduler]   Job 23 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:03,623: INFO/job_scheduler]   Job 24 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:03,623: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:03,623: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:40:03,624: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 11:40:03,626: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com restart_agent {}
[2017-05-10 11:40:03,640: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b4e67fc1-e487-4d20-a4ba-a7f30b44015b
[2017-05-10 11:40:03,640: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b4e67fc1-e487-4d20-a4ba-a7f30b44015b
[2017-05-10 11:40:03,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/f5b67c69-2b19-49db-b10b-eb8e48cfebfa
[2017-05-10 11:40:03,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/f5b67c69-2b19-49db-b10b-eb8e48cfebfa
[2017-05-10 11:40:03,720: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/f5b67c69-2b19-49db-b10b-eb8e48cfebfa
[2017-05-10 11:40:03,720: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/f5b67c69-2b19-49db-b10b-eb8e48cfebfa
[2017-05-10 11:40:03,720: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b4e67fc1-e487-4d20-a4ba-a7f30b44015b
[2017-05-10 11:40:03,720: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b4e67fc1-e487-4d20-a4ba-a7f30b44015b
[2017-05-10 11:40:03,754: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/29af1680-4488-4ada-84be-651b007b2840
[2017-05-10 11:40:03,754: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/29af1680-4488-4ada-84be-651b007b2840
[2017-05-10 11:40:03,754: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b4e67fc1-e487-4d20-a4ba-a7f30b44015b
[2017-05-10 11:40:03,755: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/29af1680-4488-4ada-84be-651b007b2840
[2017-05-10 11:40:03,754: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b4e67fc1-e487-4d20-a4ba-a7f30b44015b
[2017-05-10 11:40:03,756: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm6.lotus.hpdd.lab.intel.com (old f5b67c69-2b19-49db-b10b-eb8e48cfebfa)
[2017-05-10 11:40:03,755: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/29af1680-4488-4ada-84be-651b007b2840
[2017-05-10 11:40:03,756: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm6.lotus.hpdd.lab.intel.com (old f5b67c69-2b19-49db-b10b-eb8e48cfebfa)
[2017-05-10 11:40:03,756: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d3057b3a-4c75-4553-9828-fd2af0509d72
[2017-05-10 11:40:03,756: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d3057b3a-4c75-4553-9828-fd2af0509d72
[2017-05-10 11:40:03,761: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d3057b3a-4c75-4553-9828-fd2af0509d72
[2017-05-10 11:40:03,761: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d3057b3a-4c75-4553-9828-fd2af0509d72
[2017-05-10 11:40:03,761: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-05-10 11:40:03.761588+00:00:
[2017-05-10 11:40:03,761: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-05-10 11:40:03.761588+00:00:
[2017-05-10 11:40:03,761: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 11:40:03,761: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 11:40:03,775: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:40:03,775: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-05-10 11:40:03.775636+00:00:
[2017-05-10 11:40:03,775: INFO/job_scheduler]   Job 31 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 11:40:03,775: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-05-10 11:40:03.775636+00:00:
[2017-05-10 11:40:03,776: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 11:40:03,776: INFO/job_scheduler]   Job 32 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:03,776: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 11:40:03,777: INFO/job_scheduler]   Job 33 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:03,789: INFO/job_scheduler]   Job 34 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:03,790: INFO/job_scheduler]   Job 35 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:03,792: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:03,793: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:40:03,794: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 11:40:03,798: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:40:03,798: INFO/job_scheduler]   Job 31 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 11:40:03,798: INFO/job_scheduler]   Job 32 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:03,798: INFO/job_scheduler]   Job 33 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:03,799: INFO/job_scheduler]   Job 34 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:03,799: INFO/job_scheduler]   Job 35 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:03,799: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:03,799: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:40:03,799: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 11:40:03,801: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com restart_agent {}
[2017-05-10 11:40:03,813: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f35a0030-fe83-4340-833b-56bb580f3d5b
[2017-05-10 11:40:03,813: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f35a0030-fe83-4340-833b-56bb580f3d5b
[2017-05-10 11:40:03,878: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/29af1680-4488-4ada-84be-651b007b2840
[2017-05-10 11:40:03,878: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/29af1680-4488-4ada-84be-651b007b2840
[2017-05-10 11:40:03,878: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/29af1680-4488-4ada-84be-651b007b2840
[2017-05-10 11:40:03,878: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/29af1680-4488-4ada-84be-651b007b2840
[2017-05-10 11:40:03,879: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f35a0030-fe83-4340-833b-56bb580f3d5b
[2017-05-10 11:40:03,879: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f35a0030-fe83-4340-833b-56bb580f3d5b
[2017-05-10 11:40:03,879: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f35a0030-fe83-4340-833b-56bb580f3d5b
[2017-05-10 11:40:03,879: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f35a0030-fe83-4340-833b-56bb580f3d5b
[2017-05-10 11:40:03,880: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm7.lotus.hpdd.lab.intel.com (old 29af1680-4488-4ada-84be-651b007b2840)
[2017-05-10 11:40:03,880: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm7.lotus.hpdd.lab.intel.com (old 29af1680-4488-4ada-84be-651b007b2840)
[2017-05-10 11:40:03,946: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/b09fa0eb-e6af-4b39-82b2-b5a1a56266c9
[2017-05-10 11:40:03,946: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/b09fa0eb-e6af-4b39-82b2-b5a1a56266c9
[2017-05-10 11:40:03,946: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/b09fa0eb-e6af-4b39-82b2-b5a1a56266c9
[2017-05-10 11:40:03,946: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/b09fa0eb-e6af-4b39-82b2-b5a1a56266c9
[2017-05-10 11:40:03,946: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 89984a6d-c7fe-4bc8-8494-2176ef3111d2
[2017-05-10 11:40:03,946: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 89984a6d-c7fe-4bc8-8494-2176ef3111d2
[2017-05-10 11:40:03,968: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 89984a6d-c7fe-4bc8-8494-2176ef3111d2
[2017-05-10 11:40:03,968: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 89984a6d-c7fe-4bc8-8494-2176ef3111d2
[2017-05-10 11:40:03,969: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-05-10 11:40:03.969449+00:00:
[2017-05-10 11:40:03,969: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-05-10 11:40:03.969449+00:00:
[2017-05-10 11:40:03,969: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 11:40:03,969: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 11:40:03,989: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:40:03,989: INFO/job_scheduler]   Job 42 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 11:40:03,990: INFO/job_scheduler]   Job 43 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:03,990: INFO/job_scheduler]   Job 44 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:03,990: INFO/job_scheduler]   Job 45 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:03,990: INFO/job_scheduler]   Job 46 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:03,990: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:03,990: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:40:03,991: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 11:40:03,993: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-05-10 11:40:03.993499+00:00:
[2017-05-10 11:40:03,993: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-05-10 11:40:03.993499+00:00:
[2017-05-10 11:40:03,993: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 11:40:03,993: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 11:40:04,010: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com restart_agent {}
[2017-05-10 11:40:04,023: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:40:04,023: INFO/job_scheduler]   Job 42 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 11:40:04,024: INFO/job_scheduler]   Job 43 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:04,024: INFO/job_scheduler]   Job 44 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:04,024: INFO/job_scheduler]   Job 45 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:04,024: INFO/job_scheduler]   Job 46 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:04,024: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:04,025: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:40:04,026: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 11:40:04,033: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc be9b42c0-44c3-44b0-afe9-90594f979e51
[2017-05-10 11:40:04,033: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc be9b42c0-44c3-44b0-afe9-90594f979e51
[2017-05-10 11:40:04,117: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/b09fa0eb-e6af-4b39-82b2-b5a1a56266c9
[2017-05-10 11:40:04,117: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/b09fa0eb-e6af-4b39-82b2-b5a1a56266c9
[2017-05-10 11:40:04,118: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/b09fa0eb-e6af-4b39-82b2-b5a1a56266c9
[2017-05-10 11:40:04,118: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/b09fa0eb-e6af-4b39-82b2-b5a1a56266c9
[2017-05-10 11:40:04,118: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc be9b42c0-44c3-44b0-afe9-90594f979e51
[2017-05-10 11:40:04,118: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc be9b42c0-44c3-44b0-afe9-90594f979e51
[2017-05-10 11:40:04,147: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc be9b42c0-44c3-44b0-afe9-90594f979e51
[2017-05-10 11:40:04,147: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc be9b42c0-44c3-44b0-afe9-90594f979e51
[2017-05-10 11:40:04,148: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm8.lotus.hpdd.lab.intel.com (old b09fa0eb-e6af-4b39-82b2-b5a1a56266c9)
[2017-05-10 11:40:04,148: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm8.lotus.hpdd.lab.intel.com (old b09fa0eb-e6af-4b39-82b2-b5a1a56266c9)
[2017-05-10 11:40:07,290: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/587f5840-0a36-43b8-85ab-da19a13c3973
[2017-05-10 11:40:07,290: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/587f5840-0a36-43b8-85ab-da19a13c3973
[2017-05-10 11:40:07,291: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/587f5840-0a36-43b8-85ab-da19a13c3973
[2017-05-10 11:40:07,291: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/587f5840-0a36-43b8-85ab-da19a13c3973
[2017-05-10 11:40:07,291: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8dc497ee-e931-4f4a-9a7a-2bf4a061baf1
[2017-05-10 11:40:07,291: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8dc497ee-e931-4f4a-9a7a-2bf4a061baf1
[2017-05-10 11:40:07,301: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8dc497ee-e931-4f4a-9a7a-2bf4a061baf1
[2017-05-10 11:40:07,301: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8dc497ee-e931-4f4a-9a7a-2bf4a061baf1
[2017-05-10 11:40:07,302: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-10 11:40:07.302234+00:00:
[2017-05-10 11:40:07,302: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-10 11:40:07.302234+00:00:
[2017-05-10 11:40:07,302: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 11:40:07,302: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 11:40:07,322: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:40:07,322: INFO/job_scheduler]   Job 9 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 11:40:07,322: INFO/job_scheduler]   Job 10 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:07,322: INFO/job_scheduler]   Job 11 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:07,322: INFO/job_scheduler]   Job 12 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:07,322: INFO/job_scheduler]   Job 13 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:07,323: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:07,323: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:40:07,323: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 11:40:07,325: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-10 11:40:07.325547+00:00:
[2017-05-10 11:40:07,325: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-10 11:40:07.325547+00:00:
[2017-05-10 11:40:07,325: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 11:40:07,325: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 11:40:07,340: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:40:07,340: INFO/job_scheduler]   Job 9 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 11:40:07,340: INFO/job_scheduler]   Job 10 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:07,340: INFO/job_scheduler]   Job 11 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:07,340: INFO/job_scheduler]   Job 12 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:07,340: INFO/job_scheduler]   Job 13 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:07,340: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:07,340: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:40:07,341: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 11:40:07,343: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com restart_agent {}
[2017-05-10 11:40:07,357: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 23fc0eef-93f8-432f-8411-770d2d787de1
[2017-05-10 11:40:07,357: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 23fc0eef-93f8-432f-8411-770d2d787de1
[2017-05-10 11:40:07,410: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/587f5840-0a36-43b8-85ab-da19a13c3973
[2017-05-10 11:40:07,410: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/587f5840-0a36-43b8-85ab-da19a13c3973
[2017-05-10 11:40:07,411: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/587f5840-0a36-43b8-85ab-da19a13c3973
[2017-05-10 11:40:07,411: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/587f5840-0a36-43b8-85ab-da19a13c3973
[2017-05-10 11:40:07,411: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 23fc0eef-93f8-432f-8411-770d2d787de1
[2017-05-10 11:40:07,411: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 23fc0eef-93f8-432f-8411-770d2d787de1
[2017-05-10 11:40:07,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 23fc0eef-93f8-432f-8411-770d2d787de1
[2017-05-10 11:40:07,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 23fc0eef-93f8-432f-8411-770d2d787de1
[2017-05-10 11:40:07,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old 587f5840-0a36-43b8-85ab-da19a13c3973)
[2017-05-10 11:40:07,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old 587f5840-0a36-43b8-85ab-da19a13c3973)
[2017-05-10 11:40:33,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/f5b67c69-2b19-49db-b10b-eb8e48cfebfa
[2017-05-10 11:40:33,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/f5b67c69-2b19-49db-b10b-eb8e48cfebfa
[2017-05-10 11:40:33,526: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session f5b67c69-2b19-49db-b10b-eb8e48cfebfa because session terminated
[2017-05-10 11:40:33,526: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session f5b67c69-2b19-49db-b10b-eb8e48cfebfa because session terminated
[2017-05-10 11:40:33,531: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:40:33,531: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:40:33,773: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/29af1680-4488-4ada-84be-651b007b2840
[2017-05-10 11:40:33,773: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/29af1680-4488-4ada-84be-651b007b2840
[2017-05-10 11:40:33,774: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 29af1680-4488-4ada-84be-651b007b2840 because session terminated
[2017-05-10 11:40:33,774: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 29af1680-4488-4ada-84be-651b007b2840 because session terminated
[2017-05-10 11:40:33,776: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:40:33,776: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:40:33,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm6.lotus.hpdd.lab.intel.com new bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:40:33,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm6.lotus.hpdd.lab.intel.com new bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:40:33,832: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 11:40:33,851: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 82b4039a-16aa-42af-9f2c-3b3bb8f44101
[2017-05-10 11:40:33,851: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 82b4039a-16aa-42af-9f2c-3b3bb8f44101
[2017-05-10 11:40:33,902: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/b09fa0eb-e6af-4b39-82b2-b5a1a56266c9
[2017-05-10 11:40:33,902: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/b09fa0eb-e6af-4b39-82b2-b5a1a56266c9
[2017-05-10 11:40:33,902: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session b09fa0eb-e6af-4b39-82b2-b5a1a56266c9 because session terminated
[2017-05-10 11:40:33,902: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session b09fa0eb-e6af-4b39-82b2-b5a1a56266c9 because session terminated
[2017-05-10 11:40:33,906: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:40:33,906: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:40:33,909: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm7.lotus.hpdd.lab.intel.com new bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:40:33,909: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm7.lotus.hpdd.lab.intel.com new bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:40:33,910: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 11:40:33,935: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d35405c1-e3ad-4a5c-b5f5-d08c6163ca45
[2017-05-10 11:40:33,935: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d35405c1-e3ad-4a5c-b5f5-d08c6163ca45
[2017-05-10 11:40:33,955: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:40:33,955: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:40:33,955: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:40:33,955: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:40:33,956: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 82b4039a-16aa-42af-9f2c-3b3bb8f44101
[2017-05-10 11:40:33,956: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 82b4039a-16aa-42af-9f2c-3b3bb8f44101
[2017-05-10 11:40:33,970: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 82b4039a-16aa-42af-9f2c-3b3bb8f44101
[2017-05-10 11:40:33,970: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 82b4039a-16aa-42af-9f2c-3b3bb8f44101
[2017-05-10 11:40:33,970: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-05-10 11:40:33,987: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 90eb326c-3dac-48d1-90d3-32c277dbf07f
[2017-05-10 11:40:33,987: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 90eb326c-3dac-48d1-90d3-32c277dbf07f
[2017-05-10 11:40:34,024: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:40:34,024: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:40:34,025: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:40:34,025: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:40:34,025: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d35405c1-e3ad-4a5c-b5f5-d08c6163ca45
[2017-05-10 11:40:34,025: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d35405c1-e3ad-4a5c-b5f5-d08c6163ca45
[2017-05-10 11:40:34,050: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d35405c1-e3ad-4a5c-b5f5-d08c6163ca45
[2017-05-10 11:40:34,050: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d35405c1-e3ad-4a5c-b5f5-d08c6163ca45
[2017-05-10 11:40:34,050: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-05-10 11:40:34,070: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b2241bac-12c3-4e35-a787-bfadf8879014
[2017-05-10 11:40:34,070: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b2241bac-12c3-4e35-a787-bfadf8879014
[2017-05-10 11:40:34,177: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm8.lotus.hpdd.lab.intel.com new 5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:40:34,177: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm8.lotus.hpdd.lab.intel.com new 5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:40:34,178: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 11:40:34,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b656c280-7b2f-47ad-98e8-e35d9483ec71
[2017-05-10 11:40:34,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b656c280-7b2f-47ad-98e8-e35d9483ec71
[2017-05-10 11:40:34,298: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:40:34,298: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:40:34,298: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:40:34,298: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:40:34,298: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b656c280-7b2f-47ad-98e8-e35d9483ec71
[2017-05-10 11:40:34,298: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b656c280-7b2f-47ad-98e8-e35d9483ec71
[2017-05-10 11:40:34,312: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b656c280-7b2f-47ad-98e8-e35d9483ec71
[2017-05-10 11:40:34,312: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b656c280-7b2f-47ad-98e8-e35d9483ec71
[2017-05-10 11:40:34,312: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-05-10 11:40:34,330: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ba18d87d-e608-4880-b812-6d8a76e5a884
[2017-05-10 11:40:34,330: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ba18d87d-e608-4880-b812-6d8a76e5a884
[2017-05-10 11:40:36,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/587f5840-0a36-43b8-85ab-da19a13c3973
[2017-05-10 11:40:36,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/587f5840-0a36-43b8-85ab-da19a13c3973
[2017-05-10 11:40:36,255: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 587f5840-0a36-43b8-85ab-da19a13c3973 because session terminated
[2017-05-10 11:40:36,255: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 587f5840-0a36-43b8-85ab-da19a13c3973 because session terminated
[2017-05-10 11:40:36,258: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:40:36,258: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:40:36,449: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new 6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:40:36,449: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new 6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:40:36,450: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 11:40:36,465: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f69b6ae6-6e1c-4ff0-882b-729e9711f65e
[2017-05-10 11:40:36,465: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f69b6ae6-6e1c-4ff0-882b-729e9711f65e
[2017-05-10 11:40:36,558: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:40:36,558: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:40:36,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:40:36,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:40:36,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f69b6ae6-6e1c-4ff0-882b-729e9711f65e
[2017-05-10 11:40:36,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f69b6ae6-6e1c-4ff0-882b-729e9711f65e
[2017-05-10 11:40:36,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f69b6ae6-6e1c-4ff0-882b-729e9711f65e
[2017-05-10 11:40:36,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f69b6ae6-6e1c-4ff0-882b-729e9711f65e
[2017-05-10 11:40:36,580: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-05-10 11:40:36,592: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0849b543-8f8c-48f6-99a8-7517c85e2561
[2017-05-10 11:40:36,592: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0849b543-8f8c-48f6-99a8-7517c85e2561
[2017-05-10 11:40:42,848: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:40:42,848: INFO/job_scheduler]   Job 31 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 11:40:42,848: INFO/job_scheduler]   Job 32 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:42,848: INFO/job_scheduler]   Job 33 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:42,848: INFO/job_scheduler]   Job 34 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:42,848: INFO/job_scheduler]   Job 35 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:42,849: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:42,849: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:40:42,849: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 11:40:43,597: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:40:43,597: INFO/job_scheduler]   Job 20 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 11:40:43,597: INFO/job_scheduler]   Job 21 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:43,597: INFO/job_scheduler]   Job 22 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:43,598: INFO/job_scheduler]   Job 23 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:43,598: INFO/job_scheduler]   Job 24 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:43,598: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:43,598: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:40:43,599: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 11:40:45,071: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:40:45,071: INFO/job_scheduler]   Job 42 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 11:40:45,071: INFO/job_scheduler]   Job 43 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:45,072: INFO/job_scheduler]   Job 44 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:45,072: INFO/job_scheduler]   Job 45 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:45,072: INFO/job_scheduler]   Job 46 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:45,072: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:45,072: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:40:45,073: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 11:40:46,195: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:40:46,195: INFO/job_scheduler]   Job 9 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 11:40:46,195: INFO/job_scheduler]   Job 10 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:46,196: INFO/job_scheduler]   Job 11 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:46,196: INFO/job_scheduler]   Job 12 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:46,196: INFO/job_scheduler]   Job 13 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:46,196: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:40:46,196: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:40:46,197: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 11:41:19,987: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:19,987: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:19,988: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:19,988: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:19,988: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 90eb326c-3dac-48d1-90d3-32c277dbf07f
[2017-05-10 11:41:19,988: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 90eb326c-3dac-48d1-90d3-32c277dbf07f
[2017-05-10 11:41:20,001: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 90eb326c-3dac-48d1-90d3-32c277dbf07f
[2017-05-10 11:41:20,001: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 90eb326c-3dac-48d1-90d3-32c277dbf07f
[2017-05-10 11:41:20,623: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 11:41:20,625: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 11:41:20,646: INFO/command_plan] add_jobs: created Job 53 (Trigger plugin poll for linux plugins)
[2017-05-10 11:41:21,004: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com install_packages {'repos': [u'e2fsprogs', u'iml-agent', u'lustre', u'zfs'], 'packages': [u'chroma-agent-management', u'kernel-devel-3.10.0-514.10.2.el7_lustre', u'lustre-osd-zfs', u'lustre-osd-ldiskfs', u'lustre-modules', u'lustre', u'zfs']}
[2017-05-10 11:41:21,018: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 59625572-f25f-4291-9659-cd22e8e91c69
[2017-05-10 11:41:21,018: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 59625572-f25f-4291-9659-cd22e8e91c69
[2017-05-10 11:41:21,243: INFO/job_scheduler] run_next: 1 runnable jobs of (41 pending, 4 tasked)
[2017-05-10 11:41:21,247: INFO/job_scheduler] Job 53: RunJobThread.run
[2017-05-10 11:41:21,249: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 11:41:21,256: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 11:41:21,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7d98a783-8f70-4908-b52a-cadf7264675d
[2017-05-10 11:41:21,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7d98a783-8f70-4908-b52a-cadf7264675d
[2017-05-10 11:41:21,283: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 43bed0d7-1259-4b28-b814-20098d18d4bd
[2017-05-10 11:41:21,283: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 43bed0d7-1259-4b28-b814-20098d18d4bd
[2017-05-10 11:41:21,327: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:21,327: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:21,327: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:21,327: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:21,327: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7d98a783-8f70-4908-b52a-cadf7264675d
[2017-05-10 11:41:21,327: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7d98a783-8f70-4908-b52a-cadf7264675d
[2017-05-10 11:41:21,349: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7d98a783-8f70-4908-b52a-cadf7264675d
[2017-05-10 11:41:21,349: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7d98a783-8f70-4908-b52a-cadf7264675d
[2017-05-10 11:41:21,358: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 11:41:21,363: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:21,363: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:21,363: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:21,363: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:21,363: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 43bed0d7-1259-4b28-b814-20098d18d4bd
[2017-05-10 11:41:21,363: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 43bed0d7-1259-4b28-b814-20098d18d4bd
[2017-05-10 11:41:21,403: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 43bed0d7-1259-4b28-b814-20098d18d4bd
[2017-05-10 11:41:21,403: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 43bed0d7-1259-4b28-b814-20098d18d4bd
[2017-05-10 11:41:21,403: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 11:41:21,403: INFO/job_scheduler] Job 53 finished 1 steps successfully
[2017-05-10 11:41:21,418: INFO/job_scheduler] Job 53: complete_job: Updating cache
[2017-05-10 11:41:21,418: INFO/job_scheduler] Job 53 completing (errored=False, cancelled=False)
[2017-05-10 11:41:21,468: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 11:41:24,143: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:24,143: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:24,144: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:24,144: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:24,144: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ba18d87d-e608-4880-b812-6d8a76e5a884
[2017-05-10 11:41:24,144: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ba18d87d-e608-4880-b812-6d8a76e5a884
[2017-05-10 11:41:24,154: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ba18d87d-e608-4880-b812-6d8a76e5a884
[2017-05-10 11:41:24,154: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ba18d87d-e608-4880-b812-6d8a76e5a884
[2017-05-10 11:41:24,778: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com install_packages {'repos': [u'e2fsprogs', u'iml-agent', u'lustre', u'zfs'], 'packages': [u'chroma-agent-management', u'kernel-devel-3.10.0-514.10.2.el7_lustre', u'lustre-osd-zfs', u'lustre-osd-ldiskfs', u'lustre-modules', u'lustre', u'zfs']}
[2017-05-10 11:41:24,799: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c9c74c38-8021-4402-a2e2-2437d097759e
[2017-05-10 11:41:24,799: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c9c74c38-8021-4402-a2e2-2437d097759e
[2017-05-10 11:41:25,941: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:25,941: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:25,942: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:25,942: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:25,942: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b2241bac-12c3-4e35-a787-bfadf8879014
[2017-05-10 11:41:25,942: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b2241bac-12c3-4e35-a787-bfadf8879014
[2017-05-10 11:41:25,962: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b2241bac-12c3-4e35-a787-bfadf8879014
[2017-05-10 11:41:25,962: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b2241bac-12c3-4e35-a787-bfadf8879014
[2017-05-10 11:41:26,372: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com install_packages {'repos': [u'e2fsprogs', u'iml-agent', u'lustre', u'zfs'], 'packages': [u'chroma-agent-management', u'kernel-devel-3.10.0-514.10.2.el7_lustre', u'lustre-osd-zfs', u'lustre-osd-ldiskfs', u'lustre-modules', u'lustre', u'zfs']}
[2017-05-10 11:41:26,385: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dd288bb4-d1a5-43c5-ad88-d5a203073e15
[2017-05-10 11:41:26,385: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dd288bb4-d1a5-43c5-ad88-d5a203073e15
[2017-05-10 11:41:27,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:27,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:27,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:27,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:27,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0849b543-8f8c-48f6-99a8-7517c85e2561
[2017-05-10 11:41:27,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0849b543-8f8c-48f6-99a8-7517c85e2561
[2017-05-10 11:41:27,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0849b543-8f8c-48f6-99a8-7517c85e2561
[2017-05-10 11:41:27,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0849b543-8f8c-48f6-99a8-7517c85e2561
[2017-05-10 11:41:27,717: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 11:41:27,718: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 11:41:27,748: INFO/command_plan] add_jobs: created Job 54 (Trigger plugin poll for linux plugins)
[2017-05-10 11:41:28,268: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com install_packages {'repos': [u'e2fsprogs', u'iml-agent', u'lustre', u'zfs'], 'packages': [u'chroma-agent-management', u'kernel-devel-3.10.0-514.10.2.el7_lustre', u'lustre-osd-zfs', u'lustre-osd-ldiskfs', u'lustre-modules', u'lustre', u'zfs']}
[2017-05-10 11:41:28,295: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ba45b311-9792-41cb-b742-7bf0bf18d622
[2017-05-10 11:41:28,295: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ba45b311-9792-41cb-b742-7bf0bf18d622
[2017-05-10 11:41:28,430: INFO/job_scheduler] run_next: 1 runnable jobs of (41 pending, 4 tasked)
[2017-05-10 11:41:28,434: INFO/job_scheduler] Job 54: RunJobThread.run
[2017-05-10 11:41:28,435: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 11:41:28,445: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 11:41:28,449: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 11:41:28,453: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc eb5addd9-0140-4f0f-b851-a809db1408ee
[2017-05-10 11:41:28,453: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc eb5addd9-0140-4f0f-b851-a809db1408ee
[2017-05-10 11:41:28,465: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cb7e2d85-e9ee-48cf-8d15-eb193c88b7a0
[2017-05-10 11:41:28,465: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cb7e2d85-e9ee-48cf-8d15-eb193c88b7a0
[2017-05-10 11:41:28,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 97dd28f1-3ec6-4868-a5e8-766d0185f99b
[2017-05-10 11:41:28,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 97dd28f1-3ec6-4868-a5e8-766d0185f99b
[2017-05-10 11:41:28,533: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:28,533: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:28,533: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:28,533: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:28,533: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc eb5addd9-0140-4f0f-b851-a809db1408ee
[2017-05-10 11:41:28,533: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc eb5addd9-0140-4f0f-b851-a809db1408ee
[2017-05-10 11:41:28,534: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:28,535: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc eb5addd9-0140-4f0f-b851-a809db1408ee
[2017-05-10 11:41:28,534: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:28,535: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc eb5addd9-0140-4f0f-b851-a809db1408ee
[2017-05-10 11:41:28,542: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 11:41:28,535: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:28,535: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:28,542: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cb7e2d85-e9ee-48cf-8d15-eb193c88b7a0
[2017-05-10 11:41:28,542: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cb7e2d85-e9ee-48cf-8d15-eb193c88b7a0
[2017-05-10 11:41:28,542: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cb7e2d85-e9ee-48cf-8d15-eb193c88b7a0
[2017-05-10 11:41:28,542: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cb7e2d85-e9ee-48cf-8d15-eb193c88b7a0
[2017-05-10 11:41:28,542: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 11:41:28,557: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:28,557: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:28,557: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:28,557: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:28,558: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 97dd28f1-3ec6-4868-a5e8-766d0185f99b
[2017-05-10 11:41:28,558: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 97dd28f1-3ec6-4868-a5e8-766d0185f99b
[2017-05-10 11:41:28,594: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 97dd28f1-3ec6-4868-a5e8-766d0185f99b
[2017-05-10 11:41:28,594: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 97dd28f1-3ec6-4868-a5e8-766d0185f99b
[2017-05-10 11:41:28,594: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 11:41:28,595: INFO/job_scheduler] Job 54 finished 1 steps successfully
[2017-05-10 11:41:28,646: INFO/job_scheduler] Job 54: complete_job: Updating cache
[2017-05-10 11:41:28,646: INFO/job_scheduler] Job 54 completing (errored=False, cancelled=False)
[2017-05-10 11:41:28,689: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 11:41:39,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:39,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:39,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:39,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:39,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 59625572-f25f-4291-9659-cd22e8e91c69
[2017-05-10 11:41:39,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 59625572-f25f-4291-9659-cd22e8e91c69
[2017-05-10 11:41:39,585: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 59625572-f25f-4291-9659-cd22e8e91c69
[2017-05-10 11:41:39,585: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 59625572-f25f-4291-9659-cd22e8e91c69
[2017-05-10 11:41:40,059: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com kernel_status {}
[2017-05-10 11:41:40,074: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f717ce06-aa59-47a6-916f-9f9de8dbcdb1
[2017-05-10 11:41:40,074: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f717ce06-aa59-47a6-916f-9f9de8dbcdb1
[2017-05-10 11:41:43,173: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:43,173: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:43,173: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:43,173: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:43,185: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f717ce06-aa59-47a6-916f-9f9de8dbcdb1
[2017-05-10 11:41:43,185: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f717ce06-aa59-47a6-916f-9f9de8dbcdb1
[2017-05-10 11:41:43,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f717ce06-aa59-47a6-916f-9f9de8dbcdb1
[2017-05-10 11:41:43,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f717ce06-aa59-47a6-916f-9f9de8dbcdb1
[2017-05-10 11:41:43,189: INFO/job_scheduler] Job 20 finished 4 steps successfully
[2017-05-10 11:41:43,268: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 139877354709136
[2017-05-10 11:41:43,272: INFO/job] Job 20: StateChangeJob complete, setting state packages_installed on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 11:41:43,272: INFO/job] Job 20: not_deleted=True
[2017-05-10 11:41:43,272: INFO/job_scheduler] Job 20: complete_job: Updating cache
[2017-05-10 11:41:43,273: INFO/job_scheduler] Job 20: locked_item 139877354793616 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 11:41:43,275: INFO/job_scheduler] Job 20 completing (errored=False, cancelled=False)
[2017-05-10 11:41:43,291: INFO/job_scheduler] run_next: 4 runnable jobs of (40 pending, 3 tasked)
[2017-05-10 11:41:43,314: INFO/job_scheduler] Job 21: RunJobThread.run
[2017-05-10 11:41:43,315: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-05-10 11:41:43,326: INFO/job_scheduler] Job 22: RunJobThread.run
[2017-05-10 11:41:43,328: INFO/job_scheduler] Job 23: RunJobThread.run
[2017-05-10 11:41:43,329: INFO/job_scheduler] Job 24: open DB connection during completion
[2017-05-10 11:41:43,330: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-05-10 11:41:43,331: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-05-10 11:41:43,334: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1188a14d-9d76-439f-bc5e-bd51450a8cde
[2017-05-10 11:41:43,334: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1188a14d-9d76-439f-bc5e-bd51450a8cde
[2017-05-10 11:41:43,348: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 560a08c9-091f-4d24-b9c6-9afb6b6a62c2
[2017-05-10 11:41:43,348: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 560a08c9-091f-4d24-b9c6-9afb6b6a62c2
[2017-05-10 11:41:43,366: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 33088415-c8b3-4675-ad29-1dd661ab0340
[2017-05-10 11:41:43,366: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 33088415-c8b3-4675-ad29-1dd661ab0340
[2017-05-10 11:41:43,393: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 139877354793744
[2017-05-10 11:41:43,403: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 11:41:43,410: INFO/job] Job 24: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 11:41:43,410: INFO/job] Job 24: not_deleted=True
[2017-05-10 11:41:43,411: INFO/job_scheduler] Job 24: complete_job: Updating cache
[2017-05-10 11:41:43,412: INFO/job_scheduler] Job 24: locked_item 139877354793744 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 11:41:43,421: INFO/job_scheduler] Job 24 completing (errored=False, cancelled=False)
[2017-05-10 11:41:43,449: INFO/job_scheduler] run_next: 1 runnable jobs of (36 pending, 6 tasked)
[2017-05-10 11:41:43,454: INFO/job_scheduler] Job 25: RunJobThread.run
[2017-05-10 11:41:43,454: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com load_lnet {}
[2017-05-10 11:41:43,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 42c57ab2-d2ae-416a-9f72-10a2b3ef1d41
[2017-05-10 11:41:43,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 42c57ab2-d2ae-416a-9f72-10a2b3ef1d41
[2017-05-10 11:41:44,292: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:44,292: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:44,292: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:44,292: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:44,293: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c9c74c38-8021-4402-a2e2-2437d097759e
[2017-05-10 11:41:44,293: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c9c74c38-8021-4402-a2e2-2437d097759e
[2017-05-10 11:41:44,318: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c9c74c38-8021-4402-a2e2-2437d097759e
[2017-05-10 11:41:44,318: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c9c74c38-8021-4402-a2e2-2437d097759e
[2017-05-10 11:41:44,568: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:44,568: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:44,568: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:44,568: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:44,568: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 42c57ab2-d2ae-416a-9f72-10a2b3ef1d41
[2017-05-10 11:41:44,568: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 42c57ab2-d2ae-416a-9f72-10a2b3ef1d41
[2017-05-10 11:41:44,591: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 42c57ab2-d2ae-416a-9f72-10a2b3ef1d41
[2017-05-10 11:41:44,591: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 42c57ab2-d2ae-416a-9f72-10a2b3ef1d41
[2017-05-10 11:41:44,591: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 11:41:44,613: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9cbedfe4-0efc-443b-bd8a-36e2efa8d9f6
[2017-05-10 11:41:44,613: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9cbedfe4-0efc-443b-bd8a-36e2efa8d9f6
[2017-05-10 11:41:44,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:44,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:44,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:44,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:44,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9cbedfe4-0efc-443b-bd8a-36e2efa8d9f6
[2017-05-10 11:41:44,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9cbedfe4-0efc-443b-bd8a-36e2efa8d9f6
[2017-05-10 11:41:44,742: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9cbedfe4-0efc-443b-bd8a-36e2efa8d9f6
[2017-05-10 11:41:44,742: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9cbedfe4-0efc-443b-bd8a-36e2efa8d9f6
[2017-05-10 11:41:44,828: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com kernel_status {}
[2017-05-10 11:41:44,842: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bd8ad1df-d52f-43ce-b504-7ef8f9a07fa0
[2017-05-10 11:41:44,842: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bd8ad1df-d52f-43ce-b504-7ef8f9a07fa0
[2017-05-10 11:41:44,902: INFO/job_scheduler] Job 25 finished 2 steps successfully
[2017-05-10 11:41:44,919: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 139877354793744
[2017-05-10 11:41:44,928: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 11:41:44,931: INFO/job] Job 25: StateChangeJob complete, setting state lnet_down on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 11:41:44,931: INFO/job] Job 25: not_deleted=True
[2017-05-10 11:41:44,932: INFO/job_scheduler] Job 25: complete_job: Updating cache
[2017-05-10 11:41:44,932: INFO/job_scheduler] Job 25: locked_item 139877354793744 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 11:41:44,935: INFO/job_scheduler] Job 25 completing (errored=False, cancelled=False)
[2017-05-10 11:41:44,961: INFO/job_scheduler] run_next: 1 runnable jobs of (35 pending, 6 tasked)
[2017-05-10 11:41:44,965: INFO/job_scheduler] Job 28: RunJobThread.run
[2017-05-10 11:41:44,966: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_lnet {}
[2017-05-10 11:41:44,985: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fb0f7e2a-629f-463b-9bd7-13d278ed2c37
[2017-05-10 11:41:44,985: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fb0f7e2a-629f-463b-9bd7-13d278ed2c37
[2017-05-10 11:41:46,112: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:46,112: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:46,112: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:46,112: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:46,112: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fb0f7e2a-629f-463b-9bd7-13d278ed2c37
[2017-05-10 11:41:46,112: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fb0f7e2a-629f-463b-9bd7-13d278ed2c37
[2017-05-10 11:41:46,149: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fb0f7e2a-629f-463b-9bd7-13d278ed2c37
[2017-05-10 11:41:46,149: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fb0f7e2a-629f-463b-9bd7-13d278ed2c37
[2017-05-10 11:41:46,150: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 11:41:46,165: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 25716c5c-99cb-4cba-ac3d-ac4542eeb090
[2017-05-10 11:41:46,165: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 25716c5c-99cb-4cba-ac3d-ac4542eeb090
[2017-05-10 11:41:46,281: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:46,281: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:46,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:46,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:46,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 25716c5c-99cb-4cba-ac3d-ac4542eeb090
[2017-05-10 11:41:46,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 25716c5c-99cb-4cba-ac3d-ac4542eeb090
[2017-05-10 11:41:46,331: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 25716c5c-99cb-4cba-ac3d-ac4542eeb090
[2017-05-10 11:41:46,331: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 25716c5c-99cb-4cba-ac3d-ac4542eeb090
[2017-05-10 11:41:46,500: INFO/job_scheduler] Job 28 finished 2 steps successfully
[2017-05-10 11:41:46,510: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 139877354793744
[2017-05-10 11:41:46,517: INFO/job] Job 28: StateChangeJob complete, setting state lnet_up on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 11:41:46,517: INFO/job] Job 28: not_deleted=True
[2017-05-10 11:41:46,517: INFO/job_scheduler] Job 28: complete_job: Updating cache
[2017-05-10 11:41:46,518: INFO/job_scheduler] Job 28: locked_item 139877354793744 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 11:41:46,522: INFO/job_scheduler] Job 28 completing (errored=False, cancelled=False)
[2017-05-10 11:41:46,542: INFO/job_scheduler] run_next: 0 runnable jobs of (34 pending, 6 tasked)
[2017-05-10 11:41:46,893: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:46,893: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:46,894: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:46,894: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:46,894: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dd288bb4-d1a5-43c5-ad88-d5a203073e15
[2017-05-10 11:41:46,894: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dd288bb4-d1a5-43c5-ad88-d5a203073e15
[2017-05-10 11:41:46,921: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dd288bb4-d1a5-43c5-ad88-d5a203073e15
[2017-05-10 11:41:46,921: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dd288bb4-d1a5-43c5-ad88-d5a203073e15
[2017-05-10 11:41:46,960: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:46,960: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:46,960: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:46,960: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:46,962: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bd8ad1df-d52f-43ce-b504-7ef8f9a07fa0
[2017-05-10 11:41:46,962: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bd8ad1df-d52f-43ce-b504-7ef8f9a07fa0
[2017-05-10 11:41:47,007: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bd8ad1df-d52f-43ce-b504-7ef8f9a07fa0
[2017-05-10 11:41:47,007: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bd8ad1df-d52f-43ce-b504-7ef8f9a07fa0
[2017-05-10 11:41:47,008: INFO/job_scheduler] Job 42 finished 4 steps successfully
[2017-05-10 11:41:47,227: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 139877354827408
[2017-05-10 11:41:47,232: INFO/job] Job 42: StateChangeJob complete, setting state packages_installed on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 11:41:47,232: INFO/job] Job 42: not_deleted=True
[2017-05-10 11:41:47,232: INFO/job_scheduler] Job 42: complete_job: Updating cache
[2017-05-10 11:41:47,233: INFO/job_scheduler] Job 42: locked_item 139877354826640 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 11:41:47,235: INFO/job_scheduler] Job 42 completing (errored=False, cancelled=False)
[2017-05-10 11:41:47,266: INFO/job_scheduler] run_next: 4 runnable jobs of (34 pending, 5 tasked)
[2017-05-10 11:41:47,311: INFO/job_scheduler] Job 43: RunJobThread.run
[2017-05-10 11:41:47,311: INFO/job_scheduler] Job 44: open DB connection during completion
[2017-05-10 11:41:47,318: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-05-10 11:41:47,329: INFO/job_scheduler] Job 45: RunJobThread.run
[2017-05-10 11:41:47,332: INFO/job_scheduler] Job 46: RunJobThread.run
[2017-05-10 11:41:47,334: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-05-10 11:41:47,336: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-05-10 11:41:47,344: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 139877354807696
[2017-05-10 11:41:47,349: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ecf7627b-1021-4a17-8923-ccd893dcad3f
[2017-05-10 11:41:47,349: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ecf7627b-1021-4a17-8923-ccd893dcad3f
[2017-05-10 11:41:47,382: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f9100e5d-33b5-4dff-8fe6-1be673f21499
[2017-05-10 11:41:47,382: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f9100e5d-33b5-4dff-8fe6-1be673f21499
[2017-05-10 11:41:47,374: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 11:41:47,417: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bf5dc5d7-12bc-41bc-a63c-4a25c52c91fd
[2017-05-10 11:41:47,417: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bf5dc5d7-12bc-41bc-a63c-4a25c52c91fd
[2017-05-10 11:41:47,414: INFO/job] Job 44: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 11:41:47,419: INFO/job] Job 44: not_deleted=True
[2017-05-10 11:41:47,419: INFO/job_scheduler] Job 44: complete_job: Updating cache
[2017-05-10 11:41:47,420: INFO/job_scheduler] Job 44: locked_item 139877354807696 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 11:41:47,437: INFO/job_scheduler] Job 44 completing (errored=False, cancelled=False)
[2017-05-10 11:41:47,499: INFO/job_scheduler] run_next: 1 runnable jobs of (30 pending, 8 tasked)
[2017-05-10 11:41:47,526: INFO/job_scheduler] Job 48: RunJobThread.run
[2017-05-10 11:41:47,526: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com load_lnet {}
[2017-05-10 11:41:47,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3789d196-f498-4ae9-a1f4-2d92dbda1b1f
[2017-05-10 11:41:47,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3789d196-f498-4ae9-a1f4-2d92dbda1b1f
[2017-05-10 11:41:47,583: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com kernel_status {}
[2017-05-10 11:41:47,596: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8addea39-8be1-4e3f-b851-36366ae8eac1
[2017-05-10 11:41:47,596: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8addea39-8be1-4e3f-b851-36366ae8eac1
[2017-05-10 11:41:48,646: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:48,646: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:48,646: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:48,646: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:48,646: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3789d196-f498-4ae9-a1f4-2d92dbda1b1f
[2017-05-10 11:41:48,646: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3789d196-f498-4ae9-a1f4-2d92dbda1b1f
[2017-05-10 11:41:48,663: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3789d196-f498-4ae9-a1f4-2d92dbda1b1f
[2017-05-10 11:41:48,663: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3789d196-f498-4ae9-a1f4-2d92dbda1b1f
[2017-05-10 11:41:48,665: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 11:41:48,679: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fc9eb9fb-c7be-4a20-8cd1-cf7d1dddaa12
[2017-05-10 11:41:48,679: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fc9eb9fb-c7be-4a20-8cd1-cf7d1dddaa12
[2017-05-10 11:41:48,764: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:48,764: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:48,765: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:48,765: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:48,765: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fc9eb9fb-c7be-4a20-8cd1-cf7d1dddaa12
[2017-05-10 11:41:48,765: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fc9eb9fb-c7be-4a20-8cd1-cf7d1dddaa12
[2017-05-10 11:41:48,795: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fc9eb9fb-c7be-4a20-8cd1-cf7d1dddaa12
[2017-05-10 11:41:48,795: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fc9eb9fb-c7be-4a20-8cd1-cf7d1dddaa12
[2017-05-10 11:41:49,188: INFO/job_scheduler] Job 48 finished 2 steps successfully
[2017-05-10 11:41:49,218: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 139877354807696
[2017-05-10 11:41:49,226: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 11:41:49,231: INFO/job] Job 48: StateChangeJob complete, setting state lnet_down on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 11:41:49,231: INFO/job] Job 48: not_deleted=True
[2017-05-10 11:41:49,231: INFO/job_scheduler] Job 48: complete_job: Updating cache
[2017-05-10 11:41:49,232: INFO/job_scheduler] Job 48: locked_item 139877354807696 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 11:41:49,242: INFO/job_scheduler] Job 48 completing (errored=False, cancelled=False)
[2017-05-10 11:41:49,290: INFO/job_scheduler] run_next: 1 runnable jobs of (29 pending, 8 tasked)
[2017-05-10 11:41:49,300: INFO/job_scheduler] Job 49: RunJobThread.run
[2017-05-10 11:41:49,300: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_lnet {}
[2017-05-10 11:41:49,319: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc eddb0689-64fd-4b7a-9510-034bc1e40652
[2017-05-10 11:41:49,319: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc eddb0689-64fd-4b7a-9510-034bc1e40652
[2017-05-10 11:41:49,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:49,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:49,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:49,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:49,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc eddb0689-64fd-4b7a-9510-034bc1e40652
[2017-05-10 11:41:49,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc eddb0689-64fd-4b7a-9510-034bc1e40652
[2017-05-10 11:41:49,433: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc eddb0689-64fd-4b7a-9510-034bc1e40652
[2017-05-10 11:41:49,433: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc eddb0689-64fd-4b7a-9510-034bc1e40652
[2017-05-10 11:41:49,434: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 11:41:49,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0e366946-3cd6-412f-bee0-e32bbba22f8e
[2017-05-10 11:41:49,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0e366946-3cd6-412f-bee0-e32bbba22f8e
[2017-05-10 11:41:49,530: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:49,530: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:49,531: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:49,531: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:49,531: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 560a08c9-091f-4d24-b9c6-9afb6b6a62c2
[2017-05-10 11:41:49,531: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 560a08c9-091f-4d24-b9c6-9afb6b6a62c2
[2017-05-10 11:41:49,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:49,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:49,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:49,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:49,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0e366946-3cd6-412f-bee0-e32bbba22f8e
[2017-05-10 11:41:49,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0e366946-3cd6-412f-bee0-e32bbba22f8e
[2017-05-10 11:41:49,565: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0e366946-3cd6-412f-bee0-e32bbba22f8e
[2017-05-10 11:41:49,565: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0e366946-3cd6-412f-bee0-e32bbba22f8e
[2017-05-10 11:41:49,574: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 560a08c9-091f-4d24-b9c6-9afb6b6a62c2
[2017-05-10 11:41:49,574: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 560a08c9-091f-4d24-b9c6-9afb6b6a62c2
[2017-05-10 11:41:49,574: INFO/job_scheduler] Job 22 finished 1 steps successfully
[2017-05-10 11:41:49,604: INFO/job_scheduler] Job 49 finished 2 steps successfully
[2017-05-10 11:41:49,702: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 139877354793808
[2017-05-10 11:41:49,706: INFO/job] Job 22: StateChangeJob complete, setting state configured on lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-10 11:41:49,706: INFO/job] Job 22: not_deleted=True
[2017-05-10 11:41:49,706: INFO/job_scheduler] Job 22: complete_job: Updating cache
[2017-05-10 11:41:49,706: INFO/job_scheduler] Job 22: locked_item 139877354793808 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-10 11:41:49,709: INFO/job_scheduler] Job 22 completing (errored=False, cancelled=False)
[2017-05-10 11:41:49,724: INFO/job_scheduler] run_next: 0 runnable jobs of (28 pending, 8 tasked)
[2017-05-10 11:41:49,735: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 139877354807696
[2017-05-10 11:41:49,788: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm8.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 11:41:49,800: INFO/job] Job 49: StateChangeJob complete, setting state lnet_up on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 11:41:49,800: INFO/job] Job 49: not_deleted=True
[2017-05-10 11:41:49,800: INFO/job_scheduler] Job 49: complete_job: Updating cache
[2017-05-10 11:41:49,800: INFO/job_scheduler] Job 49: locked_item 139877354807696 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 11:41:49,806: INFO/job_scheduler] Job 49 completing (errored=False, cancelled=False)
[2017-05-10 11:41:49,817: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:49,817: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:49,817: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:49,817: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:49,817: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ba45b311-9792-41cb-b742-7bf0bf18d622
[2017-05-10 11:41:49,817: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ba45b311-9792-41cb-b742-7bf0bf18d622
[2017-05-10 11:41:49,824: INFO/job_scheduler] run_next: 0 runnable jobs of (28 pending, 7 tasked)
[2017-05-10 11:41:49,828: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ba45b311-9792-41cb-b742-7bf0bf18d622
[2017-05-10 11:41:49,828: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ba45b311-9792-41cb-b742-7bf0bf18d622
[2017-05-10 11:41:50,291: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com kernel_status {}
[2017-05-10 11:41:50,304: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 17626192-5261-45a9-823d-7ee16f861124
[2017-05-10 11:41:50,304: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 17626192-5261-45a9-823d-7ee16f861124
[2017-05-10 11:41:50,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:50,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:50,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:50,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:41:50,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1188a14d-9d76-439f-bc5e-bd51450a8cde
[2017-05-10 11:41:50,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1188a14d-9d76-439f-bc5e-bd51450a8cde
[2017-05-10 11:41:50,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1188a14d-9d76-439f-bc5e-bd51450a8cde
[2017-05-10 11:41:50,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1188a14d-9d76-439f-bc5e-bd51450a8cde
[2017-05-10 11:41:50,553: INFO/job_scheduler] Job 21 finished 1 steps successfully
[2017-05-10 11:41:50,622: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 139877354794448
[2017-05-10 11:41:50,626: INFO/job] Job 21: StateChangeJob complete, setting state configured on lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-10 11:41:50,626: INFO/job] Job 21: not_deleted=True
[2017-05-10 11:41:50,627: INFO/job_scheduler] Job 21: complete_job: Updating cache
[2017-05-10 11:41:50,627: INFO/job_scheduler] Job 21: locked_item 139877354794448 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-10 11:41:50,630: INFO/job_scheduler] Job 21 completing (errored=False, cancelled=False)
[2017-05-10 11:41:50,654: INFO/job_scheduler] run_next: 0 runnable jobs of (28 pending, 6 tasked)
[2017-05-10 11:41:50,714: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:50,714: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:50,714: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:50,714: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:50,714: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8addea39-8be1-4e3f-b851-36366ae8eac1
[2017-05-10 11:41:50,714: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8addea39-8be1-4e3f-b851-36366ae8eac1
[2017-05-10 11:41:50,764: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8addea39-8be1-4e3f-b851-36366ae8eac1
[2017-05-10 11:41:50,764: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8addea39-8be1-4e3f-b851-36366ae8eac1
[2017-05-10 11:41:50,765: INFO/job_scheduler] Job 31 finished 4 steps successfully
[2017-05-10 11:41:50,805: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 139877354786640
[2017-05-10 11:41:50,809: INFO/job] Job 31: StateChangeJob complete, setting state packages_installed on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 11:41:50,810: INFO/job] Job 31: not_deleted=True
[2017-05-10 11:41:50,810: INFO/job_scheduler] Job 31: complete_job: Updating cache
[2017-05-10 11:41:50,810: INFO/job_scheduler] Job 31: locked_item 139877354710608 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 11:41:50,813: INFO/job_scheduler] Job 31 completing (errored=False, cancelled=False)
[2017-05-10 11:41:50,831: INFO/job_scheduler] run_next: 4 runnable jobs of (28 pending, 5 tasked)
[2017-05-10 11:41:50,853: INFO/job_scheduler] Job 32: RunJobThread.run
[2017-05-10 11:41:50,853: INFO/job_scheduler] Job 33: RunJobThread.run
[2017-05-10 11:41:50,855: INFO/job_scheduler] Job 34: RunJobThread.run
[2017-05-10 11:41:50,855: INFO/job_scheduler] Job 35: open DB connection during completion
[2017-05-10 11:41:50,857: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-05-10 11:41:50,869: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-05-10 11:41:50,871: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-05-10 11:41:50,874: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 16572d62-9dfc-44b3-ab68-2a6d14f8cc59
[2017-05-10 11:41:50,874: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 16572d62-9dfc-44b3-ab68-2a6d14f8cc59
[2017-05-10 11:41:50,887: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d41f052a-d8c8-4550-9fc7-405f633772ff
[2017-05-10 11:41:50,887: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d41f052a-d8c8-4550-9fc7-405f633772ff
[2017-05-10 11:41:50,902: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4a16fe9c-46e5-4ba5-a205-297cded7edcd
[2017-05-10 11:41:50,902: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4a16fe9c-46e5-4ba5-a205-297cded7edcd
[2017-05-10 11:41:51,022: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 139877354783312
[2017-05-10 11:41:51,028: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 11:41:51,035: INFO/job] Job 35: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 11:41:51,036: INFO/job] Job 35: not_deleted=True
[2017-05-10 11:41:51,037: INFO/job_scheduler] Job 35: complete_job: Updating cache
[2017-05-10 11:41:51,037: INFO/job_scheduler] Job 35: locked_item 139877354783312 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 11:41:51,046: INFO/job_scheduler] Job 35 completing (errored=False, cancelled=False)
[2017-05-10 11:41:51,087: INFO/job_scheduler] run_next: 1 runnable jobs of (24 pending, 8 tasked)
[2017-05-10 11:41:51,092: INFO/job_scheduler] Job 36: RunJobThread.run
[2017-05-10 11:41:51,093: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com load_lnet {}
[2017-05-10 11:41:51,108: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a4273549-2dec-4c52-b97d-d74c427aadc9
[2017-05-10 11:41:51,108: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a4273549-2dec-4c52-b97d-d74c427aadc9
[2017-05-10 11:41:52,203: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:52,203: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:52,203: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:52,203: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:52,204: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a4273549-2dec-4c52-b97d-d74c427aadc9
[2017-05-10 11:41:52,204: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a4273549-2dec-4c52-b97d-d74c427aadc9
[2017-05-10 11:41:52,222: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a4273549-2dec-4c52-b97d-d74c427aadc9
[2017-05-10 11:41:52,222: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a4273549-2dec-4c52-b97d-d74c427aadc9
[2017-05-10 11:41:52,223: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 11:41:52,242: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 92d475be-1f19-4f40-880b-9b436c9d13a2
[2017-05-10 11:41:52,242: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 92d475be-1f19-4f40-880b-9b436c9d13a2
[2017-05-10 11:41:52,323: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:52,323: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:52,323: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:52,323: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:52,324: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 92d475be-1f19-4f40-880b-9b436c9d13a2
[2017-05-10 11:41:52,324: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 92d475be-1f19-4f40-880b-9b436c9d13a2
[2017-05-10 11:41:52,358: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 92d475be-1f19-4f40-880b-9b436c9d13a2
[2017-05-10 11:41:52,358: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 92d475be-1f19-4f40-880b-9b436c9d13a2
[2017-05-10 11:41:52,390: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:52,390: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:52,390: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:52,390: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:52,391: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 17626192-5261-45a9-823d-7ee16f861124
[2017-05-10 11:41:52,391: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 17626192-5261-45a9-823d-7ee16f861124
[2017-05-10 11:41:52,419: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 17626192-5261-45a9-823d-7ee16f861124
[2017-05-10 11:41:52,419: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 17626192-5261-45a9-823d-7ee16f861124
[2017-05-10 11:41:52,420: INFO/job_scheduler] Job 9 finished 4 steps successfully
[2017-05-10 11:41:52,456: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 139877219970768
[2017-05-10 11:41:52,461: INFO/job] Job 9: StateChangeJob complete, setting state packages_installed on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:41:52,461: INFO/job] Job 9: not_deleted=True
[2017-05-10 11:41:52,461: INFO/job_scheduler] Job 9: complete_job: Updating cache
[2017-05-10 11:41:52,461: INFO/job_scheduler] Job 9: locked_item 139877354711440 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 11:41:52,464: INFO/job_scheduler] Job 9 completing (errored=False, cancelled=False)
[2017-05-10 11:41:52,495: INFO/job_scheduler] run_next: 4 runnable jobs of (23 pending, 8 tasked)
[2017-05-10 11:41:52,515: INFO/job_scheduler] Job 10: RunJobThread.run
[2017-05-10 11:41:52,516: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-05-10 11:41:52,528: INFO/job_scheduler] Job 11: RunJobThread.run
[2017-05-10 11:41:52,528: INFO/job_scheduler] Job 12: open DB connection during completion
[2017-05-10 11:41:52,538: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-05-10 11:41:52,540: INFO/job_scheduler] Job 13: RunJobThread.run
[2017-05-10 11:41:52,540: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-05-10 11:41:52,550: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f98e7918-355f-40e4-ad87-cc54dcb3d54b
[2017-05-10 11:41:52,550: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f98e7918-355f-40e4-ad87-cc54dcb3d54b
[2017-05-10 11:41:52,560: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 139877354710096
[2017-05-10 11:41:52,572: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 11:41:52,588: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 274864ea-30dc-4e58-8934-61b815980e2f
[2017-05-10 11:41:52,588: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 274864ea-30dc-4e58-8934-61b815980e2f
[2017-05-10 11:41:52,600: INFO/job] Job 12: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 11:41:52,600: INFO/job] Job 12: not_deleted=True
[2017-05-10 11:41:52,601: INFO/job_scheduler] Job 12: complete_job: Updating cache
[2017-05-10 11:41:52,601: INFO/job_scheduler] Job 12: locked_item 139877354710096 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 11:41:52,604: INFO/job_scheduler] Job 12 completing (errored=False, cancelled=False)
[2017-05-10 11:41:52,638: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3489d384-605d-422a-81cd-5928a1b14b64
[2017-05-10 11:41:52,638: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3489d384-605d-422a-81cd-5928a1b14b64
[2017-05-10 11:41:52,689: INFO/job_scheduler] run_next: 1 runnable jobs of (19 pending, 11 tasked)
[2017-05-10 11:41:52,696: INFO/job_scheduler] Job 14: RunJobThread.run
[2017-05-10 11:41:52,697: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com load_lnet {}
[2017-05-10 11:41:52,712: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bd61e0d7-9cf8-4d7b-88e9-f7227763332a
[2017-05-10 11:41:52,712: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bd61e0d7-9cf8-4d7b-88e9-f7227763332a
[2017-05-10 11:41:52,728: INFO/job_scheduler] Job 36 finished 2 steps successfully
[2017-05-10 11:41:52,763: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:52,763: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:52,765: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:52,765: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:52,766: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bd61e0d7-9cf8-4d7b-88e9-f7227763332a
[2017-05-10 11:41:52,766: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bd61e0d7-9cf8-4d7b-88e9-f7227763332a
[2017-05-10 11:41:52,778: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bd61e0d7-9cf8-4d7b-88e9-f7227763332a
[2017-05-10 11:41:52,778: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bd61e0d7-9cf8-4d7b-88e9-f7227763332a
[2017-05-10 11:41:52,779: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 11:41:52,788: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 139877354783312
[2017-05-10 11:41:52,800: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 11:41:52,801: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 596a0ab3-7762-4aaa-98e0-31a743319569
[2017-05-10 11:41:52,801: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 596a0ab3-7762-4aaa-98e0-31a743319569
[2017-05-10 11:41:52,807: INFO/job] Job 36: StateChangeJob complete, setting state lnet_down on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 11:41:52,808: INFO/job] Job 36: not_deleted=True
[2017-05-10 11:41:52,809: INFO/job_scheduler] Job 36: complete_job: Updating cache
[2017-05-10 11:41:52,809: INFO/job_scheduler] Job 36: locked_item 139877354783312 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 11:41:52,817: INFO/job_scheduler] Job 36 completing (errored=False, cancelled=False)
[2017-05-10 11:41:52,869: INFO/job_scheduler] run_next: 1 runnable jobs of (18 pending, 11 tasked)
[2017-05-10 11:41:52,873: INFO/job_scheduler] Job 39: RunJobThread.run
[2017-05-10 11:41:52,874: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_lnet {}
[2017-05-10 11:41:52,890: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c68c1ae8-6649-4c63-a0c0-ca6187840006
[2017-05-10 11:41:52,890: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c68c1ae8-6649-4c63-a0c0-ca6187840006
[2017-05-10 11:41:52,984: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:52,984: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:52,985: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:52,985: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:52,985: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c68c1ae8-6649-4c63-a0c0-ca6187840006
[2017-05-10 11:41:52,985: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c68c1ae8-6649-4c63-a0c0-ca6187840006
[2017-05-10 11:41:53,004: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c68c1ae8-6649-4c63-a0c0-ca6187840006
[2017-05-10 11:41:53,004: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c68c1ae8-6649-4c63-a0c0-ca6187840006
[2017-05-10 11:41:53,005: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 11:41:53,020: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1be7cae5-81db-44b6-b75c-7191de9da8a8
[2017-05-10 11:41:53,020: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1be7cae5-81db-44b6-b75c-7191de9da8a8
[2017-05-10 11:41:53,107: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:53,107: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:53,107: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:53,107: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:53,108: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1be7cae5-81db-44b6-b75c-7191de9da8a8
[2017-05-10 11:41:53,108: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1be7cae5-81db-44b6-b75c-7191de9da8a8
[2017-05-10 11:41:53,135: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1be7cae5-81db-44b6-b75c-7191de9da8a8
[2017-05-10 11:41:53,135: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1be7cae5-81db-44b6-b75c-7191de9da8a8
[2017-05-10 11:41:53,175: INFO/job_scheduler] Job 39 finished 2 steps successfully
[2017-05-10 11:41:53,213: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 139877354783312
[2017-05-10 11:41:53,249: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 11:41:53,259: INFO/job] Job 39: StateChangeJob complete, setting state lnet_up on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 11:41:53,259: INFO/job] Job 39: not_deleted=True
[2017-05-10 11:41:53,259: INFO/job_scheduler] Job 39: complete_job: Updating cache
[2017-05-10 11:41:53,259: INFO/job_scheduler] Job 39: locked_item 139877354783312 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 11:41:53,262: INFO/job_scheduler] Job 39 completing (errored=False, cancelled=False)
[2017-05-10 11:41:53,286: INFO/job_scheduler] run_next: 0 runnable jobs of (17 pending, 11 tasked)
[2017-05-10 11:41:53,581: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:53,581: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:53,581: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:53,581: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:53,581: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ecf7627b-1021-4a17-8923-ccd893dcad3f
[2017-05-10 11:41:53,581: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ecf7627b-1021-4a17-8923-ccd893dcad3f
[2017-05-10 11:41:53,625: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ecf7627b-1021-4a17-8923-ccd893dcad3f
[2017-05-10 11:41:53,625: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ecf7627b-1021-4a17-8923-ccd893dcad3f
[2017-05-10 11:41:53,625: INFO/job_scheduler] Job 43 finished 1 steps successfully
[2017-05-10 11:41:53,650: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:53,650: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:53,651: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:53,651: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:41:53,651: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f9100e5d-33b5-4dff-8fe6-1be673f21499
[2017-05-10 11:41:53,651: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f9100e5d-33b5-4dff-8fe6-1be673f21499
[2017-05-10 11:41:53,698: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f9100e5d-33b5-4dff-8fe6-1be673f21499
[2017-05-10 11:41:53,698: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f9100e5d-33b5-4dff-8fe6-1be673f21499
[2017-05-10 11:41:53,698: INFO/job_scheduler] Job 46 finished 1 steps successfully
[2017-05-10 11:41:53,830: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 139877354808528
[2017-05-10 11:41:53,834: INFO/job] Job 43: StateChangeJob complete, setting state configured on lotus-32vm8.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-10 11:41:53,834: INFO/job] Job 43: not_deleted=True
[2017-05-10 11:41:53,834: INFO/job_scheduler] Job 43: complete_job: Updating cache
[2017-05-10 11:41:53,835: INFO/job_scheduler] Job 43: locked_item 139877354808528 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-10 11:41:53,842: INFO/job_scheduler] Job 43 completing (errored=False, cancelled=False)
[2017-05-10 11:41:53,872: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:53,872: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:53,872: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:53,872: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:53,872: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 596a0ab3-7762-4aaa-98e0-31a743319569
[2017-05-10 11:41:53,872: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 596a0ab3-7762-4aaa-98e0-31a743319569
[2017-05-10 11:41:53,916: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 596a0ab3-7762-4aaa-98e0-31a743319569
[2017-05-10 11:41:53,916: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 596a0ab3-7762-4aaa-98e0-31a743319569
[2017-05-10 11:41:53,997: INFO/job_scheduler] run_next: 0 runnable jobs of (17 pending, 10 tasked)
[2017-05-10 11:41:54,113: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 139877354808144
[2017-05-10 11:41:54,118: INFO/job] Job 46: StateChangeJob complete, setting state configured on lotus-32vm8.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-10 11:41:54,118: INFO/job] Job 46: not_deleted=True
[2017-05-10 11:41:54,118: INFO/job_scheduler] Job 46: complete_job: Updating cache
[2017-05-10 11:41:54,119: INFO/job_scheduler] Job 46: locked_item 139877354808144 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-10 11:41:54,125: INFO/job_scheduler] Job 46 completing (errored=False, cancelled=False)
[2017-05-10 11:41:54,156: INFO/job_scheduler] run_next: 0 runnable jobs of (17 pending, 9 tasked)
[2017-05-10 11:41:54,482: INFO/job_scheduler] Job 14 finished 2 steps successfully
[2017-05-10 11:41:54,655: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 139877354710096
[2017-05-10 11:41:54,663: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 11:41:54,675: INFO/job] Job 14: StateChangeJob complete, setting state lnet_down on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 11:41:54,679: INFO/job] Job 14: not_deleted=True
[2017-05-10 11:41:54,681: INFO/job_scheduler] Job 14: complete_job: Updating cache
[2017-05-10 11:41:54,683: INFO/job_scheduler] Job 14: locked_item 139877354710096 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 11:41:54,689: INFO/job_scheduler] Job 14 completing (errored=False, cancelled=False)
[2017-05-10 11:41:54,863: INFO/job_scheduler] run_next: 1 runnable jobs of (17 pending, 8 tasked)
[2017-05-10 11:41:54,868: INFO/job_scheduler] Job 17: RunJobThread.run
[2017-05-10 11:41:54,868: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_lnet {}
[2017-05-10 11:41:54,883: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1afcc69e-9d5f-4fdc-a05b-bfc9bb690086
[2017-05-10 11:41:54,883: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1afcc69e-9d5f-4fdc-a05b-bfc9bb690086
[2017-05-10 11:41:55,072: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:55,072: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:55,072: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:55,072: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:55,073: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1afcc69e-9d5f-4fdc-a05b-bfc9bb690086
[2017-05-10 11:41:55,073: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1afcc69e-9d5f-4fdc-a05b-bfc9bb690086
[2017-05-10 11:41:55,108: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1afcc69e-9d5f-4fdc-a05b-bfc9bb690086
[2017-05-10 11:41:55,108: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1afcc69e-9d5f-4fdc-a05b-bfc9bb690086
[2017-05-10 11:41:55,109: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 11:41:55,140: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d4b0d573-67c9-4562-9615-17f585ce5f15
[2017-05-10 11:41:55,140: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d4b0d573-67c9-4562-9615-17f585ce5f15
[2017-05-10 11:41:55,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:55,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:55,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:55,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:55,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d4b0d573-67c9-4562-9615-17f585ce5f15
[2017-05-10 11:41:55,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d4b0d573-67c9-4562-9615-17f585ce5f15
[2017-05-10 11:41:55,407: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d4b0d573-67c9-4562-9615-17f585ce5f15
[2017-05-10 11:41:55,407: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d4b0d573-67c9-4562-9615-17f585ce5f15
[2017-05-10 11:41:55,612: INFO/job_scheduler] Job 17 finished 2 steps successfully
[2017-05-10 11:41:55,630: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 139877354710096
[2017-05-10 11:41:55,638: INFO/job] Job 17: StateChangeJob complete, setting state lnet_up on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 11:41:55,638: INFO/job] Job 17: not_deleted=True
[2017-05-10 11:41:55,638: INFO/job_scheduler] Job 17: complete_job: Updating cache
[2017-05-10 11:41:55,638: INFO/job_scheduler] Job 17: locked_item 139877354710096 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 11:41:55,642: INFO/job_scheduler] Job 17 completing (errored=False, cancelled=False)
[2017-05-10 11:41:55,664: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 8 tasked)
[2017-05-10 11:41:56,127: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:56,127: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:56,127: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:56,127: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:56,127: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4a16fe9c-46e5-4ba5-a205-297cded7edcd
[2017-05-10 11:41:56,127: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4a16fe9c-46e5-4ba5-a205-297cded7edcd
[2017-05-10 11:41:56,167: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4a16fe9c-46e5-4ba5-a205-297cded7edcd
[2017-05-10 11:41:56,167: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4a16fe9c-46e5-4ba5-a205-297cded7edcd
[2017-05-10 11:41:56,168: INFO/job_scheduler] Job 32 finished 1 steps successfully
[2017-05-10 11:41:56,339: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 139877354783824
[2017-05-10 11:41:56,345: INFO/job] Job 32: StateChangeJob complete, setting state configured on lotus-32vm7.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-10 11:41:56,346: INFO/job] Job 32: not_deleted=True
[2017-05-10 11:41:56,346: INFO/job_scheduler] Job 32: complete_job: Updating cache
[2017-05-10 11:41:56,346: INFO/job_scheduler] Job 32: locked_item 139877354783824 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-10 11:41:56,352: INFO/job_scheduler] Job 32 completing (errored=False, cancelled=False)
[2017-05-10 11:41:56,376: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 7 tasked)
[2017-05-10 11:41:56,845: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:56,845: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:56,845: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:56,845: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:56,846: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3489d384-605d-422a-81cd-5928a1b14b64
[2017-05-10 11:41:56,846: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3489d384-605d-422a-81cd-5928a1b14b64
[2017-05-10 11:41:56,854: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3489d384-605d-422a-81cd-5928a1b14b64
[2017-05-10 11:41:56,854: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3489d384-605d-422a-81cd-5928a1b14b64
[2017-05-10 11:41:56,855: INFO/job_scheduler] Job 13 finished 1 steps successfully
[2017-05-10 11:41:56,980: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 139877219970064
[2017-05-10 11:41:56,984: INFO/job] Job 13: StateChangeJob complete, setting state configured on lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-10 11:41:56,984: INFO/job] Job 13: not_deleted=True
[2017-05-10 11:41:56,984: INFO/job_scheduler] Job 13: complete_job: Updating cache
[2017-05-10 11:41:56,984: INFO/job_scheduler] Job 13: locked_item 139877219970064 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-10 11:41:56,988: INFO/job_scheduler] Job 13 completing (errored=False, cancelled=False)
[2017-05-10 11:41:57,014: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 6 tasked)
[2017-05-10 11:41:57,066: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:57,066: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:57,066: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:57,066: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:41:57,067: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 16572d62-9dfc-44b3-ab68-2a6d14f8cc59
[2017-05-10 11:41:57,067: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 16572d62-9dfc-44b3-ab68-2a6d14f8cc59
[2017-05-10 11:41:57,097: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 16572d62-9dfc-44b3-ab68-2a6d14f8cc59
[2017-05-10 11:41:57,097: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 16572d62-9dfc-44b3-ab68-2a6d14f8cc59
[2017-05-10 11:41:57,098: INFO/job_scheduler] Job 33 finished 1 steps successfully
[2017-05-10 11:41:57,192: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 139877354783056
[2017-05-10 11:41:57,196: INFO/job] Job 33: StateChangeJob complete, setting state configured on lotus-32vm7.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-10 11:41:57,197: INFO/job] Job 33: not_deleted=True
[2017-05-10 11:41:57,197: INFO/job_scheduler] Job 33: complete_job: Updating cache
[2017-05-10 11:41:57,197: INFO/job_scheduler] Job 33: locked_item 139877354783056 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-10 11:41:57,200: INFO/job_scheduler] Job 33 completing (errored=False, cancelled=False)
[2017-05-10 11:41:57,224: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 5 tasked)
[2017-05-10 11:41:57,858: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:57,858: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:57,858: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:57,858: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:41:57,858: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 274864ea-30dc-4e58-8934-61b815980e2f
[2017-05-10 11:41:57,858: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 274864ea-30dc-4e58-8934-61b815980e2f
[2017-05-10 11:41:57,905: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 274864ea-30dc-4e58-8934-61b815980e2f
[2017-05-10 11:41:57,905: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 274864ea-30dc-4e58-8934-61b815980e2f
[2017-05-10 11:41:57,906: INFO/job_scheduler] Job 11 finished 1 steps successfully
[2017-05-10 11:41:57,992: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 139877354712336
[2017-05-10 11:41:57,997: INFO/job] Job 11: StateChangeJob complete, setting state configured on lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-10 11:41:57,997: INFO/job] Job 11: not_deleted=True
[2017-05-10 11:41:57,997: INFO/job_scheduler] Job 11: complete_job: Updating cache
[2017-05-10 11:41:57,997: INFO/job_scheduler] Job 11: locked_item 139877354712336 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-10 11:41:58,001: INFO/job_scheduler] Job 11 completing (errored=False, cancelled=False)
[2017-05-10 11:41:58,024: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 4 tasked)
[2017-05-10 11:42:06,239: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:42:06,239: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:42:06,240: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:42:06,240: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:42:06,240: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f98e7918-355f-40e4-ad87-cc54dcb3d54b
[2017-05-10 11:42:06,240: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f98e7918-355f-40e4-ad87-cc54dcb3d54b
[2017-05-10 11:42:06,275: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f98e7918-355f-40e4-ad87-cc54dcb3d54b
[2017-05-10 11:42:06,275: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f98e7918-355f-40e4-ad87-cc54dcb3d54b
[2017-05-10 11:42:06,276: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.224', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-05-10 11:42:06,291: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a2be15d8-0640-4a38-8c3f-cee47dbcb80b
[2017-05-10 11:42:06,291: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a2be15d8-0640-4a38-8c3f-cee47dbcb80b
[2017-05-10 11:42:06,389: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:42:06,389: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:42:06,389: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:42:06,389: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:42:06,390: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a2be15d8-0640-4a38-8c3f-cee47dbcb80b
[2017-05-10 11:42:06,390: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a2be15d8-0640-4a38-8c3f-cee47dbcb80b
[2017-05-10 11:42:06,411: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a2be15d8-0640-4a38-8c3f-cee47dbcb80b
[2017-05-10 11:42:06,411: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a2be15d8-0640-4a38-8c3f-cee47dbcb80b
[2017-05-10 11:42:06,426: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 55179, 'pcs_password': 'bjhclFt5wlGurebMaQdh'}
[2017-05-10 11:42:06,442: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 952ff0ca-7a78-41ac-b8cd-f23e29c9a595
[2017-05-10 11:42:06,442: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 952ff0ca-7a78-41ac-b8cd-f23e29c9a595
[2017-05-10 11:42:11,125: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:42:11,125: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:42:11,125: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:42:11,125: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:42:11,125: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d41f052a-d8c8-4550-9fc7-405f633772ff
[2017-05-10 11:42:11,125: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d41f052a-d8c8-4550-9fc7-405f633772ff
[2017-05-10 11:42:11,155: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d41f052a-d8c8-4550-9fc7-405f633772ff
[2017-05-10 11:42:11,155: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d41f052a-d8c8-4550-9fc7-405f633772ff
[2017-05-10 11:42:11,156: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.226', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-05-10 11:42:11,171: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 79ff10d6-1f8a-40c8-a0a2-992c4d840d08
[2017-05-10 11:42:11,171: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 79ff10d6-1f8a-40c8-a0a2-992c4d840d08
[2017-05-10 11:42:11,247: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:42:11,247: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:42:11,247: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:42:11,247: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:42:11,248: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 79ff10d6-1f8a-40c8-a0a2-992c4d840d08
[2017-05-10 11:42:11,248: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 79ff10d6-1f8a-40c8-a0a2-992c4d840d08
[2017-05-10 11:42:11,286: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 79ff10d6-1f8a-40c8-a0a2-992c4d840d08
[2017-05-10 11:42:11,286: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 79ff10d6-1f8a-40c8-a0a2-992c4d840d08
[2017-05-10 11:42:11,302: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 51971, 'pcs_password': 'ha6fBuJNaLKrgz6AAA3e'}
[2017-05-10 11:42:11,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ba1481ad-974c-45e3-b358-96b33c3795ee
[2017-05-10 11:42:11,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ba1481ad-974c-45e3-b358-96b33c3795ee
[2017-05-10 11:42:18,806: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:42:18,806: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:42:18,806: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:42:18,806: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:42:18,807: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 952ff0ca-7a78-41ac-b8cd-f23e29c9a595
[2017-05-10 11:42:18,807: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 952ff0ca-7a78-41ac-b8cd-f23e29c9a595
[2017-05-10 11:42:18,809: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 952ff0ca-7a78-41ac-b8cd-f23e29c9a595
[2017-05-10 11:42:18,809: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 952ff0ca-7a78-41ac-b8cd-f23e29c9a595
[2017-05-10 11:42:18,809: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': True, 'mcast_port': 55179, 'new_node_fqdn': u'lotus-32vm5.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'bjhclFt5wlGurebMaQdh'}
[2017-05-10 11:42:18,830: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f1df196d-c587-4c31-b6d6-f18df96b7f91
[2017-05-10 11:42:18,830: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f1df196d-c587-4c31-b6d6-f18df96b7f91
[2017-05-10 11:42:23,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:42:23,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:42:23,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:42:23,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:42:23,496: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ba1481ad-974c-45e3-b358-96b33c3795ee
[2017-05-10 11:42:23,496: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ba1481ad-974c-45e3-b358-96b33c3795ee
[2017-05-10 11:42:23,531: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ba1481ad-974c-45e3-b358-96b33c3795ee
[2017-05-10 11:42:23,531: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ba1481ad-974c-45e3-b358-96b33c3795ee
[2017-05-10 11:42:23,532: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': True, 'mcast_port': 51971, 'new_node_fqdn': u'lotus-32vm7.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'ha6fBuJNaLKrgz6AAA3e'}
[2017-05-10 11:42:23,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 25c58fdd-86f2-44de-a576-a1dd8923053a
[2017-05-10 11:42:23,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 25c58fdd-86f2-44de-a576-a1dd8923053a
[2017-05-10 11:42:24,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:42:24,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:42:24,940: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:42:24,940: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:42:24,940: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f1df196d-c587-4c31-b6d6-f18df96b7f91
[2017-05-10 11:42:24,940: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f1df196d-c587-4c31-b6d6-f18df96b7f91
[2017-05-10 11:42:24,945: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f1df196d-c587-4c31-b6d6-f18df96b7f91
[2017-05-10 11:42:24,945: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f1df196d-c587-4c31-b6d6-f18df96b7f91
[2017-05-10 11:42:24,945: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 11:42:24.945699+00:00:
[2017-05-10 11:42:24,945: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 11:42:24.945699+00:00:
[2017-05-10 11:42:24,945: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->55179
[2017-05-10 11:42:24,945: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->55179
[2017-05-10 11:42:24,950: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 11:42:24,950: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 11:42:24,973: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 11:42:24,974: INFO/job_scheduler]   Job 10 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-05-10 11:42:24,974: INFO/job_scheduler]   Job 15 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-05-10 11:42:24,974: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:42:24,974: INFO/job_scheduler]   Job 18 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:42:24,975: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 4 tasked)
[2017-05-10 11:42:24,981: INFO/job_scheduler] Job 10 finished 1 steps successfully
[2017-05-10 11:42:25,018: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 139877220035344
[2017-05-10 11:42:25,028: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-10 11:42:25,039: INFO/job] Job 10: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:42:25,040: INFO/job] Job 10: not_deleted=True
[2017-05-10 11:42:25,041: INFO/job_scheduler] Job 10: complete_job: Updating cache
[2017-05-10 11:42:25,041: INFO/job_scheduler] Job 10: locked_item 139877220035344 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 11:42:25,057: INFO/job_scheduler] Job 10 completing (errored=False, cancelled=False)
[2017-05-10 11:42:25,096: INFO/job_scheduler] run_next: 1 runnable jobs of (16 pending, 3 tasked)
[2017-05-10 11:42:25,101: INFO/job_scheduler] Job 15: RunJobThread.run
[2017-05-10 11:42:25,101: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-05-10 11:42:25,117: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dfe175fe-0268-4266-b64f-26e804daf424
[2017-05-10 11:42:25,117: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dfe175fe-0268-4266-b64f-26e804daf424
[2017-05-10 11:42:26,430: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:42:26,430: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:42:26,431: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:42:26,431: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:42:26,431: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 33088415-c8b3-4675-ad29-1dd661ab0340
[2017-05-10 11:42:26,431: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 33088415-c8b3-4675-ad29-1dd661ab0340
[2017-05-10 11:42:26,445: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 33088415-c8b3-4675-ad29-1dd661ab0340
[2017-05-10 11:42:26,445: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 33088415-c8b3-4675-ad29-1dd661ab0340
[2017-05-10 11:42:26,445: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.225', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-05-10 11:42:26,464: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 894f7cd1-3283-4535-8cf4-0599eb72326c
[2017-05-10 11:42:26,464: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 894f7cd1-3283-4535-8cf4-0599eb72326c
[2017-05-10 11:42:26,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:42:26,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:42:26,554: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:42:26,554: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:42:26,554: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 894f7cd1-3283-4535-8cf4-0599eb72326c
[2017-05-10 11:42:26,554: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 894f7cd1-3283-4535-8cf4-0599eb72326c
[2017-05-10 11:42:26,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 894f7cd1-3283-4535-8cf4-0599eb72326c
[2017-05-10 11:42:26,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 894f7cd1-3283-4535-8cf4-0599eb72326c
[2017-05-10 11:42:26,598: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 55179, 'pcs_password': 'JHZA9ezZfwlF8LDAAL98'}
[2017-05-10 11:42:26,613: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f6ba1033-861e-4e72-bb25-9b47dd6007d2
[2017-05-10 11:42:26,613: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f6ba1033-861e-4e72-bb25-9b47dd6007d2
[2017-05-10 11:42:29,642: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:42:29,642: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:42:29,642: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:42:29,642: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:42:29,642: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 25c58fdd-86f2-44de-a576-a1dd8923053a
[2017-05-10 11:42:29,642: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 25c58fdd-86f2-44de-a576-a1dd8923053a
[2017-05-10 11:42:29,660: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 25c58fdd-86f2-44de-a576-a1dd8923053a
[2017-05-10 11:42:29,660: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 25c58fdd-86f2-44de-a576-a1dd8923053a
[2017-05-10 11:42:29,661: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 11:42:29.661204+00:00:
[2017-05-10 11:42:29,661: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 11:42:29.661204+00:00:
[2017-05-10 11:42:29,661: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->51971
[2017-05-10 11:42:29,661: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->51971
[2017-05-10 11:42:29,663: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 11:42:29,663: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 11:42:29,683: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 11:42:29,683: INFO/job_scheduler]   Job 34 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-05-10 11:42:29,685: INFO/job_scheduler]   Job 37 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-05-10 11:42:29,685: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:42:29,685: INFO/job_scheduler]   Job 40 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:42:29,686: INFO/job_scheduler] run_next: 0 runnable jobs of (15 pending, 4 tasked)
[2017-05-10 11:42:29,684: INFO/job_scheduler] Job 34 finished 1 steps successfully
[2017-05-10 11:42:29,792: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 139877354773264
[2017-05-10 11:42:29,801: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-10 11:42:29,814: INFO/job] Job 34: StateChangeJob complete, setting state stopped on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:42:29,815: INFO/job] Job 34: not_deleted=True
[2017-05-10 11:42:29,818: INFO/job_scheduler] Job 34: complete_job: Updating cache
[2017-05-10 11:42:29,820: INFO/job_scheduler] Job 34: locked_item 139877354773264 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 11:42:29,840: INFO/job_scheduler] Job 34 completing (errored=False, cancelled=False)
[2017-05-10 11:42:29,885: INFO/job_scheduler] run_next: 1 runnable jobs of (15 pending, 3 tasked)
[2017-05-10 11:42:29,889: INFO/job_scheduler] Job 37: RunJobThread.run
[2017-05-10 11:42:29,890: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-05-10 11:42:29,907: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 23dd2ab8-0845-4351-9863-42338a85ce0d
[2017-05-10 11:42:29,907: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 23dd2ab8-0845-4351-9863-42338a85ce0d
[2017-05-10 11:42:31,094: INFO/job_scheduler] run_next: 0 runnable jobs of (14 pending, 4 tasked)
[2017-05-10 11:42:31,116: INFO/job_scheduler] run_next: 0 runnable jobs of (14 pending, 4 tasked)
[2017-05-10 11:42:31,187: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:42:31,187: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:42:31,188: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:42:31,188: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:42:31,188: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bf5dc5d7-12bc-41bc-a63c-4a25c52c91fd
[2017-05-10 11:42:31,188: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bf5dc5d7-12bc-41bc-a63c-4a25c52c91fd
[2017-05-10 11:42:31,192: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bf5dc5d7-12bc-41bc-a63c-4a25c52c91fd
[2017-05-10 11:42:31,192: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bf5dc5d7-12bc-41bc-a63c-4a25c52c91fd
[2017-05-10 11:42:31,193: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.227', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-05-10 11:42:31,209: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6fadcbc4-2800-4d98-aced-d60f0dee807e
[2017-05-10 11:42:31,209: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6fadcbc4-2800-4d98-aced-d60f0dee807e
[2017-05-10 11:42:31,262: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:42:31,262: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:42:31,263: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:42:31,263: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:42:31,263: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6fadcbc4-2800-4d98-aced-d60f0dee807e
[2017-05-10 11:42:31,263: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6fadcbc4-2800-4d98-aced-d60f0dee807e
[2017-05-10 11:42:31,276: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6fadcbc4-2800-4d98-aced-d60f0dee807e
[2017-05-10 11:42:31,276: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6fadcbc4-2800-4d98-aced-d60f0dee807e
[2017-05-10 11:42:31,302: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 51971, 'pcs_password': '5PlyQvBWVvx4XTycp5Bv'}
[2017-05-10 11:42:31,316: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2b5bd9a5-4f65-41f2-8d36-1e41f0a11e67
[2017-05-10 11:42:31,316: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2b5bd9a5-4f65-41f2-8d36-1e41f0a11e67
[2017-05-10 11:42:33,245: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:42:33,245: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:42:33,245: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:42:33,245: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:42:33,245: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dfe175fe-0268-4266-b64f-26e804daf424
[2017-05-10 11:42:33,245: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dfe175fe-0268-4266-b64f-26e804daf424
[2017-05-10 11:42:33,283: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dfe175fe-0268-4266-b64f-26e804daf424
[2017-05-10 11:42:33,283: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dfe175fe-0268-4266-b64f-26e804daf424
[2017-05-10 11:42:33,283: INFO/job_scheduler] Job 15 finished 1 steps successfully
[2017-05-10 11:42:33,350: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 139877220035344
[2017-05-10 11:42:33,378: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 11:42:33,390: INFO/job] Job 15: StateChangeJob complete, setting state started on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:42:33,390: INFO/job] Job 15: not_deleted=True
[2017-05-10 11:42:33,390: INFO/job_scheduler] Job 15: complete_job: Updating cache
[2017-05-10 11:42:33,390: INFO/job_scheduler] Job 15: locked_item 139877220035344 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 11:42:33,399: INFO/job_scheduler] Job 15 completing (errored=False, cancelled=False)
[2017-05-10 11:42:33,433: INFO/job_scheduler] run_next: 1 runnable jobs of (14 pending, 3 tasked)
[2017-05-10 11:42:33,447: INFO/job_scheduler] Job 16: RunJobThread.run
[2017-05-10 11:42:33,448: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 11:42:33,464: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3a2a69c8-e1db-4d30-83ef-f446618479b5
[2017-05-10 11:42:33,464: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3a2a69c8-e1db-4d30-83ef-f446618479b5
[2017-05-10 11:42:38,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:42:38,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:42:38,086: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:42:38,086: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:42:38,086: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 23dd2ab8-0845-4351-9863-42338a85ce0d
[2017-05-10 11:42:38,086: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 23dd2ab8-0845-4351-9863-42338a85ce0d
[2017-05-10 11:42:38,125: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 23dd2ab8-0845-4351-9863-42338a85ce0d
[2017-05-10 11:42:38,125: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 23dd2ab8-0845-4351-9863-42338a85ce0d
[2017-05-10 11:42:38,125: INFO/job_scheduler] Job 37 finished 1 steps successfully
[2017-05-10 11:42:38,166: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 139877354773264
[2017-05-10 11:42:38,201: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 11:42:38,230: INFO/job] Job 37: StateChangeJob complete, setting state started on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:42:38,232: INFO/job] Job 37: not_deleted=True
[2017-05-10 11:42:38,232: INFO/job_scheduler] Job 37: complete_job: Updating cache
[2017-05-10 11:42:38,233: INFO/job_scheduler] Job 37: locked_item 139877354773264 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 11:42:38,243: INFO/job_scheduler] Job 37 completing (errored=False, cancelled=False)
[2017-05-10 11:42:38,274: INFO/job_scheduler] run_next: 1 runnable jobs of (13 pending, 3 tasked)
[2017-05-10 11:42:38,287: INFO/job_scheduler] Job 38: RunJobThread.run
[2017-05-10 11:42:38,288: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 11:42:38,303: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ef064d0f-ca82-4454-820d-c9fd7e2350b9
[2017-05-10 11:42:38,303: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ef064d0f-ca82-4454-820d-c9fd7e2350b9
[2017-05-10 11:42:38,714: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-10 11:42:38,732: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-10 11:42:38,767: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:42:38,767: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:42:38,767: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:42:38,767: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:42:38,767: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f6ba1033-861e-4e72-bb25-9b47dd6007d2
[2017-05-10 11:42:38,767: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f6ba1033-861e-4e72-bb25-9b47dd6007d2
[2017-05-10 11:42:38,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f6ba1033-861e-4e72-bb25-9b47dd6007d2
[2017-05-10 11:42:38,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f6ba1033-861e-4e72-bb25-9b47dd6007d2
[2017-05-10 11:42:38,785: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': False, 'mcast_port': 55179, 'new_node_fqdn': u'lotus-32vm6.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'JHZA9ezZfwlF8LDAAL98'}
[2017-05-10 11:42:38,797: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 67e0634b-70c7-4c87-b9ee-f61218005395
[2017-05-10 11:42:38,797: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 67e0634b-70c7-4c87-b9ee-f61218005395
[2017-05-10 11:42:40,530: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:42:40,530: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:42:40,530: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:42:40,530: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:42:40,530: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3a2a69c8-e1db-4d30-83ef-f446618479b5
[2017-05-10 11:42:40,530: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3a2a69c8-e1db-4d30-83ef-f446618479b5
[2017-05-10 11:42:40,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3a2a69c8-e1db-4d30-83ef-f446618479b5
[2017-05-10 11:42:40,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3a2a69c8-e1db-4d30-83ef-f446618479b5
[2017-05-10 11:42:40,582: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-05-10 11:42:40,600: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f245ab03-2524-42ec-abc5-4fad60d32850
[2017-05-10 11:42:40,600: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f245ab03-2524-42ec-abc5-4fad60d32850
[2017-05-10 11:42:41,178: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-10 11:42:41,197: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-10 11:42:43,482: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:42:43,482: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:42:43,482: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:42:43,482: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:42:43,482: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2b5bd9a5-4f65-41f2-8d36-1e41f0a11e67
[2017-05-10 11:42:43,482: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2b5bd9a5-4f65-41f2-8d36-1e41f0a11e67
[2017-05-10 11:42:43,532: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2b5bd9a5-4f65-41f2-8d36-1e41f0a11e67
[2017-05-10 11:42:43,532: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2b5bd9a5-4f65-41f2-8d36-1e41f0a11e67
[2017-05-10 11:42:43,533: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': False, 'mcast_port': 51971, 'new_node_fqdn': u'lotus-32vm8.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': '5PlyQvBWVvx4XTycp5Bv'}
[2017-05-10 11:42:43,547: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 36139664-c930-428d-beef-736573e7118e
[2017-05-10 11:42:43,547: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 36139664-c930-428d-beef-736573e7118e
[2017-05-10 11:42:44,876: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:42:44,876: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:42:44,877: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:42:44,877: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:42:44,877: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 67e0634b-70c7-4c87-b9ee-f61218005395
[2017-05-10 11:42:44,877: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 67e0634b-70c7-4c87-b9ee-f61218005395
[2017-05-10 11:42:44,914: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 67e0634b-70c7-4c87-b9ee-f61218005395
[2017-05-10 11:42:44,914: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 67e0634b-70c7-4c87-b9ee-f61218005395
[2017-05-10 11:42:44,915: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 11:42:44.914942+00:00:
[2017-05-10 11:42:44,915: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 11:42:44.914942+00:00:
[2017-05-10 11:42:44,915: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->55179
[2017-05-10 11:42:44,915: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->55179
[2017-05-10 11:42:44,917: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 11:42:44,917: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 11:42:44,931: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 11:42:44,931: INFO/job_scheduler]   Job 23 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-05-10 11:42:44,931: INFO/job_scheduler]   Job 26 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-05-10 11:42:44,931: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:42:44,932: INFO/job_scheduler]   Job 29 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:42:44,932: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-10 11:42:44,934: INFO/job_scheduler] Job 23 finished 1 steps successfully
[2017-05-10 11:42:44,973: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 139877219969296
[2017-05-10 11:42:44,980: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-10 11:42:45,000: INFO/job] Job 23: StateChangeJob complete, setting state stopped on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:42:45,000: INFO/job] Job 23: not_deleted=True
[2017-05-10 11:42:45,000: INFO/job_scheduler] Job 23: complete_job: Updating cache
[2017-05-10 11:42:45,000: INFO/job_scheduler] Job 23: locked_item 139877219969296 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 11:42:45,004: INFO/job_scheduler] Job 23 completing (errored=False, cancelled=False)
[2017-05-10 11:42:45,037: INFO/job_scheduler] run_next: 1 runnable jobs of (12 pending, 3 tasked)
[2017-05-10 11:42:45,041: INFO/job_scheduler] Job 26: RunJobThread.run
[2017-05-10 11:42:45,041: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-05-10 11:42:45,055: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3b3048a2-0bf6-48ac-8918-4485350bb856
[2017-05-10 11:42:45,055: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3b3048a2-0bf6-48ac-8918-4485350bb856
[2017-05-10 11:42:45,402: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:42:45,402: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:42:45,403: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:42:45,403: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:42:45,403: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ef064d0f-ca82-4454-820d-c9fd7e2350b9
[2017-05-10 11:42:45,403: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ef064d0f-ca82-4454-820d-c9fd7e2350b9
[2017-05-10 11:42:45,418: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ef064d0f-ca82-4454-820d-c9fd7e2350b9
[2017-05-10 11:42:45,418: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ef064d0f-ca82-4454-820d-c9fd7e2350b9
[2017-05-10 11:42:45,419: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-05-10 11:42:45,439: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7f3cc0d5-5002-45a8-80a9-6f49c805a080
[2017-05-10 11:42:45,439: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7f3cc0d5-5002-45a8-80a9-6f49c805a080
[2017-05-10 11:42:49,657: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:42:49,657: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:42:49,657: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:42:49,657: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:42:49,658: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 36139664-c930-428d-beef-736573e7118e
[2017-05-10 11:42:49,658: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 36139664-c930-428d-beef-736573e7118e
[2017-05-10 11:42:49,663: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 36139664-c930-428d-beef-736573e7118e
[2017-05-10 11:42:49,663: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 36139664-c930-428d-beef-736573e7118e
[2017-05-10 11:42:49,664: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 11:42:49.664230+00:00:
[2017-05-10 11:42:49,664: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 11:42:49.664230+00:00:
[2017-05-10 11:42:49,664: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->51971
[2017-05-10 11:42:49,664: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->51971
[2017-05-10 11:42:49,666: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 11:42:49,666: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 11:42:49,682: INFO/job_scheduler] Job 45 finished 1 steps successfully
[2017-05-10 11:42:49,683: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 11:42:49,683: INFO/job_scheduler]   Job 45 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-05-10 11:42:49,683: INFO/job_scheduler]   Job 47 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-05-10 11:42:49,684: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:42:49,684: INFO/job_scheduler]   Job 51 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:42:49,684: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 4 tasked)
[2017-05-10 11:42:49,747: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 139877354826128
[2017-05-10 11:42:49,757: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-10 11:42:49,774: INFO/job] Job 45: StateChangeJob complete, setting state stopped on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:42:49,775: INFO/job] Job 45: not_deleted=True
[2017-05-10 11:42:49,777: INFO/job_scheduler] Job 45: complete_job: Updating cache
[2017-05-10 11:42:49,778: INFO/job_scheduler] Job 45: locked_item 139877354826128 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 11:42:49,792: INFO/job_scheduler] Job 45 completing (errored=False, cancelled=False)
[2017-05-10 11:42:49,844: INFO/job_scheduler] run_next: 1 runnable jobs of (11 pending, 3 tasked)
[2017-05-10 11:42:49,849: INFO/job_scheduler] Job 47: RunJobThread.run
[2017-05-10 11:42:49,849: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-05-10 11:42:49,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ccc1958b-4311-4b9a-92fd-6a460b0f1266
[2017-05-10 11:42:49,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ccc1958b-4311-4b9a-92fd-6a460b0f1266
[2017-05-10 11:42:51,186: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 4 tasked)
[2017-05-10 11:42:51,203: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 4 tasked)
[2017-05-10 11:42:53,211: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:42:53,211: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:42:53,211: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:42:53,211: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:42:53,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3b3048a2-0bf6-48ac-8918-4485350bb856
[2017-05-10 11:42:53,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3b3048a2-0bf6-48ac-8918-4485350bb856
[2017-05-10 11:42:53,221: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3b3048a2-0bf6-48ac-8918-4485350bb856
[2017-05-10 11:42:53,221: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3b3048a2-0bf6-48ac-8918-4485350bb856
[2017-05-10 11:42:53,221: INFO/job_scheduler] Job 26 finished 1 steps successfully
[2017-05-10 11:42:53,272: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 139877219969296
[2017-05-10 11:42:53,311: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm6.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 11:42:53,326: INFO/job] Job 26: StateChangeJob complete, setting state started on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:42:53,327: INFO/job] Job 26: not_deleted=True
[2017-05-10 11:42:53,327: INFO/job_scheduler] Job 26: complete_job: Updating cache
[2017-05-10 11:42:53,328: INFO/job_scheduler] Job 26: locked_item 139877219969296 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 11:42:53,347: INFO/job_scheduler] Job 26 completing (errored=False, cancelled=False)
[2017-05-10 11:42:53,397: INFO/job_scheduler] run_next: 1 runnable jobs of (10 pending, 3 tasked)
[2017-05-10 11:42:53,414: INFO/job_scheduler] Job 27: RunJobThread.run
[2017-05-10 11:42:53,415: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 11:42:53,431: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ea336cf6-fcbc-40ba-bd32-262f486f13bf
[2017-05-10 11:42:53,431: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ea336cf6-fcbc-40ba-bd32-262f486f13bf
[2017-05-10 11:42:57,986: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:42:57,986: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:42:57,987: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:42:57,987: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:42:57,987: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ccc1958b-4311-4b9a-92fd-6a460b0f1266
[2017-05-10 11:42:57,987: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ccc1958b-4311-4b9a-92fd-6a460b0f1266
[2017-05-10 11:42:58,032: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ccc1958b-4311-4b9a-92fd-6a460b0f1266
[2017-05-10 11:42:58,032: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ccc1958b-4311-4b9a-92fd-6a460b0f1266
[2017-05-10 11:42:58,033: INFO/job_scheduler] Job 47 finished 1 steps successfully
[2017-05-10 11:42:58,071: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 139877354826128
[2017-05-10 11:42:58,112: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm8.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 11:42:58,140: INFO/job] Job 47: StateChangeJob complete, setting state started on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:42:58,140: INFO/job] Job 47: not_deleted=True
[2017-05-10 11:42:58,140: INFO/job_scheduler] Job 47: complete_job: Updating cache
[2017-05-10 11:42:58,140: INFO/job_scheduler] Job 47: locked_item 139877354826128 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 11:42:58,151: INFO/job_scheduler] Job 47 completing (errored=False, cancelled=False)
[2017-05-10 11:42:58,194: INFO/job_scheduler] run_next: 1 runnable jobs of (9 pending, 3 tasked)
[2017-05-10 11:42:58,209: INFO/job_scheduler] Job 50: RunJobThread.run
[2017-05-10 11:42:58,210: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 11:42:58,227: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4c3b478a-fb79-42c1-a609-c57b7d36de03
[2017-05-10 11:42:58,227: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4c3b478a-fb79-42c1-a609-c57b7d36de03
[2017-05-10 11:42:58,233: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:42:58,251: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:42:58,282: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:42:58,282: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:42:58,282: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:42:58,283: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:42:58,305: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:42:58,306: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 11:42:58,306: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:42:58,306: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:42:58,905: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:42:58,922: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:00,548: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:43:00,548: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:43:00,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:43:00,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:43:00,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ea336cf6-fcbc-40ba-bd32-262f486f13bf
[2017-05-10 11:43:00,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ea336cf6-fcbc-40ba-bd32-262f486f13bf
[2017-05-10 11:43:00,596: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ea336cf6-fcbc-40ba-bd32-262f486f13bf
[2017-05-10 11:43:00,596: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ea336cf6-fcbc-40ba-bd32-262f486f13bf
[2017-05-10 11:43:00,597: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-05-10 11:43:00,619: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 242652de-ffc7-40ee-95e4-32ed1e8116da
[2017-05-10 11:43:00,619: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 242652de-ffc7-40ee-95e4-32ed1e8116da
[2017-05-10 11:43:05,344: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:43:05,344: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:43:05,344: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:43:05,344: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:43:05,344: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4c3b478a-fb79-42c1-a609-c57b7d36de03
[2017-05-10 11:43:05,344: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4c3b478a-fb79-42c1-a609-c57b7d36de03
[2017-05-10 11:43:05,393: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4c3b478a-fb79-42c1-a609-c57b7d36de03
[2017-05-10 11:43:05,393: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4c3b478a-fb79-42c1-a609-c57b7d36de03
[2017-05-10 11:43:05,394: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-05-10 11:43:05,408: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b15463b1-94b3-40e9-9750-73595ef55e8d
[2017-05-10 11:43:05,408: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b15463b1-94b3-40e9-9750-73595ef55e8d
[2017-05-10 11:43:08,276: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:08,290: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:08,965: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 11:43:08,965: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:08,965: INFO/job_scheduler]   Job 40 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:08,966: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:08,981: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:08,981: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 11:43:08,982: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:08,982: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:09,004: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 11:43:09,004: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:09,004: INFO/job_scheduler]   Job 51 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:09,004: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:09,022: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:09,022: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 11:43:09,022: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:09,022: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:11,287: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:11,303: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:11,337: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:11,337: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:43:11,337: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:11,337: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:11,360: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:11,360: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 11:43:11,360: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:11,360: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:18,266: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:18,284: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:18,314: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:18,314: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:43:18,315: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:18,315: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:18,337: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:18,337: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 11:43:18,337: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:18,337: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:18,889: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:18,906: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:18,945: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:18,945: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 11:43:18,945: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:18,945: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:18,971: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:18,971: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 11:43:18,971: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:18,972: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:19,003: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:19,022: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:21,330: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:21,330: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:43:21,330: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:21,330: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:21,351: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:21,352: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 11:43:21,352: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:21,352: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:28,314: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:28,315: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:43:28,315: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:28,315: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:28,338: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:28,339: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 11:43:28,339: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:28,339: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:28,909: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:28,910: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 11:43:28,910: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:28,910: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:28,930: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:28,930: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 11:43:28,930: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:28,930: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:31,372: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 11:43:31,373: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:31,373: INFO/job_scheduler]   Job 18 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:31,373: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:31,390: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:31,391: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:43:31,391: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:31,391: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:31,429: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 11:43:31,429: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:31,429: INFO/job_scheduler]   Job 29 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:31,429: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:31,446: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:31,446: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 11:43:31,446: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:31,446: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:38,326: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 11:43:38,326: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:38,326: INFO/job_scheduler]   Job 18 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:38,327: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:38,343: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:38,344: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:43:38,344: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:38,344: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:38,367: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 11:43:38,367: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:38,367: INFO/job_scheduler]   Job 29 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:38,368: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:38,385: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:38,385: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 11:43:38,385: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:38,386: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:39,019: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 11:43:39,019: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:39,019: INFO/job_scheduler]   Job 40 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:39,020: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:39,035: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:39,035: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 11:43:39,035: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:39,035: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:39,080: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 11:43:39,081: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:39,081: INFO/job_scheduler]   Job 51 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:39,081: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:39,096: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:39,096: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 11:43:39,096: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:39,096: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:41,369: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 11:43:41,369: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:41,370: INFO/job_scheduler]   Job 18 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:41,370: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:41,386: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:41,386: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:43:41,386: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:41,387: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:41,410: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 11:43:41,410: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:41,410: INFO/job_scheduler]   Job 29 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:41,410: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:41,427: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:41,428: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 11:43:41,428: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:41,428: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:41,863: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:43:41,863: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:43:41,863: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:43:41,863: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:43:41,864: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f245ab03-2524-42ec-abc5-4fad60d32850
[2017-05-10 11:43:41,864: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f245ab03-2524-42ec-abc5-4fad60d32850
[2017-05-10 11:43:41,877: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f245ab03-2524-42ec-abc5-4fad60d32850
[2017-05-10 11:43:41,877: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f245ab03-2524-42ec-abc5-4fad60d32850
[2017-05-10 11:43:41,879: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-10 11:43:41,895: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b9ab869e-4f8f-4b23-bac1-b8219f9e67da
[2017-05-10 11:43:41,895: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b9ab869e-4f8f-4b23-bac1-b8219f9e67da
[2017-05-10 11:43:48,386: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 11:43:48,386: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:48,386: INFO/job_scheduler]   Job 18 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:48,386: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:48,400: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:48,400: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:43:48,401: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:48,401: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:48,423: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 11:43:48,423: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:48,424: INFO/job_scheduler]   Job 29 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:48,424: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:48,440: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:48,440: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 11:43:48,440: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:48,441: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:48,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:43:48,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:43:48,977: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:43:48,977: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:43:48,977: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b9ab869e-4f8f-4b23-bac1-b8219f9e67da
[2017-05-10 11:43:48,977: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b9ab869e-4f8f-4b23-bac1-b8219f9e67da
[2017-05-10 11:43:48,978: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 11:43:48,978: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:48,979: INFO/job_scheduler]   Job 40 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:48,979: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:48,995: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:48,995: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 11:43:48,996: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:48,996: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:49,012: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b9ab869e-4f8f-4b23-bac1-b8219f9e67da
[2017-05-10 11:43:49,012: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b9ab869e-4f8f-4b23-bac1-b8219f9e67da
[2017-05-10 11:43:49,013: INFO/job_scheduler] Job 16 finished 3 steps successfully
[2017-05-10 11:43:49,018: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 11:43:49,019: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:49,019: INFO/job_scheduler]   Job 51 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:49,019: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:49,039: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:49,039: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 11:43:49,039: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:49,039: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 11:43:49,112: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 139877354712080
[2017-05-10 11:43:49,123: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-10 11:43:49,136: INFO/job] Job 16: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 11:43:49,138: INFO/job] Job 16: not_deleted=True
[2017-05-10 11:43:49,138: INFO/job_scheduler] Job 16: complete_job: Updating cache
[2017-05-10 11:43:49,139: INFO/job_scheduler] Job 16: locked_item 139877354712080 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 11:43:49,151: INFO/job_scheduler] Job 16 completing (errored=False, cancelled=False)
[2017-05-10 11:43:49,230: INFO/job_scheduler] run_next: 1 runnable jobs of (8 pending, 3 tasked)
[2017-05-10 11:43:49,245: INFO/job_scheduler] Job 18: RunJobThread.run
[2017-05-10 11:43:49,247: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 11:43:49,264: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0739e705-62eb-45bb-8a5f-e2154ddb00a9
[2017-05-10 11:43:49,264: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0739e705-62eb-45bb-8a5f-e2154ddb00a9
[2017-05-10 11:43:54,944: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:43:54,944: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:43:54,945: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:43:54,945: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:43:54,945: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 242652de-ffc7-40ee-95e4-32ed1e8116da
[2017-05-10 11:43:54,945: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 242652de-ffc7-40ee-95e4-32ed1e8116da
[2017-05-10 11:43:54,945: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 242652de-ffc7-40ee-95e4-32ed1e8116da
[2017-05-10 11:43:54,945: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 242652de-ffc7-40ee-95e4-32ed1e8116da
[2017-05-10 11:43:54,948: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-10 11:43:54,973: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e17793ce-cc4d-4cc0-a5de-2a14fc2f17be
[2017-05-10 11:43:54,973: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e17793ce-cc4d-4cc0-a5de-2a14fc2f17be
[2017-05-10 11:43:56,409: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:43:56,409: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:43:56,409: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:43:56,409: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:43:56,410: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0739e705-62eb-45bb-8a5f-e2154ddb00a9
[2017-05-10 11:43:56,410: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0739e705-62eb-45bb-8a5f-e2154ddb00a9
[2017-05-10 11:43:56,429: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0739e705-62eb-45bb-8a5f-e2154ddb00a9
[2017-05-10 11:43:56,429: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0739e705-62eb-45bb-8a5f-e2154ddb00a9
[2017-05-10 11:43:56,430: INFO/job_scheduler] Job 18 finished 1 steps successfully
[2017-05-10 11:43:56,512: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 139877354712080
[2017-05-10 11:43:56,565: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 11:43:56,577: INFO/job] Job 18: StateChangeJob complete, setting state started on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 11:43:56,578: INFO/job] Job 18: not_deleted=True
[2017-05-10 11:43:56,578: INFO/job_scheduler] Job 18: complete_job: Updating cache
[2017-05-10 11:43:56,578: INFO/job_scheduler] Job 18: locked_item 139877354712080 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 11:43:56,581: INFO/job_scheduler] Job 18 completing (errored=False, cancelled=False)
[2017-05-10 11:43:56,602: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-1
[2017-05-10 11:43:56,602: INFO/job_scheduler] _notify: Updating .mcast_port of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 55179
[2017-05-10 11:43:56,633: INFO/job_scheduler] _notify: Set mcast_port=55179 on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-05-10 11:43:56,634: INFO/job_scheduler] _notify: Updating .network_interfaces of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-05-10 11:43:56,645: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-05-10 11:43:56,653: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-10 11:43:56,660: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-05-10 11:43:56,679: INFO/job_scheduler] run_next: 1 runnable jobs of (7 pending, 3 tasked)
[2017-05-10 11:43:56,704: INFO/job_scheduler] Job 19: RunJobThread.run
[2017-05-10 11:43:56,707: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-05-10 11:43:56,731: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a04b9ddb-aff3-46cf-99c4-be50a0e316f8
[2017-05-10 11:43:56,731: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a04b9ddb-aff3-46cf-99c4-be50a0e316f8
[2017-05-10 11:43:58,372: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-05-10 11:43:58,390: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-05-10 11:43:58,422: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:58,422: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:58,423: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-05-10 11:43:58,442: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 11:43:58,443: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:58,443: INFO/job_scheduler]   Job 29 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:58,443: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-05-10 11:43:58,460: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:58,460: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 11:43:58,460: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:58,460: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-05-10 11:43:59,041: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 11:43:59,042: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:59,042: INFO/job_scheduler]   Job 40 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:59,042: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-05-10 11:43:59,056: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:59,056: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 11:43:59,056: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:59,057: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-05-10 11:43:59,076: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 11:43:59,076: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:59,076: INFO/job_scheduler]   Job 51 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:59,077: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-05-10 11:43:59,094: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:59,094: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 11:43:59,094: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:59,094: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-05-10 11:43:59,130: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 11:43:59,130: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:59,131: INFO/job_scheduler]   Job 40 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:59,131: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-05-10 11:43:59,146: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:59,146: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 11:43:59,146: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:59,146: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-05-10 11:43:59,166: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 11:43:59,166: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:59,166: INFO/job_scheduler]   Job 51 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:43:59,166: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-05-10 11:43:59,180: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:43:59,180: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 11:43:59,180: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:43:59,180: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-05-10 11:44:01,464: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:44:01,466: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:44:01,467: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-05-10 11:44:01,487: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 11:44:01,487: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:44:01,488: INFO/job_scheduler]   Job 29 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:44:01,488: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-05-10 11:44:01,504: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:44:01,504: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 11:44:01,504: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:44:01,504: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-05-10 11:44:03,100: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:44:03,100: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:44:03,101: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:44:03,101: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:44:03,101: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e17793ce-cc4d-4cc0-a5de-2a14fc2f17be
[2017-05-10 11:44:03,101: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e17793ce-cc4d-4cc0-a5de-2a14fc2f17be
[2017-05-10 11:44:03,139: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e17793ce-cc4d-4cc0-a5de-2a14fc2f17be
[2017-05-10 11:44:03,139: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e17793ce-cc4d-4cc0-a5de-2a14fc2f17be
[2017-05-10 11:44:03,140: INFO/job_scheduler] Job 27 finished 3 steps successfully
[2017-05-10 11:44:03,187: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 139877354794064
[2017-05-10 11:44:03,194: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-10 11:44:03,201: INFO/job] Job 27: StateChangeJob complete, setting state stopped on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 11:44:03,204: INFO/job] Job 27: not_deleted=True
[2017-05-10 11:44:03,205: INFO/job_scheduler] Job 27: complete_job: Updating cache
[2017-05-10 11:44:03,205: INFO/job_scheduler] Job 27: locked_item 139877354794064 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 11:44:03,215: INFO/job_scheduler] Job 27 completing (errored=False, cancelled=False)
[2017-05-10 11:44:03,258: INFO/job_scheduler] run_next: 1 runnable jobs of (6 pending, 3 tasked)
[2017-05-10 11:44:03,271: INFO/job_scheduler] Job 29: RunJobThread.run
[2017-05-10 11:44:03,272: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 11:44:03,286: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8302fe48-80c8-465d-bcb2-49ddaf2a2aad
[2017-05-10 11:44:03,286: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8302fe48-80c8-465d-bcb2-49ddaf2a2aad
[2017-05-10 11:44:06,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:44:06,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:44:06,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:44:06,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:44:06,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b15463b1-94b3-40e9-9750-73595ef55e8d
[2017-05-10 11:44:06,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b15463b1-94b3-40e9-9750-73595ef55e8d
[2017-05-10 11:44:06,786: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b15463b1-94b3-40e9-9750-73595ef55e8d
[2017-05-10 11:44:06,786: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b15463b1-94b3-40e9-9750-73595ef55e8d
[2017-05-10 11:44:06,788: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-10 11:44:06,809: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 03334889-3a43-4be7-a2e3-3cbcd073861d
[2017-05-10 11:44:06,809: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 03334889-3a43-4be7-a2e3-3cbcd073861d
[2017-05-10 11:44:07,814: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:44:07,814: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:44:07,815: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:44:07,815: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:44:07,815: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7f3cc0d5-5002-45a8-80a9-6f49c805a080
[2017-05-10 11:44:07,815: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7f3cc0d5-5002-45a8-80a9-6f49c805a080
[2017-05-10 11:44:07,818: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7f3cc0d5-5002-45a8-80a9-6f49c805a080
[2017-05-10 11:44:07,818: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7f3cc0d5-5002-45a8-80a9-6f49c805a080
[2017-05-10 11:44:07,821: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-10 11:44:07,838: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 005b0221-1afe-4c98-b707-cd2c9f8047c9
[2017-05-10 11:44:07,838: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 005b0221-1afe-4c98-b707-cd2c9f8047c9
[2017-05-10 11:44:08,407: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-10 11:44:08,422: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-10 11:44:08,456: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from True to False
[2017-05-10 11:44:08,464: INFO/job_scheduler] _notify: Set corosync_reported_up=False on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-05-10 11:44:08,471: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-10 11:44:08,481: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:44:08,481: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:44:08,481: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-10 11:44:08,495: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:44:08,495: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:44:08,495: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-10 11:44:09,033: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-10 11:44:09,050: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-10 11:44:09,086: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 11:44:09,086: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:44:09,086: INFO/job_scheduler]   Job 40 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:44:09,086: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-10 11:44:09,108: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:44:09,109: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 11:44:09,109: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:44:09,109: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-10 11:44:09,138: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 11:44:09,138: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:44:09,139: INFO/job_scheduler]   Job 51 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:44:09,139: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-10 11:44:09,160: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:44:09,160: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 11:44:09,160: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:44:09,161: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-10 11:44:09,183: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-10 11:44:09,205: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-10 11:44:09,240: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 11:44:09,240: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:44:09,240: INFO/job_scheduler]   Job 40 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:44:09,241: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-10 11:44:09,259: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:44:09,259: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 11:44:09,259: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:44:09,259: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-10 11:44:09,280: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 11:44:09,281: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:44:09,281: INFO/job_scheduler]   Job 51 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 11:44:09,281: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-10 11:44:09,298: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:44:09,298: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 11:44:09,298: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:44:09,299: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-10 11:44:10,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:44:10,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:44:10,413: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:44:10,413: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:44:10,413: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8302fe48-80c8-465d-bcb2-49ddaf2a2aad
[2017-05-10 11:44:10,413: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8302fe48-80c8-465d-bcb2-49ddaf2a2aad
[2017-05-10 11:44:10,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8302fe48-80c8-465d-bcb2-49ddaf2a2aad
[2017-05-10 11:44:10,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8302fe48-80c8-465d-bcb2-49ddaf2a2aad
[2017-05-10 11:44:10,451: INFO/job_scheduler] Job 29 finished 1 steps successfully
[2017-05-10 11:44:10,480: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 139877354794064
[2017-05-10 11:44:10,498: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm6.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 11:44:10,517: INFO/job] Job 29: StateChangeJob complete, setting state started on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 11:44:10,519: INFO/job] Job 29: not_deleted=True
[2017-05-10 11:44:10,521: INFO/job_scheduler] Job 29: complete_job: Updating cache
[2017-05-10 11:44:10,524: INFO/job_scheduler] Job 29: locked_item 139877354794064 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 11:44:10,529: INFO/job_scheduler] Job 29 completing (errored=False, cancelled=False)
[2017-05-10 11:44:10,562: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-2
[2017-05-10 11:44:10,562: INFO/job_scheduler] _notify: Updating .mcast_port of item 2 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 55179
[2017-05-10 11:44:10,569: INFO/job_scheduler] _notify: Set mcast_port=55179 on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-2) and saved
[2017-05-10 11:44:10,570: INFO/job_scheduler] _notify: Updating .network_interfaces of item 2 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-05-10 11:44:10,584: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-2) and saved
[2017-05-10 11:44:10,590: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 2 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-10 11:44:10,597: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-2) and saved
[2017-05-10 11:44:10,606: INFO/job_scheduler] run_next: 1 runnable jobs of (5 pending, 3 tasked)
[2017-05-10 11:44:10,628: INFO/job_scheduler] Job 30: RunJobThread.run
[2017-05-10 11:44:10,635: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-05-10 11:44:10,669: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 07c1c467-4a00-4f79-827e-359403032042
[2017-05-10 11:44:10,669: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 07c1c467-4a00-4f79-827e-359403032042
[2017-05-10 11:44:11,464: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-10 11:44:11,469: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-05-10 11:44:11,476: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 4 tasked)
[2017-05-10 11:44:11,489: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:44:11,489: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:44:11,489: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 4 tasked)
[2017-05-10 11:44:11,520: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:44:11,520: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:44:11,521: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 4 tasked)
[2017-05-10 11:44:13,887: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:44:13,887: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:44:13,887: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:44:13,887: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:44:13,887: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 03334889-3a43-4be7-a2e3-3cbcd073861d
[2017-05-10 11:44:13,887: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 03334889-3a43-4be7-a2e3-3cbcd073861d
[2017-05-10 11:44:13,924: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 03334889-3a43-4be7-a2e3-3cbcd073861d
[2017-05-10 11:44:13,924: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 03334889-3a43-4be7-a2e3-3cbcd073861d
[2017-05-10 11:44:13,925: INFO/job_scheduler] Job 50 finished 3 steps successfully
[2017-05-10 11:44:13,962: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 139877354808720
[2017-05-10 11:44:13,971: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-10 11:44:13,978: INFO/job] Job 50: StateChangeJob complete, setting state stopped on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 11:44:13,980: INFO/job] Job 50: not_deleted=True
[2017-05-10 11:44:13,981: INFO/job_scheduler] Job 50: complete_job: Updating cache
[2017-05-10 11:44:13,981: INFO/job_scheduler] Job 50: locked_item 139877354808720 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 11:44:13,998: INFO/job_scheduler] Job 50 completing (errored=False, cancelled=False)
[2017-05-10 11:44:14,030: INFO/job_scheduler] run_next: 1 runnable jobs of (4 pending, 3 tasked)
[2017-05-10 11:44:14,043: INFO/job_scheduler] Job 51: RunJobThread.run
[2017-05-10 11:44:14,044: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 11:44:14,060: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5bcfef34-5efc-461f-b6d8-ee65eb977840
[2017-05-10 11:44:14,060: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5bcfef34-5efc-461f-b6d8-ee65eb977840
[2017-05-10 11:44:14,906: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:44:14,906: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:44:14,906: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:44:14,906: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:44:14,906: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 005b0221-1afe-4c98-b707-cd2c9f8047c9
[2017-05-10 11:44:14,906: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 005b0221-1afe-4c98-b707-cd2c9f8047c9
[2017-05-10 11:44:14,907: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 005b0221-1afe-4c98-b707-cd2c9f8047c9
[2017-05-10 11:44:14,907: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 005b0221-1afe-4c98-b707-cd2c9f8047c9
[2017-05-10 11:44:14,907: INFO/job_scheduler] Job 38 finished 3 steps successfully
[2017-05-10 11:44:14,971: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 139877354784016
[2017-05-10 11:44:14,978: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-10 11:44:14,992: INFO/job] Job 38: StateChangeJob complete, setting state stopped on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 11:44:14,993: INFO/job] Job 38: not_deleted=True
[2017-05-10 11:44:14,994: INFO/job_scheduler] Job 38: complete_job: Updating cache
[2017-05-10 11:44:14,994: INFO/job_scheduler] Job 38: locked_item 139877354784016 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 11:44:15,002: INFO/job_scheduler] Job 38 completing (errored=False, cancelled=False)
[2017-05-10 11:44:15,043: INFO/job_scheduler] run_next: 1 runnable jobs of (3 pending, 3 tasked)
[2017-05-10 11:44:15,054: INFO/job_scheduler] Job 40: RunJobThread.run
[2017-05-10 11:44:15,055: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 11:44:15,068: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d867ca72-7cc2-4cf5-8a25-ee6c32e4dbe4
[2017-05-10 11:44:15,068: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d867ca72-7cc2-4cf5-8a25-ee6c32e4dbe4
[2017-05-10 11:44:18,465: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:44:18,465: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:44:18,465: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-05-10 11:44:18,488: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:44:18,489: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:44:18,489: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-05-10 11:44:19,061: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-05-10 11:44:19,084: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-05-10 11:44:19,128: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:44:19,128: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:44:19,128: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-05-10 11:44:19,161: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:44:19,161: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:44:19,161: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-05-10 11:44:19,201: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-05-10 11:44:19,221: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-05-10 11:44:19,252: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:44:19,252: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:44:19,252: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-05-10 11:44:19,276: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:44:19,277: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:44:19,277: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-05-10 11:44:21,133: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:44:21,133: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:44:21,134: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:44:21,134: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:44:21,134: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5bcfef34-5efc-461f-b6d8-ee65eb977840
[2017-05-10 11:44:21,134: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5bcfef34-5efc-461f-b6d8-ee65eb977840
[2017-05-10 11:44:21,178: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5bcfef34-5efc-461f-b6d8-ee65eb977840
[2017-05-10 11:44:21,178: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5bcfef34-5efc-461f-b6d8-ee65eb977840
[2017-05-10 11:44:21,179: INFO/job_scheduler] Job 51 finished 1 steps successfully
[2017-05-10 11:44:21,220: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 139877354808720
[2017-05-10 11:44:21,269: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm8.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 11:44:21,283: INFO/job] Job 51: StateChangeJob complete, setting state started on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 11:44:21,283: INFO/job] Job 51: not_deleted=True
[2017-05-10 11:44:21,284: INFO/job_scheduler] Job 51: complete_job: Updating cache
[2017-05-10 11:44:21,284: INFO/job_scheduler] Job 51: locked_item 139877354808720 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 11:44:21,303: INFO/job_scheduler] Job 51 completing (errored=False, cancelled=False)
[2017-05-10 11:44:21,337: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-4
[2017-05-10 11:44:21,338: INFO/job_scheduler] _notify: Updating .mcast_port of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 51971
[2017-05-10 11:44:21,342: INFO/job_scheduler] _notify: Set mcast_port=51971 on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-4) and saved
[2017-05-10 11:44:21,343: INFO/job_scheduler] _notify: Updating .network_interfaces of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-05-10 11:44:21,352: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-4) and saved
[2017-05-10 11:44:21,358: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-10 11:44:21,364: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-4) and saved
[2017-05-10 11:44:21,375: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 3 tasked)
[2017-05-10 11:44:21,400: INFO/job_scheduler] Job 52: RunJobThread.run
[2017-05-10 11:44:21,403: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-05-10 11:44:21,427: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 944e2d5f-be1a-45f4-876a-7963c5f3f665
[2017-05-10 11:44:21,427: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 944e2d5f-be1a-45f4-876a-7963c5f3f665
[2017-05-10 11:44:21,496: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:44:21,497: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:44:21,497: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 4 tasked)
[2017-05-10 11:44:21,520: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:44:21,520: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:44:21,520: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 4 tasked)
[2017-05-10 11:44:22,037: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:44:22,037: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:44:22,038: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:44:22,038: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:44:22,038: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a04b9ddb-aff3-46cf-99c4-be50a0e316f8
[2017-05-10 11:44:22,038: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a04b9ddb-aff3-46cf-99c4-be50a0e316f8
[2017-05-10 11:44:22,049: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a04b9ddb-aff3-46cf-99c4-be50a0e316f8
[2017-05-10 11:44:22,049: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a04b9ddb-aff3-46cf-99c4-be50a0e316f8
[2017-05-10 11:44:22,051: INFO/job_scheduler] Job 19 finished 1 steps successfully
[2017-05-10 11:44:22,157: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:44:22,157: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:44:22,158: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:44:22,158: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:44:22,160: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d867ca72-7cc2-4cf5-8a25-ee6c32e4dbe4
[2017-05-10 11:44:22,160: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d867ca72-7cc2-4cf5-8a25-ee6c32e4dbe4
[2017-05-10 11:44:22,185: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d867ca72-7cc2-4cf5-8a25-ee6c32e4dbe4
[2017-05-10 11:44:22,185: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d867ca72-7cc2-4cf5-8a25-ee6c32e4dbe4
[2017-05-10 11:44:22,185: INFO/job_scheduler] Job 40 finished 1 steps successfully
[2017-05-10 11:44:22,541: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 139877354711440
[2017-05-10 11:44:22,548: INFO/job] Job 19: StateChangeJob complete, setting state managed on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 11:44:22,548: INFO/job] Job 19: not_deleted=True
[2017-05-10 11:44:22,548: INFO/job_scheduler] Job 19: complete_job: Updating cache
[2017-05-10 11:44:22,548: INFO/job_scheduler] Job 19: locked_item 139877354711440 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 11:44:22,552: INFO/job_scheduler] Job 19 completing (errored=False, cancelled=False)
[2017-05-10 11:44:22,612: INFO/job_scheduler] Replaying 4 buffered notifications for ManagedHost-1
[2017-05-10 11:44:22,619: INFO/job_scheduler] _notify: Updating .immutable_state of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com) from True to False
[2017-05-10 11:44:22,623: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-1) and saved
[2017-05-10 11:44:22,638: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7f37bc0b42d0> to [2]
[2017-05-10 11:44:22,650: INFO/job_scheduler] _notify: Set ha_cluster_peers=[2] on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-1) and saved
[2017-05-10 11:44:22,712: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 11:44:22,741: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 139877354784016
[2017-05-10 11:44:22,773: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 11:44:22,792: INFO/job] Job 40: StateChangeJob complete, setting state started on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 11:44:22,793: INFO/job] Job 40: not_deleted=True
[2017-05-10 11:44:22,793: INFO/job_scheduler] Job 40: complete_job: Updating cache
[2017-05-10 11:44:22,794: INFO/job_scheduler] Job 40: locked_item 139877354784016 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 11:44:22,805: INFO/job_scheduler] Job 40 completing (errored=False, cancelled=False)
[2017-05-10 11:44:22,834: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-3
[2017-05-10 11:44:22,834: INFO/job_scheduler] _notify: Updating .mcast_port of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 51971
[2017-05-10 11:44:22,838: INFO/job_scheduler] _notify: Set mcast_port=51971 on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-3) and saved
[2017-05-10 11:44:22,839: INFO/job_scheduler] _notify: Updating .network_interfaces of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-05-10 11:44:22,848: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-3) and saved
[2017-05-10 11:44:22,854: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-10 11:44:22,858: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-3) and saved
[2017-05-10 11:44:22,868: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 11:44:22,889: INFO/job_scheduler] Job 41: RunJobThread.run
[2017-05-10 11:44:22,892: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-05-10 11:44:22,905: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc adece6e7-f877-47ae-8c0e-020cde54ad28
[2017-05-10 11:44:22,905: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc adece6e7-f877-47ae-8c0e-020cde54ad28
[2017-05-10 11:44:29,104: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from True to False
[2017-05-10 11:44:29,111: INFO/job_scheduler] _notify: Set corosync_reported_up=False on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-3) and saved
[2017-05-10 11:44:29,116: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 11:44:29,124: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:44:29,124: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:44:29,124: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 11:44:29,137: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from True to False
[2017-05-10 11:44:29,146: INFO/job_scheduler] _notify: Set corosync_reported_up=False on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-4) and saved
[2017-05-10 11:44:29,156: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 11:44:29,166: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:44:29,166: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:44:29,166: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 11:44:36,988: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:44:36,988: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:44:36,988: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:44:36,988: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:44:36,988: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 07c1c467-4a00-4f79-827e-359403032042
[2017-05-10 11:44:36,988: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 07c1c467-4a00-4f79-827e-359403032042
[2017-05-10 11:44:37,037: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 07c1c467-4a00-4f79-827e-359403032042
[2017-05-10 11:44:37,037: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 07c1c467-4a00-4f79-827e-359403032042
[2017-05-10 11:44:37,039: INFO/job_scheduler] Job 30 finished 1 steps successfully
[2017-05-10 11:44:37,574: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 139877354793616
[2017-05-10 11:44:37,580: INFO/job] Job 30: StateChangeJob complete, setting state managed on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 11:44:37,581: INFO/job] Job 30: not_deleted=True
[2017-05-10 11:44:37,581: INFO/job_scheduler] Job 30: complete_job: Updating cache
[2017-05-10 11:44:37,581: INFO/job_scheduler] Job 30: locked_item 139877354793616 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 11:44:37,584: INFO/job_scheduler] Job 30 completing (errored=False, cancelled=False)
[2017-05-10 11:44:37,673: INFO/job_scheduler] Replaying 4 buffered notifications for ManagedHost-2
[2017-05-10 11:44:37,683: INFO/job_scheduler] _notify: Updating .immutable_state of item 2 (lotus-32vm6.lotus.hpdd.lab.intel.com) from True to False
[2017-05-10 11:44:37,688: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-2) and saved
[2017-05-10 11:44:37,704: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 2 (lotus-32vm6.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7f37c8076c10> to [1]
[2017-05-10 11:44:37,716: INFO/job_scheduler] _notify: Set ha_cluster_peers=[1] on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-2) and saved
[2017-05-10 11:44:37,786: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-05-10 11:44:39,184: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:44:39,184: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:44:39,184: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-05-10 11:44:39,206: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 11:44:39,207: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 11:44:39,207: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-05-10 11:44:46,873: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:44:46,873: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:44:46,873: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:44:46,873: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:44:46,873: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 944e2d5f-be1a-45f4-876a-7963c5f3f665
[2017-05-10 11:44:46,873: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 944e2d5f-be1a-45f4-876a-7963c5f3f665
[2017-05-10 11:44:46,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 944e2d5f-be1a-45f4-876a-7963c5f3f665
[2017-05-10 11:44:46,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 944e2d5f-be1a-45f4-876a-7963c5f3f665
[2017-05-10 11:44:46,900: INFO/job_scheduler] Job 52 finished 1 steps successfully
[2017-05-10 11:44:47,391: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 139877354826640
[2017-05-10 11:44:47,397: INFO/job] Job 52: StateChangeJob complete, setting state managed on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 11:44:47,397: INFO/job] Job 52: not_deleted=True
[2017-05-10 11:44:47,397: INFO/job_scheduler] Job 52: complete_job: Updating cache
[2017-05-10 11:44:47,397: INFO/job_scheduler] Job 52: locked_item 139877354826640 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 11:44:47,400: INFO/job_scheduler] Job 52 completing (errored=False, cancelled=False)
[2017-05-10 11:44:47,465: INFO/job_scheduler] Replaying 4 buffered notifications for ManagedHost-4
[2017-05-10 11:44:47,483: INFO/job_scheduler] _notify: Updating .immutable_state of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com) from True to False
[2017-05-10 11:44:47,488: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-4) and saved
[2017-05-10 11:44:47,505: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7f37b404f650> to [3]
[2017-05-10 11:44:47,516: INFO/job_scheduler] _notify: Set ha_cluster_peers=[3] on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-4) and saved
[2017-05-10 11:44:47,587: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 11:44:48,249: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:44:48,249: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:44:48,249: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:44:48,249: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:44:48,249: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc adece6e7-f877-47ae-8c0e-020cde54ad28
[2017-05-10 11:44:48,249: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc adece6e7-f877-47ae-8c0e-020cde54ad28
[2017-05-10 11:44:48,278: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc adece6e7-f877-47ae-8c0e-020cde54ad28
[2017-05-10 11:44:48,278: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc adece6e7-f877-47ae-8c0e-020cde54ad28
[2017-05-10 11:44:48,280: INFO/job_scheduler] Job 41 finished 1 steps successfully
[2017-05-10 11:44:48,739: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 139877354710608
[2017-05-10 11:44:48,745: INFO/job] Job 41: StateChangeJob complete, setting state managed on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 11:44:48,746: INFO/job] Job 41: not_deleted=True
[2017-05-10 11:44:48,746: INFO/job_scheduler] Job 41: complete_job: Updating cache
[2017-05-10 11:44:48,746: INFO/job_scheduler] Job 41: locked_item 139877354710608 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 11:44:48,749: INFO/job_scheduler] Job 41 completing (errored=False, cancelled=False)
[2017-05-10 11:44:48,825: INFO/job_scheduler] Replaying 4 buffered notifications for ManagedHost-3
[2017-05-10 11:44:48,853: INFO/job_scheduler] _notify: Updating .immutable_state of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com) from True to False
[2017-05-10 11:44:48,868: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-3) and saved
[2017-05-10 11:44:48,888: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7f37bc0c9b90> to [4]
[2017-05-10 11:44:48,900: INFO/job_scheduler] _notify: Set ha_cluster_peers=[4] on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-3) and saved
[2017-05-10 11:44:48,966: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 11:44:49,210: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-10 11:44:49,216: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-3) and saved
[2017-05-10 11:44:49,228: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 11:44:49,267: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-10 11:44:49,272: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-4) and saved
[2017-05-10 11:44:49,280: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 11:47:40,716: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 11:47:40,733: INFO/command_plan] set_state: <class 'chroma_core.models.filesystem.ManagedFilesystem'>-1 transitioning from unavailable to available
[2017-05-10 11:47:41,259: INFO/command_plan] Created command 15 (Creating filesystem testfs) with 17 jobs
[2017-05-10 11:47:41,455: INFO/job_scheduler] run_next: 4 runnable jobs of (17 pending, 0 tasked)
[2017-05-10 11:47:42,142: INFO/job_scheduler] Job 55: RunJobThread.run
[2017-05-10 11:47:42,142: INFO/job_scheduler] Job 56: RunJobThread.run
[2017-05-10 11:47:42,143: INFO/job_scheduler] Job 57: RunJobThread.run
[2017-05-10 11:47:42,143: INFO/job_scheduler] Job 58: RunJobThread.run
[2017-05-10 11:47:42,164: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux'}
[2017-05-10 11:47:42,178: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1', 'device_type': 'zfs'}
[2017-05-10 11:47:42,180: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3', 'device_type': 'zfs'}
[2017-05-10 11:47:42,179: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5', 'device_type': 'zfs'}
[2017-05-10 11:47:42,187: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux'}
[2017-05-10 11:47:42,193: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3', 'device_type': 'zfs'}
[2017-05-10 11:47:42,204: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux'}
[2017-05-10 11:47:42,230: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3', 'device_type': 'zfs'}
[2017-05-10 11:47:42,230: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1', 'device_type': 'zfs'}
[2017-05-10 11:47:42,231: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5', 'device_type': 'zfs'}
[2017-05-10 11:47:42,245: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5', 'device_type': 'zfs'}
[2017-05-10 11:47:42,249: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1', 'device_type': 'zfs'}
[2017-05-10 11:47:42,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 35cbe8fa-597f-414c-be99-54dca8ba92fe
[2017-05-10 11:47:42,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 35cbe8fa-597f-414c-be99-54dca8ba92fe
[2017-05-10 11:47:42,287: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 40323c82-bb59-41fd-9990-44c478e2ae39
[2017-05-10 11:47:42,287: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 40323c82-bb59-41fd-9990-44c478e2ae39
[2017-05-10 11:47:42,334: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2e3863a6-cc2f-4421-8f59-0c377c2fe08c
[2017-05-10 11:47:42,334: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2e3863a6-cc2f-4421-8f59-0c377c2fe08c
[2017-05-10 11:47:42,353: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 70675ef9-3e40-4994-a463-42f214ab3440
[2017-05-10 11:47:42,353: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 70675ef9-3e40-4994-a463-42f214ab3440
[2017-05-10 11:47:42,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b1176a87-13e7-4e33-bb2f-6e4e54d2c679
[2017-05-10 11:47:42,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b1176a87-13e7-4e33-bb2f-6e4e54d2c679
[2017-05-10 11:47:42,400: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c16c15a5-c6a0-4e80-8453-4404b6a86f3c
[2017-05-10 11:47:42,400: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c16c15a5-c6a0-4e80-8453-4404b6a86f3c
[2017-05-10 11:47:42,430: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fa2a7cf4-2eb3-408e-90a2-f8211684183d
[2017-05-10 11:47:42,430: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fa2a7cf4-2eb3-408e-90a2-f8211684183d
[2017-05-10 11:47:42,463: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 024db111-85cf-4a7a-929e-d492d3fa64b7
[2017-05-10 11:47:42,463: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 024db111-85cf-4a7a-929e-d492d3fa64b7
[2017-05-10 11:47:42,493: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc efcec09c-5678-403a-badc-ed724ee01ec2
[2017-05-10 11:47:42,493: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc efcec09c-5678-403a-badc-ed724ee01ec2
[2017-05-10 11:47:42,512: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 39634c00-1f80-4d14-95eb-2d8682f96135
[2017-05-10 11:47:42,512: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 39634c00-1f80-4d14-95eb-2d8682f96135
[2017-05-10 11:47:42,512: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:47:42,512: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:47:42,512: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:47:42,512: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:47:42,513: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 35cbe8fa-597f-414c-be99-54dca8ba92fe
[2017-05-10 11:47:42,513: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 35cbe8fa-597f-414c-be99-54dca8ba92fe
[2017-05-10 11:47:42,532: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1a748ea3-5ca4-4f10-a1ba-1fcf02f864f5
[2017-05-10 11:47:42,532: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1a748ea3-5ca4-4f10-a1ba-1fcf02f864f5
[2017-05-10 11:47:42,542: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 35cbe8fa-597f-414c-be99-54dca8ba92fe
[2017-05-10 11:47:42,542: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 35cbe8fa-597f-414c-be99-54dca8ba92fe
[2017-05-10 11:47:42,564: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4b8f1ced-edd0-416a-a5ec-f8bb177ef18e
[2017-05-10 11:47:42,564: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4b8f1ced-edd0-416a-a5ec-f8bb177ef18e
[2017-05-10 11:47:42,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:47:42,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:47:42,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:47:42,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:47:42,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 024db111-85cf-4a7a-929e-d492d3fa64b7
[2017-05-10 11:47:42,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 024db111-85cf-4a7a-929e-d492d3fa64b7
[2017-05-10 11:47:42,592: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:47:42,592: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:47:42,592: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:47:42,592: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:47:42,592: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc efcec09c-5678-403a-badc-ed724ee01ec2
[2017-05-10 11:47:42,592: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc efcec09c-5678-403a-badc-ed724ee01ec2
[2017-05-10 11:47:42,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc efcec09c-5678-403a-badc-ed724ee01ec2
[2017-05-10 11:47:42,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc efcec09c-5678-403a-badc-ed724ee01ec2
[2017-05-10 11:47:42,611: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 024db111-85cf-4a7a-929e-d492d3fa64b7
[2017-05-10 11:47:42,611: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 024db111-85cf-4a7a-929e-d492d3fa64b7
[2017-05-10 11:47:42,612: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com import_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'pacemaker_ha_operation': False, 'device_type': 'linux', 'validate_importable': False}
[2017-05-10 11:47:42,633: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8a196f8e-4975-4c93-b65d-abf5fc3b7d9b
[2017-05-10 11:47:42,633: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8a196f8e-4975-4c93-b65d-abf5fc3b7d9b
[2017-05-10 11:47:42,633: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:47:42,633: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:47:42,633: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:47:42,633: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:47:42,633: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 39634c00-1f80-4d14-95eb-2d8682f96135
[2017-05-10 11:47:42,633: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 39634c00-1f80-4d14-95eb-2d8682f96135
[2017-05-10 11:47:42,652: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 39634c00-1f80-4d14-95eb-2d8682f96135
[2017-05-10 11:47:42,652: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 39634c00-1f80-4d14-95eb-2d8682f96135
[2017-05-10 11:47:42,699: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:47:42,699: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:47:42,700: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:47:42,700: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:47:42,700: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8a196f8e-4975-4c93-b65d-abf5fc3b7d9b
[2017-05-10 11:47:42,700: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8a196f8e-4975-4c93-b65d-abf5fc3b7d9b
[2017-05-10 11:47:42,704: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8a196f8e-4975-4c93-b65d-abf5fc3b7d9b
[2017-05-10 11:47:42,704: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8a196f8e-4975-4c93-b65d-abf5fc3b7d9b
[2017-05-10 11:47:42,704: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com check_block_device {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux'}
[2017-05-10 11:47:42,729: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e3c9e3e3-0c8e-40d1-8aa1-a7d44ea7e0d9
[2017-05-10 11:47:42,729: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e3c9e3e3-0c8e-40d1-8aa1-a7d44ea7e0d9
[2017-05-10 11:47:43,396: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:47:43,396: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:47:43,396: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:47:43,396: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:47:43,397: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 40323c82-bb59-41fd-9990-44c478e2ae39
[2017-05-10 11:47:43,397: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 40323c82-bb59-41fd-9990-44c478e2ae39
[2017-05-10 11:47:43,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:47:43,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:47:43,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:47:43,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:47:43,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2e3863a6-cc2f-4421-8f59-0c377c2fe08c
[2017-05-10 11:47:43,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2e3863a6-cc2f-4421-8f59-0c377c2fe08c
[2017-05-10 11:47:43,402: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2e3863a6-cc2f-4421-8f59-0c377c2fe08c
[2017-05-10 11:47:43,402: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2e3863a6-cc2f-4421-8f59-0c377c2fe08c
[2017-05-10 11:47:43,410: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 40323c82-bb59-41fd-9990-44c478e2ae39
[2017-05-10 11:47:43,410: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 40323c82-bb59-41fd-9990-44c478e2ae39
[2017-05-10 11:47:43,470: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:47:43,470: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:47:43,470: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:47:43,470: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:47:43,470: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b1176a87-13e7-4e33-bb2f-6e4e54d2c679
[2017-05-10 11:47:43,470: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b1176a87-13e7-4e33-bb2f-6e4e54d2c679
[2017-05-10 11:47:43,490: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:47:43,490: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:47:43,493: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:47:43,493: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:47:43,493: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fa2a7cf4-2eb3-408e-90a2-f8211684183d
[2017-05-10 11:47:43,493: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fa2a7cf4-2eb3-408e-90a2-f8211684183d
[2017-05-10 11:47:43,496: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b1176a87-13e7-4e33-bb2f-6e4e54d2c679
[2017-05-10 11:47:43,496: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b1176a87-13e7-4e33-bb2f-6e4e54d2c679
[2017-05-10 11:47:43,497: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fa2a7cf4-2eb3-408e-90a2-f8211684183d
[2017-05-10 11:47:43,497: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fa2a7cf4-2eb3-408e-90a2-f8211684183d
[2017-05-10 11:47:43,659: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:47:43,659: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:47:43,659: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:47:43,659: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:47:43,659: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1a748ea3-5ca4-4f10-a1ba-1fcf02f864f5
[2017-05-10 11:47:43,659: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1a748ea3-5ca4-4f10-a1ba-1fcf02f864f5
[2017-05-10 11:47:43,703: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1a748ea3-5ca4-4f10-a1ba-1fcf02f864f5
[2017-05-10 11:47:43,703: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1a748ea3-5ca4-4f10-a1ba-1fcf02f864f5
[2017-05-10 11:47:43,703: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com import_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1', 'pacemaker_ha_operation': False, 'device_type': 'zfs', 'validate_importable': False}
[2017-05-10 11:47:43,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3bd7dd43-abd0-47ff-b79b-65dec3c8041d
[2017-05-10 11:47:43,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3bd7dd43-abd0-47ff-b79b-65dec3c8041d
[2017-05-10 11:47:43,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:47:43,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:47:43,782: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:47:43,782: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:47:43,782: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e3c9e3e3-0c8e-40d1-8aa1-a7d44ea7e0d9
[2017-05-10 11:47:43,782: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e3c9e3e3-0c8e-40d1-8aa1-a7d44ea7e0d9
[2017-05-10 11:47:43,795: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e3c9e3e3-0c8e-40d1-8aa1-a7d44ea7e0d9
[2017-05-10 11:47:43,795: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e3c9e3e3-0c8e-40d1-8aa1-a7d44ea7e0d9
[2017-05-10 11:47:43,796: INFO/job] testfs-MDT0000 passed pre-format check, allowing subsequent reformats
[2017-05-10 11:47:43,922: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com format_target {'index': 0, 'mkfsoptions': '-J size=2048', 'backfstype': 'ldiskfs', 'target_name': u'testfs-MDT0000', 'mgsnode': ((u'10.14.81.224@tcp0',), (u'10.14.81.225@tcp0',)), 'target_types': 'mdt', 'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux', 'failnode': [u'10.14.81.224@tcp0'], 'fsname': u'testfs'}
[2017-05-10 11:47:43,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f77b775e-5fee-41a0-80b0-92a6adbc763f
[2017-05-10 11:47:43,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f77b775e-5fee-41a0-80b0-92a6adbc763f
[2017-05-10 11:47:44,466: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:47:44,466: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:47:44,466: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:47:44,466: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:47:44,466: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 70675ef9-3e40-4994-a463-42f214ab3440
[2017-05-10 11:47:44,466: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 70675ef9-3e40-4994-a463-42f214ab3440
[2017-05-10 11:47:44,470: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 70675ef9-3e40-4994-a463-42f214ab3440
[2017-05-10 11:47:44,470: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 70675ef9-3e40-4994-a463-42f214ab3440
[2017-05-10 11:47:45,505: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:47:45,505: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:47:45,505: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:47:45,505: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:47:45,505: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c16c15a5-c6a0-4e80-8453-4404b6a86f3c
[2017-05-10 11:47:45,505: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c16c15a5-c6a0-4e80-8453-4404b6a86f3c
[2017-05-10 11:47:45,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c16c15a5-c6a0-4e80-8453-4404b6a86f3c
[2017-05-10 11:47:45,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c16c15a5-c6a0-4e80-8453-4404b6a86f3c
[2017-05-10 11:47:45,539: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com import_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5', 'pacemaker_ha_operation': False, 'device_type': 'zfs', 'validate_importable': False}
[2017-05-10 11:47:45,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0b730fce-53e1-4acc-83c3-76b690a6fe13
[2017-05-10 11:47:45,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0b730fce-53e1-4acc-83c3-76b690a6fe13
[2017-05-10 11:47:45,675: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:47:45,675: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:47:45,675: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:47:45,675: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:47:45,676: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4b8f1ced-edd0-416a-a5ec-f8bb177ef18e
[2017-05-10 11:47:45,676: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4b8f1ced-edd0-416a-a5ec-f8bb177ef18e
[2017-05-10 11:47:45,698: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4b8f1ced-edd0-416a-a5ec-f8bb177ef18e
[2017-05-10 11:47:45,698: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4b8f1ced-edd0-416a-a5ec-f8bb177ef18e
[2017-05-10 11:47:45,700: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com import_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3', 'pacemaker_ha_operation': False, 'device_type': 'zfs', 'validate_importable': False}
[2017-05-10 11:47:45,734: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2e613cda-537a-4084-9b19-1e45c157e227
[2017-05-10 11:47:45,734: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2e613cda-537a-4084-9b19-1e45c157e227
[2017-05-10 11:47:47,694: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:47:47,694: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:47:47,694: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:47:47,694: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:47:47,694: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0b730fce-53e1-4acc-83c3-76b690a6fe13
[2017-05-10 11:47:47,694: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0b730fce-53e1-4acc-83c3-76b690a6fe13
[2017-05-10 11:47:47,703: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0b730fce-53e1-4acc-83c3-76b690a6fe13
[2017-05-10 11:47:47,703: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0b730fce-53e1-4acc-83c3-76b690a6fe13
[2017-05-10 11:47:47,704: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com check_block_device {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5', 'device_type': 'zfs'}
[2017-05-10 11:47:47,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc de6a9437-1dd2-40cc-8dbd-9342f7e93e4e
[2017-05-10 11:47:47,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc de6a9437-1dd2-40cc-8dbd-9342f7e93e4e
[2017-05-10 11:47:48,822: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:47:48,822: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:47:48,822: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:47:48,822: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:47:48,822: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc de6a9437-1dd2-40cc-8dbd-9342f7e93e4e
[2017-05-10 11:47:48,822: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc de6a9437-1dd2-40cc-8dbd-9342f7e93e4e
[2017-05-10 11:47:48,832: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:47:48,832: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:47:48,832: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:47:48,832: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:47:48,832: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2e613cda-537a-4084-9b19-1e45c157e227
[2017-05-10 11:47:48,832: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2e613cda-537a-4084-9b19-1e45c157e227
[2017-05-10 11:47:48,838: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc de6a9437-1dd2-40cc-8dbd-9342f7e93e4e
[2017-05-10 11:47:48,838: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc de6a9437-1dd2-40cc-8dbd-9342f7e93e4e
[2017-05-10 11:47:48,839: INFO/job] MGS passed pre-format check, allowing subsequent reformats
[2017-05-10 11:47:48,854: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2e613cda-537a-4084-9b19-1e45c157e227
[2017-05-10 11:47:48,854: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2e613cda-537a-4084-9b19-1e45c157e227
[2017-05-10 11:47:48,855: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com check_block_device {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3', 'device_type': 'zfs'}
[2017-05-10 11:47:48,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f6539cc7-dd07-4bea-bda4-6333ea794c1b
[2017-05-10 11:47:48,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f6539cc7-dd07-4bea-bda4-6333ea794c1b
[2017-05-10 11:47:48,875: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com format_target {'backfstype': 'zfs', 'target_name': u'MGS', 'target_types': 'mgs', 'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5', 'device_type': 'zfs', 'failnode': [u'10.14.81.225@tcp0']}
[2017-05-10 11:47:48,893: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2f0df696-d61a-4ad6-81d9-2951bedeb8cb
[2017-05-10 11:47:48,893: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2f0df696-d61a-4ad6-81d9-2951bedeb8cb
[2017-05-10 11:47:49,975: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:47:49,975: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:47:49,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:47:49,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:47:49,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3bd7dd43-abd0-47ff-b79b-65dec3c8041d
[2017-05-10 11:47:49,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3bd7dd43-abd0-47ff-b79b-65dec3c8041d
[2017-05-10 11:47:49,978: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:47:49,978: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:47:49,978: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:47:49,978: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:47:49,978: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f6539cc7-dd07-4bea-bda4-6333ea794c1b
[2017-05-10 11:47:49,978: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f6539cc7-dd07-4bea-bda4-6333ea794c1b
[2017-05-10 11:47:49,982: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f6539cc7-dd07-4bea-bda4-6333ea794c1b
[2017-05-10 11:47:49,982: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f6539cc7-dd07-4bea-bda4-6333ea794c1b
[2017-05-10 11:47:49,984: INFO/job] testfs-OST0001 passed pre-format check, allowing subsequent reformats
[2017-05-10 11:47:49,992: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3bd7dd43-abd0-47ff-b79b-65dec3c8041d
[2017-05-10 11:47:49,992: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3bd7dd43-abd0-47ff-b79b-65dec3c8041d
[2017-05-10 11:47:49,996: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com check_block_device {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1', 'device_type': 'zfs'}
[2017-05-10 11:47:50,015: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 76345cda-69f1-4f4d-866a-40847068c18d
[2017-05-10 11:47:50,015: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 76345cda-69f1-4f4d-866a-40847068c18d
[2017-05-10 11:47:50,025: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com format_target {'index': 1, 'backfstype': 'zfs', 'target_name': u'testfs-OST0001', 'mgsnode': ((u'10.14.81.224@tcp0',), (u'10.14.81.225@tcp0',)), 'target_types': 'ost', 'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3', 'device_type': 'zfs', 'failnode': [u'10.14.81.226@tcp0'], 'fsname': u'testfs'}
[2017-05-10 11:47:50,041: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 080d1653-9f9d-4016-bf4c-1ad6511ab160
[2017-05-10 11:47:50,041: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 080d1653-9f9d-4016-bf4c-1ad6511ab160
[2017-05-10 11:47:50,054: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:47:50,054: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:47:50,055: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:47:50,055: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:47:50,055: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f77b775e-5fee-41a0-80b0-92a6adbc763f
[2017-05-10 11:47:50,055: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f77b775e-5fee-41a0-80b0-92a6adbc763f
[2017-05-10 11:47:50,055: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f77b775e-5fee-41a0-80b0-92a6adbc763f
[2017-05-10 11:47:50,055: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f77b775e-5fee-41a0-80b0-92a6adbc763f
[2017-05-10 11:47:50,126: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:47:50,126: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:47:50,126: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:47:50,126: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:47:50,127: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 76345cda-69f1-4f4d-866a-40847068c18d
[2017-05-10 11:47:50,127: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 76345cda-69f1-4f4d-866a-40847068c18d
[2017-05-10 11:47:50,141: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 76345cda-69f1-4f4d-866a-40847068c18d
[2017-05-10 11:47:50,141: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 76345cda-69f1-4f4d-866a-40847068c18d
[2017-05-10 11:47:50,142: INFO/job] testfs-OST0000 passed pre-format check, allowing subsequent reformats
[2017-05-10 11:47:50,209: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com format_target {'index': 0, 'backfstype': 'zfs', 'target_name': u'testfs-OST0000', 'mgsnode': ((u'10.14.81.224@tcp0',), (u'10.14.81.225@tcp0',)), 'target_types': 'ost', 'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1', 'device_type': 'zfs', 'failnode': [u'10.14.81.227@tcp0'], 'fsname': u'testfs'}
[2017-05-10 11:47:50,221: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d5665c87-02ab-4e5c-9aa9-339979965db4
[2017-05-10 11:47:50,221: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d5665c87-02ab-4e5c-9aa9-339979965db4
[2017-05-10 11:47:50,242: INFO/job] Updating mtm volume_nodes for target testfs-MDT0000
[2017-05-10 11:47:50,457: INFO/job_scheduler] Job 56 finished 5 steps successfully
[2017-05-10 11:47:50,774: INFO/job] StatefulObject.set_state testfs-MDT0000 unformatted->formatted (intentional=True) 139877555852560
[2017-05-10 11:47:50,784: INFO/job] Job 56: StateChangeJob complete, setting state formatted on testfs-MDT0000
[2017-05-10 11:47:50,784: INFO/job] Job 56: not_deleted=True
[2017-05-10 11:47:50,795: INFO/job_scheduler] Job 56: complete_job: Updating cache
[2017-05-10 11:47:50,795: INFO/job_scheduler] Job 56: locked_item 139877555852560 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 11:47:50,798: INFO/job_scheduler] Job 56: locked_item 139877757381968 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-10 11:47:50,801: INFO/job_scheduler] Job 56: locked_item 139877757380176 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-10 11:47:50,805: INFO/job_scheduler] Job 56 completing (errored=False, cancelled=False)
[2017-05-10 11:47:50,884: INFO/job_scheduler] run_next: 0 runnable jobs of (13 pending, 3 tasked)
[2017-05-10 11:47:58,020: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:47:58,020: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:47:58,020: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:47:58,020: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:47:58,020: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2f0df696-d61a-4ad6-81d9-2951bedeb8cb
[2017-05-10 11:47:58,020: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2f0df696-d61a-4ad6-81d9-2951bedeb8cb
[2017-05-10 11:47:58,065: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2f0df696-d61a-4ad6-81d9-2951bedeb8cb
[2017-05-10 11:47:58,065: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2f0df696-d61a-4ad6-81d9-2951bedeb8cb
[2017-05-10 11:47:58,111: INFO/job] Updating mtm volume_nodes for target MGS
[2017-05-10 11:47:58,204: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:47:58,204: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:47:58,205: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:47:58,205: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:47:58,205: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 080d1653-9f9d-4016-bf4c-1ad6511ab160
[2017-05-10 11:47:58,205: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 080d1653-9f9d-4016-bf4c-1ad6511ab160
[2017-05-10 11:47:58,226: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 080d1653-9f9d-4016-bf4c-1ad6511ab160
[2017-05-10 11:47:58,226: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 080d1653-9f9d-4016-bf4c-1ad6511ab160
[2017-05-10 11:47:58,403: INFO/job] Updating mtm volume_nodes for target testfs-OST0001
[2017-05-10 11:47:59,435: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:47:59,435: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:47:59,435: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:47:59,435: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:47:59,435: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d5665c87-02ab-4e5c-9aa9-339979965db4
[2017-05-10 11:47:59,435: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d5665c87-02ab-4e5c-9aa9-339979965db4
[2017-05-10 11:47:59,443: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d5665c87-02ab-4e5c-9aa9-339979965db4
[2017-05-10 11:47:59,443: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d5665c87-02ab-4e5c-9aa9-339979965db4
[2017-05-10 11:47:59,495: INFO/job] Updating mtm volume_nodes for target testfs-OST0000
[2017-05-10 11:48:43,028: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 11:48:43,028: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 11:48:43,076: INFO/command_plan] add_jobs: created Job 72 (Trigger plugin poll for linux plugins)
[2017-05-10 11:48:43,151: INFO/job_scheduler] run_next: 1 runnable jobs of (14 pending, 3 tasked)
[2017-05-10 11:48:43,181: INFO/job_scheduler] Job 72: RunJobThread.run
[2017-05-10 11:48:43,232: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 11:48:43,271: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 11:48:43,285: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 11:48:43,303: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a5abf61e-7c1e-43df-9bb4-30e9af396e31
[2017-05-10 11:48:43,303: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a5abf61e-7c1e-43df-9bb4-30e9af396e31
[2017-05-10 11:48:43,339: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 682a8305-36c6-4f7f-95c8-58a832efba3d
[2017-05-10 11:48:43,339: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 682a8305-36c6-4f7f-95c8-58a832efba3d
[2017-05-10 11:48:43,362: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 769f7a53-55bd-4abf-a79f-3de1baadbdff
[2017-05-10 11:48:43,362: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 769f7a53-55bd-4abf-a79f-3de1baadbdff
[2017-05-10 11:48:43,447: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:48:43,447: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:48:43,448: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:48:43,448: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:48:43,448: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a5abf61e-7c1e-43df-9bb4-30e9af396e31
[2017-05-10 11:48:43,448: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a5abf61e-7c1e-43df-9bb4-30e9af396e31
[2017-05-10 11:48:43,465: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a5abf61e-7c1e-43df-9bb4-30e9af396e31
[2017-05-10 11:48:43,465: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a5abf61e-7c1e-43df-9bb4-30e9af396e31
[2017-05-10 11:48:43,466: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 11:48:43,493: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:48:43,493: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:48:43,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:48:43,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:48:43,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 769f7a53-55bd-4abf-a79f-3de1baadbdff
[2017-05-10 11:48:43,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 769f7a53-55bd-4abf-a79f-3de1baadbdff
[2017-05-10 11:48:43,511: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:48:43,511: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:48:43,511: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:48:43,511: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:48:43,511: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 682a8305-36c6-4f7f-95c8-58a832efba3d
[2017-05-10 11:48:43,511: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 682a8305-36c6-4f7f-95c8-58a832efba3d
[2017-05-10 11:48:43,522: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 682a8305-36c6-4f7f-95c8-58a832efba3d
[2017-05-10 11:48:43,522: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 682a8305-36c6-4f7f-95c8-58a832efba3d
[2017-05-10 11:48:43,522: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 11:48:43,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 769f7a53-55bd-4abf-a79f-3de1baadbdff
[2017-05-10 11:48:43,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 769f7a53-55bd-4abf-a79f-3de1baadbdff
[2017-05-10 11:48:43,545: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 11:48:43,545: INFO/job_scheduler] Job 72 finished 1 steps successfully
[2017-05-10 11:48:43,601: INFO/job_scheduler] Job 72: complete_job: Updating cache
[2017-05-10 11:48:43,601: INFO/job_scheduler] Job 72 completing (errored=False, cancelled=False)
[2017-05-10 11:48:43,703: INFO/job_scheduler] run_next: 0 runnable jobs of (13 pending, 3 tasked)
[2017-05-10 11:48:45,252: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 11:48:45,253: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 11:48:45,277: INFO/command_plan] add_jobs: created Job 73 (Trigger plugin poll for linux plugins)
[2017-05-10 11:48:45,334: INFO/job_scheduler] run_next: 1 runnable jobs of (14 pending, 3 tasked)
[2017-05-10 11:48:45,351: INFO/job_scheduler] Job 73: RunJobThread.run
[2017-05-10 11:48:45,395: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 11:48:45,429: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 11:48:45,438: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f3d48304-2e44-490d-9a13-0917fc5086a3
[2017-05-10 11:48:45,438: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f3d48304-2e44-490d-9a13-0917fc5086a3
[2017-05-10 11:48:45,470: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 11:48:45,475: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ed0619e9-37ad-4d03-9651-44e4d5669f9c
[2017-05-10 11:48:45,475: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ed0619e9-37ad-4d03-9651-44e4d5669f9c
[2017-05-10 11:48:45,502: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 490ce2c6-decc-4b75-8fe7-2785a5e85f7a
[2017-05-10 11:48:45,502: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 490ce2c6-decc-4b75-8fe7-2785a5e85f7a
[2017-05-10 11:48:45,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:48:45,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:48:45,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:48:45,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:48:45,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f3d48304-2e44-490d-9a13-0917fc5086a3
[2017-05-10 11:48:45,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f3d48304-2e44-490d-9a13-0917fc5086a3
[2017-05-10 11:48:45,575: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f3d48304-2e44-490d-9a13-0917fc5086a3
[2017-05-10 11:48:45,575: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f3d48304-2e44-490d-9a13-0917fc5086a3
[2017-05-10 11:48:45,576: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 11:48:45,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:48:45,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:48:45,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:48:45,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:48:45,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ed0619e9-37ad-4d03-9651-44e4d5669f9c
[2017-05-10 11:48:45,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ed0619e9-37ad-4d03-9651-44e4d5669f9c
[2017-05-10 11:48:45,601: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:48:45,601: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:48:45,601: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:48:45,601: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:48:45,601: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 490ce2c6-decc-4b75-8fe7-2785a5e85f7a
[2017-05-10 11:48:45,601: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 490ce2c6-decc-4b75-8fe7-2785a5e85f7a
[2017-05-10 11:48:45,608: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ed0619e9-37ad-4d03-9651-44e4d5669f9c
[2017-05-10 11:48:45,608: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ed0619e9-37ad-4d03-9651-44e4d5669f9c
[2017-05-10 11:48:45,608: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 11:48:45,636: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 490ce2c6-decc-4b75-8fe7-2785a5e85f7a
[2017-05-10 11:48:45,636: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 490ce2c6-decc-4b75-8fe7-2785a5e85f7a
[2017-05-10 11:48:45,636: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 11:48:45,643: INFO/job_scheduler] Job 73 finished 1 steps successfully
[2017-05-10 11:48:45,676: INFO/job_scheduler] Job 73: complete_job: Updating cache
[2017-05-10 11:48:45,677: INFO/job_scheduler] Job 73 completing (errored=False, cancelled=False)
[2017-05-10 11:48:45,806: INFO/job_scheduler] run_next: 0 runnable jobs of (13 pending, 3 tasked)
[2017-05-10 11:48:52,765: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 11:48:52,765: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 11:48:52,784: INFO/command_plan] add_jobs: created Job 74 (Trigger plugin poll for linux plugins)
[2017-05-10 11:48:52,793: INFO/job_scheduler] run_next: 1 runnable jobs of (14 pending, 3 tasked)
[2017-05-10 11:48:52,821: INFO/job_scheduler] Job 74: RunJobThread.run
[2017-05-10 11:48:52,836: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 11:48:52,864: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 48658cdf-0180-4c39-abf4-45ca0c7903ee
[2017-05-10 11:48:52,864: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 48658cdf-0180-4c39-abf4-45ca0c7903ee
[2017-05-10 11:48:52,869: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 11:48:52,912: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 11:48:52,931: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b38f0e79-4605-445b-95ee-afad38bfa412
[2017-05-10 11:48:52,931: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b38f0e79-4605-445b-95ee-afad38bfa412
[2017-05-10 11:48:52,958: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ba44e171-4780-41b7-bb3b-68b6be7314ac
[2017-05-10 11:48:52,958: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ba44e171-4780-41b7-bb3b-68b6be7314ac
[2017-05-10 11:48:52,968: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:48:52,968: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:48:52,969: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:48:52,969: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:48:52,969: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 48658cdf-0180-4c39-abf4-45ca0c7903ee
[2017-05-10 11:48:52,969: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 48658cdf-0180-4c39-abf4-45ca0c7903ee
[2017-05-10 11:48:52,995: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 48658cdf-0180-4c39-abf4-45ca0c7903ee
[2017-05-10 11:48:52,995: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 48658cdf-0180-4c39-abf4-45ca0c7903ee
[2017-05-10 11:48:52,995: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 11:48:52,997: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:48:52,997: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:48:52,997: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:48:52,997: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:48:52,997: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b38f0e79-4605-445b-95ee-afad38bfa412
[2017-05-10 11:48:52,997: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b38f0e79-4605-445b-95ee-afad38bfa412
[2017-05-10 11:48:53,009: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b38f0e79-4605-445b-95ee-afad38bfa412
[2017-05-10 11:48:53,009: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b38f0e79-4605-445b-95ee-afad38bfa412
[2017-05-10 11:48:53,009: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 11:48:53,068: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:48:53,068: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:48:53,068: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:48:53,068: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:48:53,068: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ba44e171-4780-41b7-bb3b-68b6be7314ac
[2017-05-10 11:48:53,068: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ba44e171-4780-41b7-bb3b-68b6be7314ac
[2017-05-10 11:48:53,078: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ba44e171-4780-41b7-bb3b-68b6be7314ac
[2017-05-10 11:48:53,078: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ba44e171-4780-41b7-bb3b-68b6be7314ac
[2017-05-10 11:48:53,079: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 11:48:53,079: INFO/job_scheduler] Job 74 finished 1 steps successfully
[2017-05-10 11:48:53,224: INFO/job_scheduler] Job 74: complete_job: Updating cache
[2017-05-10 11:48:53,225: INFO/job_scheduler] Job 74 completing (errored=False, cancelled=False)
[2017-05-10 11:48:53,275: INFO/job_scheduler] run_next: 0 runnable jobs of (13 pending, 3 tasked)
[2017-05-10 11:48:53,742: INFO/job_scheduler] Job 57 finished 5 steps successfully
[2017-05-10 11:48:53,786: INFO/job] StatefulObject.set_state testfs-OST0001 unformatted->formatted (intentional=True) 139877555853840
[2017-05-10 11:48:53,791: INFO/job] Job 57: StateChangeJob complete, setting state formatted on testfs-OST0001
[2017-05-10 11:48:53,792: INFO/job] Job 57: not_deleted=True
[2017-05-10 11:48:53,809: INFO/job_scheduler] Job 57: complete_job: Updating cache
[2017-05-10 11:48:53,810: INFO/job_scheduler] Job 57: locked_item 139877555853840 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 11:48:53,822: INFO/job_scheduler] Job 57: locked_item 139877556168016 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-10 11:48:53,830: INFO/job_scheduler] Job 57: locked_item 139877556168208 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-10 11:48:53,833: INFO/job_scheduler] Job 55 finished 5 steps successfully
[2017-05-10 11:48:53,836: INFO/job_scheduler] Job 57 completing (errored=False, cancelled=False)
[2017-05-10 11:48:53,925: INFO/job_scheduler] run_next: 0 runnable jobs of (13 pending, 2 tasked)
[2017-05-10 11:48:53,934: INFO/job] StatefulObject.set_state testfs-OST0000 unformatted->formatted (intentional=True) 139877219988176
[2017-05-10 11:48:53,958: INFO/job] Job 55: StateChangeJob complete, setting state formatted on testfs-OST0000
[2017-05-10 11:48:53,958: INFO/job] Job 55: not_deleted=True
[2017-05-10 11:48:53,984: INFO/job_scheduler] Job 55: complete_job: Updating cache
[2017-05-10 11:48:53,984: INFO/job_scheduler] Job 55: locked_item 139877219988176 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 11:48:53,997: INFO/job_scheduler] Job 55: locked_item 139877556164624 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-10 11:48:54,002: INFO/job_scheduler] Job 55: locked_item 139877556164880 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-10 11:48:54,013: INFO/job_scheduler] Job 55 completing (errored=False, cancelled=False)
[2017-05-10 11:48:54,084: INFO/job_scheduler] run_next: 0 runnable jobs of (13 pending, 1 tasked)
[2017-05-10 11:49:37,066: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 11:49:37,066: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 11:49:37,092: INFO/command_plan] add_jobs: created Job 75 (Trigger plugin poll for linux plugins)
[2017-05-10 11:49:37,144: INFO/job_scheduler] run_next: 1 runnable jobs of (14 pending, 1 tasked)
[2017-05-10 11:49:37,162: INFO/job_scheduler] Job 75: RunJobThread.run
[2017-05-10 11:49:37,166: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 11:49:37,180: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc be49d3e9-c4a3-4596-9fdb-22b4da17735b
[2017-05-10 11:49:37,180: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc be49d3e9-c4a3-4596-9fdb-22b4da17735b
[2017-05-10 11:49:37,180: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 11:49:37,196: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fd851c1f-a551-4a23-8332-bb7fdfdc31ea
[2017-05-10 11:49:37,196: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fd851c1f-a551-4a23-8332-bb7fdfdc31ea
[2017-05-10 11:49:37,226: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 11:49:37,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a8a7a496-f2a8-439e-9f57-8b373123d44c
[2017-05-10 11:49:37,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a8a7a496-f2a8-439e-9f57-8b373123d44c
[2017-05-10 11:49:37,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:49:37,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:49:37,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:49:37,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:49:37,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc be49d3e9-c4a3-4596-9fdb-22b4da17735b
[2017-05-10 11:49:37,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc be49d3e9-c4a3-4596-9fdb-22b4da17735b
[2017-05-10 11:49:37,280: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:49:37,280: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:49:37,280: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:49:37,280: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:49:37,280: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fd851c1f-a551-4a23-8332-bb7fdfdc31ea
[2017-05-10 11:49:37,280: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fd851c1f-a551-4a23-8332-bb7fdfdc31ea
[2017-05-10 11:49:37,314: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc be49d3e9-c4a3-4596-9fdb-22b4da17735b
[2017-05-10 11:49:37,314: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc be49d3e9-c4a3-4596-9fdb-22b4da17735b
[2017-05-10 11:49:37,314: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 11:49:37,319: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fd851c1f-a551-4a23-8332-bb7fdfdc31ea
[2017-05-10 11:49:37,319: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fd851c1f-a551-4a23-8332-bb7fdfdc31ea
[2017-05-10 11:49:37,319: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 11:49:37,357: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:49:37,357: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:49:37,357: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:49:37,357: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:49:37,357: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a8a7a496-f2a8-439e-9f57-8b373123d44c
[2017-05-10 11:49:37,357: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a8a7a496-f2a8-439e-9f57-8b373123d44c
[2017-05-10 11:49:37,392: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a8a7a496-f2a8-439e-9f57-8b373123d44c
[2017-05-10 11:49:37,392: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a8a7a496-f2a8-439e-9f57-8b373123d44c
[2017-05-10 11:49:37,393: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 11:49:37,394: INFO/job_scheduler] Job 75 finished 1 steps successfully
[2017-05-10 11:49:37,408: INFO/job_scheduler] Job 75: complete_job: Updating cache
[2017-05-10 11:49:37,408: INFO/job_scheduler] Job 75 completing (errored=False, cancelled=False)
[2017-05-10 11:49:37,483: INFO/job_scheduler] run_next: 0 runnable jobs of (13 pending, 1 tasked)
[2017-05-10 11:49:38,607: INFO/job_scheduler] Job 58 finished 5 steps successfully
[2017-05-10 11:49:38,666: INFO/job] StatefulObject.set_state MGS unformatted->formatted (intentional=True) 139877556201872
[2017-05-10 11:49:38,672: INFO/job] Job 58: StateChangeJob complete, setting state formatted on MGS
[2017-05-10 11:49:38,672: INFO/job] Job 58: not_deleted=True
[2017-05-10 11:49:38,679: INFO/job_scheduler] Job 58: complete_job: Updating cache
[2017-05-10 11:49:38,679: INFO/job_scheduler] Job 58: locked_item 139877556201872 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 11:49:38,682: INFO/job_scheduler] Job 58: locked_item 139877556169744 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-10 11:49:38,683: INFO/job_scheduler] Job 58: locked_item 139877556169936 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-10 11:49:38,684: INFO/job_scheduler] Job 58 completing (errored=False, cancelled=False)
[2017-05-10 11:49:38,716: INFO/job_scheduler] run_next: 1 runnable jobs of (13 pending, 0 tasked)
[2017-05-10 11:49:38,796: INFO/job_scheduler] Job 59: RunJobThread.run
[2017-05-10 11:49:38,798: INFO/job_scheduler] Job 59 finished 1 steps successfully
[2017-05-10 11:49:38,826: INFO/job] StatefulObject.set_state MGS formatted->registered (intentional=True) 139877556201872
[2017-05-10 11:49:38,837: INFO/job] Job 59: StateChangeJob complete, setting state registered on MGS
[2017-05-10 11:49:38,837: INFO/job] Job 59: not_deleted=True
[2017-05-10 11:49:38,837: INFO/job_scheduler] Job 59: complete_job: Updating cache
[2017-05-10 11:49:38,837: INFO/job_scheduler] Job 59: locked_item 139877556201872 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 11:49:38,842: INFO/job_scheduler] Job 59 completing (errored=False, cancelled=False)
[2017-05-10 11:49:38,875: INFO/job_scheduler] run_next: 1 runnable jobs of (12 pending, 0 tasked)
[2017-05-10 11:49:39,039: INFO/job_scheduler] Job 60: RunJobThread.run
[2017-05-10 11:49:39,039: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-05-10 11:49:39,092: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc acfc6299-f013-45f9-92e7-adb2396dece6
[2017-05-10 11:49:39,092: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc acfc6299-f013-45f9-92e7-adb2396dece6
[2017-05-10 11:49:40,207: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:49:40,207: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:49:40,207: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:49:40,207: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:49:40,207: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc acfc6299-f013-45f9-92e7-adb2396dece6
[2017-05-10 11:49:40,207: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc acfc6299-f013-45f9-92e7-adb2396dece6
[2017-05-10 11:49:40,215: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc acfc6299-f013-45f9-92e7-adb2396dece6
[2017-05-10 11:49:40,215: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc acfc6299-f013-45f9-92e7-adb2396dece6
[2017-05-10 11:49:40,216: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-05-10 11:49:40,231: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8607d52c-236f-4277-850c-ba632620d4f6
[2017-05-10 11:49:40,231: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8607d52c-236f-4277-850c-ba632620d4f6
[2017-05-10 11:49:41,294: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:49:41,294: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:49:41,294: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:49:41,294: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:49:41,295: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8607d52c-236f-4277-850c-ba632620d4f6
[2017-05-10 11:49:41,295: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8607d52c-236f-4277-850c-ba632620d4f6
[2017-05-10 11:49:41,303: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8607d52c-236f-4277-850c-ba632620d4f6
[2017-05-10 11:49:41,303: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8607d52c-236f-4277-850c-ba632620d4f6
[2017-05-10 11:49:41,305: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_target_store {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/MGS', 'device_type': 'zfs', 'mount_point': u'/mnt/MGS', 'uuid': u'3245615018417760886', 'backfstype': 'zfs'}
[2017-05-10 11:49:41,332: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9369c0bb-a3a5-4aaa-a3e6-39c6b5ee9ee6
[2017-05-10 11:49:41,332: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9369c0bb-a3a5-4aaa-a3e6-39c6b5ee9ee6
[2017-05-10 11:49:41,423: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:49:41,423: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:49:41,424: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:49:41,424: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:49:41,424: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9369c0bb-a3a5-4aaa-a3e6-39c6b5ee9ee6
[2017-05-10 11:49:41,424: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9369c0bb-a3a5-4aaa-a3e6-39c6b5ee9ee6
[2017-05-10 11:49:41,463: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9369c0bb-a3a5-4aaa-a3e6-39c6b5ee9ee6
[2017-05-10 11:49:41,463: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9369c0bb-a3a5-4aaa-a3e6-39c6b5ee9ee6
[2017-05-10 11:49:41,465: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_target_store {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/MGS', 'device_type': 'zfs', 'mount_point': u'/mnt/MGS', 'uuid': u'3245615018417760886', 'backfstype': 'zfs'}
[2017-05-10 11:49:41,486: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 59a55efa-ed16-46d3-89de-a70b613be2eb
[2017-05-10 11:49:41,486: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 59a55efa-ed16-46d3-89de-a70b613be2eb
[2017-05-10 11:49:41,594: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:49:41,594: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:49:41,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:49:41,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:49:41,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 59a55efa-ed16-46d3-89de-a70b613be2eb
[2017-05-10 11:49:41,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 59a55efa-ed16-46d3-89de-a70b613be2eb
[2017-05-10 11:49:41,603: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 59a55efa-ed16-46d3-89de-a70b613be2eb
[2017-05-10 11:49:41,603: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 59a55efa-ed16-46d3-89de-a70b613be2eb
[2017-05-10 11:49:41,605: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/MGS', 'mount_point': u'/mnt/MGS', 'uuid': u'3245615018417760886', 'ha_label': u'MGS_86cb98', 'primary': True}
[2017-05-10 11:49:41,625: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4d412294-669a-4522-95e5-35d1a2fb7ab2
[2017-05-10 11:49:41,625: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4d412294-669a-4522-95e5-35d1a2fb7ab2
[2017-05-10 11:49:44,701: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:49:44,701: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:49:44,701: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:49:44,701: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:49:44,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4d412294-669a-4522-95e5-35d1a2fb7ab2
[2017-05-10 11:49:44,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4d412294-669a-4522-95e5-35d1a2fb7ab2
[2017-05-10 11:49:44,744: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4d412294-669a-4522-95e5-35d1a2fb7ab2
[2017-05-10 11:49:44,744: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4d412294-669a-4522-95e5-35d1a2fb7ab2
[2017-05-10 11:49:44,745: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/MGS', 'mount_point': u'/mnt/MGS', 'uuid': u'3245615018417760886', 'ha_label': u'MGS_86cb98', 'primary': False}
[2017-05-10 11:49:44,758: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5c0f4063-93eb-4de0-9999-8d161cbfe631
[2017-05-10 11:49:44,758: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5c0f4063-93eb-4de0-9999-8d161cbfe631
[2017-05-10 11:49:45,858: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:49:45,858: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:49:45,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:49:45,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:49:45,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5c0f4063-93eb-4de0-9999-8d161cbfe631
[2017-05-10 11:49:45,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5c0f4063-93eb-4de0-9999-8d161cbfe631
[2017-05-10 11:49:45,872: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5c0f4063-93eb-4de0-9999-8d161cbfe631
[2017-05-10 11:49:45,872: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5c0f4063-93eb-4de0-9999-8d161cbfe631
[2017-05-10 11:49:45,873: INFO/job_scheduler] Job 60 finished 6 steps successfully
[2017-05-10 11:49:45,915: INFO/job] StatefulObject.set_state MGS registered->unmounted (intentional=True) 139877556201872
[2017-05-10 11:49:45,921: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on MGS at severity 30
[2017-05-10 11:49:45,930: INFO/job] Job 60: StateChangeJob complete, setting state unmounted on MGS
[2017-05-10 11:49:45,931: INFO/job] Job 60: not_deleted=True
[2017-05-10 11:49:45,932: INFO/job_scheduler] Job 60: complete_job: Updating cache
[2017-05-10 11:49:45,932: INFO/job_scheduler] Job 60: locked_item 139877556201872 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 11:49:45,948: INFO/job_scheduler] Job 60 completing (errored=False, cancelled=False)
[2017-05-10 11:49:45,974: INFO/job_scheduler] run_next: 1 runnable jobs of (11 pending, 0 tasked)
[2017-05-10 11:49:46,072: INFO/job_scheduler] Job 61: RunJobThread.run
[2017-05-10 11:49:46,083: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/MGS', 'device_type': 'zfs'}
[2017-05-10 11:49:46,104: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/MGS', 'device_type': 'zfs'}
[2017-05-10 11:49:46,122: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 08c24f15-0e65-4d91-995a-7e86222b0f29
[2017-05-10 11:49:46,122: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 08c24f15-0e65-4d91-995a-7e86222b0f29
[2017-05-10 11:49:46,144: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d08bc4fa-9368-4467-9a30-f32492cc7009
[2017-05-10 11:49:46,144: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d08bc4fa-9368-4467-9a30-f32492cc7009
[2017-05-10 11:49:46,159: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/MGS', 'device_type': 'zfs'}
[2017-05-10 11:49:46,175: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f94c323e-4842-4e82-9c3f-9282c5ca1df0
[2017-05-10 11:49:46,175: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f94c323e-4842-4e82-9c3f-9282c5ca1df0
[2017-05-10 11:49:47,279: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:49:47,279: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:49:47,279: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:49:47,279: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:49:47,280: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f94c323e-4842-4e82-9c3f-9282c5ca1df0
[2017-05-10 11:49:47,280: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f94c323e-4842-4e82-9c3f-9282c5ca1df0
[2017-05-10 11:49:47,289: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f94c323e-4842-4e82-9c3f-9282c5ca1df0
[2017-05-10 11:49:47,289: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f94c323e-4842-4e82-9c3f-9282c5ca1df0
[2017-05-10 11:49:48,051: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 1 tasked)
[2017-05-10 11:49:48,260: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:49:48,260: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:49:48,261: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:49:48,261: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:49:48,261: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d08bc4fa-9368-4467-9a30-f32492cc7009
[2017-05-10 11:49:48,261: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d08bc4fa-9368-4467-9a30-f32492cc7009
[2017-05-10 11:49:48,267: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d08bc4fa-9368-4467-9a30-f32492cc7009
[2017-05-10 11:49:48,267: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d08bc4fa-9368-4467-9a30-f32492cc7009
[2017-05-10 11:49:48,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:49:48,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:49:48,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:49:48,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:49:48,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 08c24f15-0e65-4d91-995a-7e86222b0f29
[2017-05-10 11:49:48,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 08c24f15-0e65-4d91-995a-7e86222b0f29
[2017-05-10 11:49:48,308: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 08c24f15-0e65-4d91-995a-7e86222b0f29
[2017-05-10 11:49:48,308: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 08c24f15-0e65-4d91-995a-7e86222b0f29
[2017-05-10 11:49:48,310: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com import_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/MGS', 'pacemaker_ha_operation': False, 'device_type': 'zfs', 'validate_importable': True}
[2017-05-10 11:49:48,353: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c694d75a-7247-4e47-ad28-d7468efb41a5
[2017-05-10 11:49:48,353: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c694d75a-7247-4e47-ad28-d7468efb41a5
[2017-05-10 11:49:50,463: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 1 tasked)
[2017-05-10 11:49:55,504: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:49:55,504: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:49:55,504: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:49:55,504: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:49:55,504: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c694d75a-7247-4e47-ad28-d7468efb41a5
[2017-05-10 11:49:55,504: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c694d75a-7247-4e47-ad28-d7468efb41a5
[2017-05-10 11:49:55,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c694d75a-7247-4e47-ad28-d7468efb41a5
[2017-05-10 11:49:55,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c694d75a-7247-4e47-ad28-d7468efb41a5
[2017-05-10 11:49:55,529: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_target {'ha_label': u'MGS_86cb98'}
[2017-05-10 11:49:55,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fca73c2e-2934-4b1d-b30b-e73b3b93294a
[2017-05-10 11:49:55,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fca73c2e-2934-4b1d-b30b-e73b3b93294a
[2017-05-10 11:50:05,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:50:05,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:50:05,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:50:05,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:50:05,692: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fca73c2e-2934-4b1d-b30b-e73b3b93294a
[2017-05-10 11:50:05,692: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fca73c2e-2934-4b1d-b30b-e73b3b93294a
[2017-05-10 11:50:05,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fca73c2e-2934-4b1d-b30b-e73b3b93294a
[2017-05-10 11:50:05,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fca73c2e-2934-4b1d-b30b-e73b3b93294a
[2017-05-10 11:50:05,720: INFO/job_scheduler] Job 61 finished 1 steps successfully
[2017-05-10 11:50:05,792: INFO/job] StatefulObject.set_state MGS unmounted->mounted (intentional=True) 139877556201872
[2017-05-10 11:50:05,842: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 20
[2017-05-10 11:50:05,849: INFO/job] Job 61: StateChangeJob complete, setting state mounted on MGS
[2017-05-10 11:50:05,849: INFO/job] Job 61: not_deleted=True
[2017-05-10 11:50:05,849: INFO/job_scheduler] Job 61: complete_job: Updating cache
[2017-05-10 11:50:05,850: INFO/job_scheduler] Job 61: locked_item 139877556201872 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 11:50:05,855: INFO/job_scheduler] Job 61 completing (errored=False, cancelled=False)
[2017-05-10 11:50:05,891: INFO/job_scheduler] run_next: 1 runnable jobs of (10 pending, 0 tasked)
[2017-05-10 11:50:05,988: INFO/job_scheduler] Job 62: RunJobThread.run
[2017-05-10 11:50:05,989: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com register_target {'device_path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'mount_point': u'/mnt/testfs-MDT0000', 'backfstype': 'ldiskfs'}
[2017-05-10 11:50:06,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8df4b616-6fef-4370-a376-45e057977691
[2017-05-10 11:50:06,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8df4b616-6fef-4370-a376-45e057977691
[2017-05-10 11:50:09,056: INFO/job_scheduler] run_next: 0 runnable jobs of (9 pending, 1 tasked)
[2017-05-10 11:50:11,132: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:11,132: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:11,133: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:11,133: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:11,133: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8df4b616-6fef-4370-a376-45e057977691
[2017-05-10 11:50:11,133: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8df4b616-6fef-4370-a376-45e057977691
[2017-05-10 11:50:11,181: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8df4b616-6fef-4370-a376-45e057977691
[2017-05-10 11:50:11,181: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8df4b616-6fef-4370-a376-45e057977691
[2017-05-10 11:50:11,183: INFO/job_scheduler] Job 62 finished 2 steps successfully
[2017-05-10 11:50:11,340: INFO/job] StatefulObject.set_state testfs-MDT0000 formatted->registered (intentional=True) 139877555852560
[2017-05-10 11:50:11,345: INFO/job] Job 62: StateChangeJob complete, setting state registered on testfs-MDT0000
[2017-05-10 11:50:11,345: INFO/job] Job 62: not_deleted=True
[2017-05-10 11:50:11,345: INFO/job_scheduler] Job 62: complete_job: Updating cache
[2017-05-10 11:50:11,346: INFO/job_scheduler] Job 62: locked_item 139877555852560 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 11:50:11,349: INFO/job_scheduler] Job 62 completing (errored=False, cancelled=False)
[2017-05-10 11:50:11,375: INFO/job_scheduler] run_next: 1 runnable jobs of (9 pending, 0 tasked)
[2017-05-10 11:50:11,435: INFO/job_scheduler] Job 63: RunJobThread.run
[2017-05-10 11:50:11,435: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-05-10 11:50:11,453: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2a81a90a-351c-4eab-a03c-36e8bb17fe52
[2017-05-10 11:50:11,453: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2a81a90a-351c-4eab-a03c-36e8bb17fe52
[2017-05-10 11:50:12,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:12,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:12,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:12,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:12,527: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2a81a90a-351c-4eab-a03c-36e8bb17fe52
[2017-05-10 11:50:12,527: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2a81a90a-351c-4eab-a03c-36e8bb17fe52
[2017-05-10 11:50:12,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2a81a90a-351c-4eab-a03c-36e8bb17fe52
[2017-05-10 11:50:12,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2a81a90a-351c-4eab-a03c-36e8bb17fe52
[2017-05-10 11:50:12,529: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-05-10 11:50:12,573: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1da04b38-ff9e-4250-a7fa-3054947fb785
[2017-05-10 11:50:12,573: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1da04b38-ff9e-4250-a7fa-3054947fb785
[2017-05-10 11:50:13,670: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:50:13,670: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:50:13,670: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:50:13,670: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:50:13,670: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1da04b38-ff9e-4250-a7fa-3054947fb785
[2017-05-10 11:50:13,670: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1da04b38-ff9e-4250-a7fa-3054947fb785
[2017-05-10 11:50:13,696: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1da04b38-ff9e-4250-a7fa-3054947fb785
[2017-05-10 11:50:13,696: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1da04b38-ff9e-4250-a7fa-3054947fb785
[2017-05-10 11:50:13,698: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_target_store {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux', 'mount_point': u'/mnt/testfs-MDT0000', 'uuid': u'40e3d7c3-097f-40f6-8438-12ea26f40592', 'backfstype': 'ldiskfs'}
[2017-05-10 11:50:13,721: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 70dd004c-b128-4cbd-9284-70910702452f
[2017-05-10 11:50:13,721: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 70dd004c-b128-4cbd-9284-70910702452f
[2017-05-10 11:50:13,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:13,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:13,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:13,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:13,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 70dd004c-b128-4cbd-9284-70910702452f
[2017-05-10 11:50:13,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 70dd004c-b128-4cbd-9284-70910702452f
[2017-05-10 11:50:13,897: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 70dd004c-b128-4cbd-9284-70910702452f
[2017-05-10 11:50:13,897: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 70dd004c-b128-4cbd-9284-70910702452f
[2017-05-10 11:50:13,900: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_target_store {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux', 'mount_point': u'/mnt/testfs-MDT0000', 'uuid': u'40e3d7c3-097f-40f6-8438-12ea26f40592', 'backfstype': 'ldiskfs'}
[2017-05-10 11:50:13,958: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c402cb94-bc59-4443-bc0c-0a8d9b70478c
[2017-05-10 11:50:13,958: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c402cb94-bc59-4443-bc0c-0a8d9b70478c
[2017-05-10 11:50:14,038: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:50:14,038: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:50:14,038: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:50:14,038: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:50:14,039: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c402cb94-bc59-4443-bc0c-0a8d9b70478c
[2017-05-10 11:50:14,039: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c402cb94-bc59-4443-bc0c-0a8d9b70478c
[2017-05-10 11:50:14,056: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c402cb94-bc59-4443-bc0c-0a8d9b70478c
[2017-05-10 11:50:14,056: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c402cb94-bc59-4443-bc0c-0a8d9b70478c
[2017-05-10 11:50:14,081: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'mount_point': u'/mnt/testfs-MDT0000', 'uuid': u'40e3d7c3-097f-40f6-8438-12ea26f40592', 'ha_label': u'testfs-MDT0000_de50dc', 'primary': True}
[2017-05-10 11:50:14,100: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a4db1e62-3754-4f92-8618-0f7257e6737d
[2017-05-10 11:50:14,100: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a4db1e62-3754-4f92-8618-0f7257e6737d
[2017-05-10 11:50:14,499: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 1 tasked)
[2017-05-10 11:50:17,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:17,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:17,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:17,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:17,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a4db1e62-3754-4f92-8618-0f7257e6737d
[2017-05-10 11:50:17,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a4db1e62-3754-4f92-8618-0f7257e6737d
[2017-05-10 11:50:17,274: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a4db1e62-3754-4f92-8618-0f7257e6737d
[2017-05-10 11:50:17,274: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a4db1e62-3754-4f92-8618-0f7257e6737d
[2017-05-10 11:50:17,276: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'mount_point': u'/mnt/testfs-MDT0000', 'uuid': u'40e3d7c3-097f-40f6-8438-12ea26f40592', 'ha_label': u'testfs-MDT0000_de50dc', 'primary': False}
[2017-05-10 11:50:17,295: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0ffebf84-b662-438e-bb5c-c652221d7ba1
[2017-05-10 11:50:17,295: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0ffebf84-b662-438e-bb5c-c652221d7ba1
[2017-05-10 11:50:18,364: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:50:18,364: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:50:18,364: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:50:18,364: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:50:18,364: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0ffebf84-b662-438e-bb5c-c652221d7ba1
[2017-05-10 11:50:18,364: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0ffebf84-b662-438e-bb5c-c652221d7ba1
[2017-05-10 11:50:18,409: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0ffebf84-b662-438e-bb5c-c652221d7ba1
[2017-05-10 11:50:18,409: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0ffebf84-b662-438e-bb5c-c652221d7ba1
[2017-05-10 11:50:18,410: INFO/job_scheduler] Job 63 finished 6 steps successfully
[2017-05-10 11:50:18,436: INFO/job] StatefulObject.set_state testfs-MDT0000 registered->unmounted (intentional=True) 139877555852560
[2017-05-10 11:50:18,441: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on testfs-MDT0000 at severity 30
[2017-05-10 11:50:18,462: INFO/job] Job 63: StateChangeJob complete, setting state unmounted on testfs-MDT0000
[2017-05-10 11:50:18,463: INFO/job] Job 63: not_deleted=True
[2017-05-10 11:50:18,463: INFO/job_scheduler] Job 63: complete_job: Updating cache
[2017-05-10 11:50:18,463: INFO/job_scheduler] Job 63: locked_item 139877555852560 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 11:50:18,467: INFO/job_scheduler] Job 63 completing (errored=False, cancelled=False)
[2017-05-10 11:50:18,566: INFO/job_scheduler] run_next: 1 runnable jobs of (8 pending, 0 tasked)
[2017-05-10 11:50:18,676: INFO/job_scheduler] Job 64: RunJobThread.run
[2017-05-10 11:50:18,703: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux'}
[2017-05-10 11:50:18,722: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc accc680d-e84d-4943-91b1-d4924133fff4
[2017-05-10 11:50:18,722: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc accc680d-e84d-4943-91b1-d4924133fff4
[2017-05-10 11:50:18,730: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux'}
[2017-05-10 11:50:18,752: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux'}
[2017-05-10 11:50:18,757: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f5723c48-2076-42a7-a173-424204e89147
[2017-05-10 11:50:18,757: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f5723c48-2076-42a7-a173-424204e89147
[2017-05-10 11:50:18,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b19f244e-cfd2-4bee-b930-0de36d04b1ef
[2017-05-10 11:50:18,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b19f244e-cfd2-4bee-b930-0de36d04b1ef
[2017-05-10 11:50:18,825: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:18,825: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:18,826: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:18,826: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:18,826: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc accc680d-e84d-4943-91b1-d4924133fff4
[2017-05-10 11:50:18,826: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc accc680d-e84d-4943-91b1-d4924133fff4
[2017-05-10 11:50:18,848: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:18,848: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:18,849: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:18,849: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:18,849: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b19f244e-cfd2-4bee-b930-0de36d04b1ef
[2017-05-10 11:50:18,849: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b19f244e-cfd2-4bee-b930-0de36d04b1ef
[2017-05-10 11:50:18,850: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b19f244e-cfd2-4bee-b930-0de36d04b1ef
[2017-05-10 11:50:18,850: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b19f244e-cfd2-4bee-b930-0de36d04b1ef
[2017-05-10 11:50:18,852: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:50:18,852: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:50:18,852: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:50:18,852: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:50:18,853: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f5723c48-2076-42a7-a173-424204e89147
[2017-05-10 11:50:18,853: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f5723c48-2076-42a7-a173-424204e89147
[2017-05-10 11:50:18,866: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc accc680d-e84d-4943-91b1-d4924133fff4
[2017-05-10 11:50:18,866: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc accc680d-e84d-4943-91b1-d4924133fff4
[2017-05-10 11:50:18,883: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f5723c48-2076-42a7-a173-424204e89147
[2017-05-10 11:50:18,883: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f5723c48-2076-42a7-a173-424204e89147
[2017-05-10 11:50:18,885: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com import_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'pacemaker_ha_operation': False, 'device_type': 'linux', 'validate_importable': True}
[2017-05-10 11:50:18,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc aaf44a88-3f44-4ffc-a172-905f0687d0fd
[2017-05-10 11:50:18,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc aaf44a88-3f44-4ffc-a172-905f0687d0fd
[2017-05-10 11:50:18,974: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:18,974: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:18,975: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:18,975: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:18,975: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc aaf44a88-3f44-4ffc-a172-905f0687d0fd
[2017-05-10 11:50:18,975: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc aaf44a88-3f44-4ffc-a172-905f0687d0fd
[2017-05-10 11:50:19,018: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc aaf44a88-3f44-4ffc-a172-905f0687d0fd
[2017-05-10 11:50:19,018: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc aaf44a88-3f44-4ffc-a172-905f0687d0fd
[2017-05-10 11:50:19,018: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_target {'ha_label': u'testfs-MDT0000_de50dc'}
[2017-05-10 11:50:19,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4bf8169e-f999-410f-9541-ca5a7ea295e0
[2017-05-10 11:50:19,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4bf8169e-f999-410f-9541-ca5a7ea295e0
[2017-05-10 11:50:19,198: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 1 tasked)
[2017-05-10 11:50:19,199: INFO/job_scheduler] notify: Dropping update of 2 (testfs-MDT0000) because it has been updated since
[2017-05-10 11:50:19,199: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 1 tasked)
[2017-05-10 11:50:20,565: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 1 tasked)
[2017-05-10 11:50:20,594: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 1 tasked)
[2017-05-10 11:50:25,209: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:25,209: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:25,210: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:25,210: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:25,210: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4bf8169e-f999-410f-9541-ca5a7ea295e0
[2017-05-10 11:50:25,210: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4bf8169e-f999-410f-9541-ca5a7ea295e0
[2017-05-10 11:50:25,219: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4bf8169e-f999-410f-9541-ca5a7ea295e0
[2017-05-10 11:50:25,219: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4bf8169e-f999-410f-9541-ca5a7ea295e0
[2017-05-10 11:50:25,220: INFO/job_scheduler] Job 64 finished 1 steps successfully
[2017-05-10 11:50:25,351: INFO/job] StatefulObject.set_state testfs-MDT0000 unmounted->mounted (intentional=True) 139877555852560
[2017-05-10 11:50:25,405: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm6.lotus.hpdd.lab.intel.com at severity 20
[2017-05-10 11:50:25,426: INFO/job] Job 64: StateChangeJob complete, setting state mounted on testfs-MDT0000
[2017-05-10 11:50:25,426: INFO/job] Job 64: not_deleted=True
[2017-05-10 11:50:25,426: INFO/job_scheduler] Job 64: complete_job: Updating cache
[2017-05-10 11:50:25,426: INFO/job_scheduler] Job 64: locked_item 139877555852560 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 11:50:25,434: INFO/job_scheduler] Job 64 completing (errored=False, cancelled=False)
[2017-05-10 11:50:25,500: INFO/job_scheduler] run_next: 2 runnable jobs of (7 pending, 0 tasked)
[2017-05-10 11:50:25,755: INFO/job_scheduler] Job 65: RunJobThread.run
[2017-05-10 11:50:25,757: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com register_target {'device_path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3/testfs-OST0001', 'mount_point': u'/mnt/testfs-OST0001', 'backfstype': 'zfs'}
[2017-05-10 11:50:25,794: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5454665d-f3dd-4502-aa4a-4641aaf66197
[2017-05-10 11:50:25,794: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5454665d-f3dd-4502-aa4a-4641aaf66197
[2017-05-10 11:50:25,798: INFO/job_scheduler] Job 66: RunJobThread.run
[2017-05-10 11:50:25,802: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com register_target {'device_path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/testfs-OST0000', 'mount_point': u'/mnt/testfs-OST0000', 'backfstype': 'zfs'}
[2017-05-10 11:50:25,877: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c88a81cc-7a7e-4fef-860f-200d2b698677
[2017-05-10 11:50:25,877: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c88a81cc-7a7e-4fef-860f-200d2b698677
[2017-05-10 11:50:29,091: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 2 tasked)
[2017-05-10 11:50:29,112: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 2 tasked)
[2017-05-10 11:50:30,656: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 2 tasked)
[2017-05-10 11:50:30,721: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 2 tasked)
[2017-05-10 11:50:37,554: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:37,554: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:37,569: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:37,569: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:37,569: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5454665d-f3dd-4502-aa4a-4641aaf66197
[2017-05-10 11:50:37,569: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5454665d-f3dd-4502-aa4a-4641aaf66197
[2017-05-10 11:50:37,593: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5454665d-f3dd-4502-aa4a-4641aaf66197
[2017-05-10 11:50:37,593: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5454665d-f3dd-4502-aa4a-4641aaf66197
[2017-05-10 11:50:37,596: INFO/job_scheduler] Job 65 finished 2 steps successfully
[2017-05-10 11:50:37,872: INFO/job] StatefulObject.set_state testfs-OST0001 formatted->registered (intentional=True) 139877555853840
[2017-05-10 11:50:37,882: INFO/job] Job 65: StateChangeJob complete, setting state registered on testfs-OST0001
[2017-05-10 11:50:37,884: INFO/job] Job 65: not_deleted=True
[2017-05-10 11:50:37,884: INFO/job_scheduler] Job 65: complete_job: Updating cache
[2017-05-10 11:50:37,884: INFO/job_scheduler] Job 65: locked_item 139877555853840 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 11:50:37,887: INFO/job_scheduler] Job 65 completing (errored=False, cancelled=False)
[2017-05-10 11:50:37,925: INFO/job_scheduler] run_next: 1 runnable jobs of (5 pending, 1 tasked)
[2017-05-10 11:50:38,082: INFO/job_scheduler] Job 68: RunJobThread.run
[2017-05-10 11:50:38,092: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-05-10 11:50:38,110: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 316bfd7c-61ed-454c-a5ab-6caa0b8b0fcc
[2017-05-10 11:50:38,110: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 316bfd7c-61ed-454c-a5ab-6caa0b8b0fcc
[2017-05-10 11:50:38,141: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-05-10 11:50:38,178: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-05-10 11:50:39,199: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:39,199: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:39,200: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:39,200: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:39,200: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 316bfd7c-61ed-454c-a5ab-6caa0b8b0fcc
[2017-05-10 11:50:39,200: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 316bfd7c-61ed-454c-a5ab-6caa0b8b0fcc
[2017-05-10 11:50:39,234: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 316bfd7c-61ed-454c-a5ab-6caa0b8b0fcc
[2017-05-10 11:50:39,234: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 316bfd7c-61ed-454c-a5ab-6caa0b8b0fcc
[2017-05-10 11:50:39,235: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-05-10 11:50:39,250: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 407a9a14-9698-4517-a07d-ad55d42cf236
[2017-05-10 11:50:39,250: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 407a9a14-9698-4517-a07d-ad55d42cf236
[2017-05-10 11:50:39,638: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-05-10 11:50:39,683: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-05-10 11:50:40,034: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:40,034: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:40,034: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:40,034: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:40,034: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c88a81cc-7a7e-4fef-860f-200d2b698677
[2017-05-10 11:50:40,034: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c88a81cc-7a7e-4fef-860f-200d2b698677
[2017-05-10 11:50:40,077: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c88a81cc-7a7e-4fef-860f-200d2b698677
[2017-05-10 11:50:40,077: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c88a81cc-7a7e-4fef-860f-200d2b698677
[2017-05-10 11:50:40,079: INFO/job_scheduler] Job 66 finished 2 steps successfully
[2017-05-10 11:50:40,241: INFO/job] StatefulObject.set_state testfs-OST0000 formatted->registered (intentional=True) 139877219988176
[2017-05-10 11:50:40,253: INFO/job] Job 66: StateChangeJob complete, setting state registered on testfs-OST0000
[2017-05-10 11:50:40,253: INFO/job] Job 66: not_deleted=True
[2017-05-10 11:50:40,253: INFO/job_scheduler] Job 66: complete_job: Updating cache
[2017-05-10 11:50:40,253: INFO/job_scheduler] Job 66: locked_item 139877219988176 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 11:50:40,256: INFO/job_scheduler] Job 66 completing (errored=False, cancelled=False)
[2017-05-10 11:50:40,298: INFO/job_scheduler] run_next: 1 runnable jobs of (4 pending, 1 tasked)
[2017-05-10 11:50:40,365: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:40,365: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:40,366: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:40,366: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:40,366: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 407a9a14-9698-4517-a07d-ad55d42cf236
[2017-05-10 11:50:40,366: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 407a9a14-9698-4517-a07d-ad55d42cf236
[2017-05-10 11:50:40,369: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 407a9a14-9698-4517-a07d-ad55d42cf236
[2017-05-10 11:50:40,369: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 407a9a14-9698-4517-a07d-ad55d42cf236
[2017-05-10 11:50:40,371: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_target_store {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3/testfs-OST0001', 'device_type': 'zfs', 'mount_point': u'/mnt/testfs-OST0001', 'uuid': u'7844491962501926787', 'backfstype': 'zfs'}
[2017-05-10 11:50:40,401: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc aa32b843-60bf-4cd7-9c0d-1ebdca084e21
[2017-05-10 11:50:40,401: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc aa32b843-60bf-4cd7-9c0d-1ebdca084e21
[2017-05-10 11:50:40,457: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:40,457: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:40,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:40,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:40,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc aa32b843-60bf-4cd7-9c0d-1ebdca084e21
[2017-05-10 11:50:40,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc aa32b843-60bf-4cd7-9c0d-1ebdca084e21
[2017-05-10 11:50:40,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc aa32b843-60bf-4cd7-9c0d-1ebdca084e21
[2017-05-10 11:50:40,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc aa32b843-60bf-4cd7-9c0d-1ebdca084e21
[2017-05-10 11:50:40,480: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_target_store {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3/testfs-OST0001', 'device_type': 'zfs', 'mount_point': u'/mnt/testfs-OST0001', 'uuid': u'7844491962501926787', 'backfstype': 'zfs'}
[2017-05-10 11:50:40,506: INFO/job_scheduler] Job 67: RunJobThread.run
[2017-05-10 11:50:40,506: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-05-10 11:50:40,511: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 28c30fda-b317-4974-9147-be2113c936d9
[2017-05-10 11:50:40,511: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 28c30fda-b317-4974-9147-be2113c936d9
[2017-05-10 11:50:40,544: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 96ec16d4-789a-48b4-b766-a09c675ca681
[2017-05-10 11:50:40,544: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 96ec16d4-789a-48b4-b766-a09c675ca681
[2017-05-10 11:50:40,553: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 2 tasked)
[2017-05-10 11:50:40,580: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 2 tasked)
[2017-05-10 11:50:40,606: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:40,606: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:40,607: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:40,607: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:40,607: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 28c30fda-b317-4974-9147-be2113c936d9
[2017-05-10 11:50:40,607: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 28c30fda-b317-4974-9147-be2113c936d9
[2017-05-10 11:50:40,632: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 28c30fda-b317-4974-9147-be2113c936d9
[2017-05-10 11:50:40,632: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 28c30fda-b317-4974-9147-be2113c936d9
[2017-05-10 11:50:40,634: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3/testfs-OST0001', 'mount_point': u'/mnt/testfs-OST0001', 'uuid': u'7844491962501926787', 'ha_label': u'testfs-OST0001_ec0738', 'primary': True}
[2017-05-10 11:50:40,667: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 90671160-a0db-461c-9b55-7c9936d3ec95
[2017-05-10 11:50:40,667: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 90671160-a0db-461c-9b55-7c9936d3ec95
[2017-05-10 11:50:40,702: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 2 tasked)
[2017-05-10 11:50:40,720: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 2 tasked)
[2017-05-10 11:50:40,728: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 2 tasked)
[2017-05-10 11:50:40,765: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 2 tasked)
[2017-05-10 11:50:41,620: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:41,620: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:41,621: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:41,621: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:41,621: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 96ec16d4-789a-48b4-b766-a09c675ca681
[2017-05-10 11:50:41,621: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 96ec16d4-789a-48b4-b766-a09c675ca681
[2017-05-10 11:50:41,658: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 96ec16d4-789a-48b4-b766-a09c675ca681
[2017-05-10 11:50:41,658: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 96ec16d4-789a-48b4-b766-a09c675ca681
[2017-05-10 11:50:41,659: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-05-10 11:50:41,679: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 44a95643-d9af-4a0f-b2a5-431e41719b94
[2017-05-10 11:50:41,679: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 44a95643-d9af-4a0f-b2a5-431e41719b94
[2017-05-10 11:50:42,764: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:42,764: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:42,765: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:42,765: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:42,765: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 44a95643-d9af-4a0f-b2a5-431e41719b94
[2017-05-10 11:50:42,765: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 44a95643-d9af-4a0f-b2a5-431e41719b94
[2017-05-10 11:50:42,799: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 44a95643-d9af-4a0f-b2a5-431e41719b94
[2017-05-10 11:50:42,799: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 44a95643-d9af-4a0f-b2a5-431e41719b94
[2017-05-10 11:50:42,801: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_target_store {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/testfs-OST0000', 'device_type': 'zfs', 'mount_point': u'/mnt/testfs-OST0000', 'uuid': u'7847514204723661707', 'backfstype': 'zfs'}
[2017-05-10 11:50:42,826: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 59090345-7770-4a87-a97f-5f8c1dc31e5d
[2017-05-10 11:50:42,826: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 59090345-7770-4a87-a97f-5f8c1dc31e5d
[2017-05-10 11:50:42,878: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 2 tasked)
[2017-05-10 11:50:42,938: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:42,938: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:42,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:42,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:42,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 59090345-7770-4a87-a97f-5f8c1dc31e5d
[2017-05-10 11:50:42,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 59090345-7770-4a87-a97f-5f8c1dc31e5d
[2017-05-10 11:50:42,943: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 59090345-7770-4a87-a97f-5f8c1dc31e5d
[2017-05-10 11:50:42,943: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 59090345-7770-4a87-a97f-5f8c1dc31e5d
[2017-05-10 11:50:42,945: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_target_store {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/testfs-OST0000', 'device_type': 'zfs', 'mount_point': u'/mnt/testfs-OST0000', 'uuid': u'7847514204723661707', 'backfstype': 'zfs'}
[2017-05-10 11:50:42,960: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 992593a8-f3fa-47d4-8ca5-69478ad2f2e9
[2017-05-10 11:50:42,960: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 992593a8-f3fa-47d4-8ca5-69478ad2f2e9
[2017-05-10 11:50:42,968: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 2 tasked)
[2017-05-10 11:50:43,012: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:43,012: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:43,012: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:43,012: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:43,012: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 992593a8-f3fa-47d4-8ca5-69478ad2f2e9
[2017-05-10 11:50:43,012: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 992593a8-f3fa-47d4-8ca5-69478ad2f2e9
[2017-05-10 11:50:43,033: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 992593a8-f3fa-47d4-8ca5-69478ad2f2e9
[2017-05-10 11:50:43,033: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 992593a8-f3fa-47d4-8ca5-69478ad2f2e9
[2017-05-10 11:50:43,036: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/testfs-OST0000', 'mount_point': u'/mnt/testfs-OST0000', 'uuid': u'7847514204723661707', 'ha_label': u'testfs-OST0000_42cd9c', 'primary': True}
[2017-05-10 11:50:43,055: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8c267dce-9269-4b6a-983d-90078783045f
[2017-05-10 11:50:43,055: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8c267dce-9269-4b6a-983d-90078783045f
[2017-05-10 11:50:43,794: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:43,794: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:43,794: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:43,794: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:43,794: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 90671160-a0db-461c-9b55-7c9936d3ec95
[2017-05-10 11:50:43,794: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 90671160-a0db-461c-9b55-7c9936d3ec95
[2017-05-10 11:50:43,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 90671160-a0db-461c-9b55-7c9936d3ec95
[2017-05-10 11:50:43,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 90671160-a0db-461c-9b55-7c9936d3ec95
[2017-05-10 11:50:43,843: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3/testfs-OST0001', 'mount_point': u'/mnt/testfs-OST0001', 'uuid': u'7844491962501926787', 'ha_label': u'testfs-OST0001_ec0738', 'primary': False}
[2017-05-10 11:50:43,878: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e7b1d9bc-a49b-489a-b543-20a0ea407ba3
[2017-05-10 11:50:43,878: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e7b1d9bc-a49b-489a-b543-20a0ea407ba3
[2017-05-10 11:50:44,970: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:44,970: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:44,971: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:44,971: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:44,971: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e7b1d9bc-a49b-489a-b543-20a0ea407ba3
[2017-05-10 11:50:44,971: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e7b1d9bc-a49b-489a-b543-20a0ea407ba3
[2017-05-10 11:50:44,993: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e7b1d9bc-a49b-489a-b543-20a0ea407ba3
[2017-05-10 11:50:44,993: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e7b1d9bc-a49b-489a-b543-20a0ea407ba3
[2017-05-10 11:50:44,993: INFO/job_scheduler] Job 68 finished 6 steps successfully
[2017-05-10 11:50:45,025: INFO/job] StatefulObject.set_state testfs-OST0001 registered->unmounted (intentional=True) 139877555853840
[2017-05-10 11:50:45,030: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on testfs-OST0001 at severity 30
[2017-05-10 11:50:45,045: INFO/job] Job 68: StateChangeJob complete, setting state unmounted on testfs-OST0001
[2017-05-10 11:50:45,045: INFO/job] Job 68: not_deleted=True
[2017-05-10 11:50:45,045: INFO/job_scheduler] Job 68: complete_job: Updating cache
[2017-05-10 11:50:45,045: INFO/job_scheduler] Job 68: locked_item 139877555853840 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 11:50:45,053: INFO/job_scheduler] Job 68 completing (errored=False, cancelled=False)
[2017-05-10 11:50:45,085: INFO/job_scheduler] run_next: 1 runnable jobs of (3 pending, 1 tasked)
[2017-05-10 11:50:45,204: INFO/job_scheduler] Job 70: RunJobThread.run
[2017-05-10 11:50:45,212: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3/testfs-OST0001', 'device_type': 'zfs'}
[2017-05-10 11:50:45,258: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d7f3d983-7827-4ca8-9fe9-297ac419d927
[2017-05-10 11:50:45,258: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d7f3d983-7827-4ca8-9fe9-297ac419d927
[2017-05-10 11:50:45,279: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3/testfs-OST0001', 'device_type': 'zfs'}
[2017-05-10 11:50:45,297: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3/testfs-OST0001', 'device_type': 'zfs'}
[2017-05-10 11:50:45,307: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c9442984-6dd2-463a-a2b2-25c49635cbf2
[2017-05-10 11:50:45,307: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c9442984-6dd2-463a-a2b2-25c49635cbf2
[2017-05-10 11:50:45,340: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2757fd30-e823-40ed-b455-4c8fb912fb34
[2017-05-10 11:50:45,340: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2757fd30-e823-40ed-b455-4c8fb912fb34
[2017-05-10 11:50:45,473: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:45,473: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:45,474: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:45,474: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:45,474: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c9442984-6dd2-463a-a2b2-25c49635cbf2
[2017-05-10 11:50:45,474: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c9442984-6dd2-463a-a2b2-25c49635cbf2
[2017-05-10 11:50:45,501: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c9442984-6dd2-463a-a2b2-25c49635cbf2
[2017-05-10 11:50:45,501: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c9442984-6dd2-463a-a2b2-25c49635cbf2
[2017-05-10 11:50:46,167: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:46,167: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:46,172: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:46,172: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:46,173: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8c267dce-9269-4b6a-983d-90078783045f
[2017-05-10 11:50:46,173: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8c267dce-9269-4b6a-983d-90078783045f
[2017-05-10 11:50:46,184: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8c267dce-9269-4b6a-983d-90078783045f
[2017-05-10 11:50:46,184: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8c267dce-9269-4b6a-983d-90078783045f
[2017-05-10 11:50:46,186: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/testfs-OST0000', 'mount_point': u'/mnt/testfs-OST0000', 'uuid': u'7847514204723661707', 'ha_label': u'testfs-OST0000_42cd9c', 'primary': False}
[2017-05-10 11:50:46,208: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9310723c-a4c1-46bb-9d94-642df83ebdb1
[2017-05-10 11:50:46,208: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9310723c-a4c1-46bb-9d94-642df83ebdb1
[2017-05-10 11:50:47,281: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:47,281: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:47,281: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:47,281: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:47,281: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9310723c-a4c1-46bb-9d94-642df83ebdb1
[2017-05-10 11:50:47,281: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9310723c-a4c1-46bb-9d94-642df83ebdb1
[2017-05-10 11:50:47,328: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9310723c-a4c1-46bb-9d94-642df83ebdb1
[2017-05-10 11:50:47,328: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9310723c-a4c1-46bb-9d94-642df83ebdb1
[2017-05-10 11:50:47,329: INFO/job_scheduler] Job 67 finished 6 steps successfully
[2017-05-10 11:50:47,351: INFO/job] StatefulObject.set_state testfs-OST0000 registered->unmounted (intentional=True) 139877219988176
[2017-05-10 11:50:47,356: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on testfs-OST0000 at severity 30
[2017-05-10 11:50:47,360: INFO/job] Job 67: StateChangeJob complete, setting state unmounted on testfs-OST0000
[2017-05-10 11:50:47,360: INFO/job] Job 67: not_deleted=True
[2017-05-10 11:50:47,360: INFO/job_scheduler] Job 67: complete_job: Updating cache
[2017-05-10 11:50:47,360: INFO/job_scheduler] Job 67: locked_item 139877219988176 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 11:50:47,374: INFO/job_scheduler] Job 67 completing (errored=False, cancelled=False)
[2017-05-10 11:50:47,400: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 1 tasked)
[2017-05-10 11:50:47,408: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:50:47,408: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:50:47,408: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:50:47,408: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:50:47,408: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d7f3d983-7827-4ca8-9fe9-297ac419d927
[2017-05-10 11:50:47,408: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d7f3d983-7827-4ca8-9fe9-297ac419d927
[2017-05-10 11:50:47,430: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:47,430: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:47,430: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:47,430: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:47,431: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2757fd30-e823-40ed-b455-4c8fb912fb34
[2017-05-10 11:50:47,431: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2757fd30-e823-40ed-b455-4c8fb912fb34
[2017-05-10 11:50:47,435: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d7f3d983-7827-4ca8-9fe9-297ac419d927
[2017-05-10 11:50:47,435: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d7f3d983-7827-4ca8-9fe9-297ac419d927
[2017-05-10 11:50:47,460: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2757fd30-e823-40ed-b455-4c8fb912fb34
[2017-05-10 11:50:47,460: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2757fd30-e823-40ed-b455-4c8fb912fb34
[2017-05-10 11:50:47,461: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com import_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3/testfs-OST0001', 'pacemaker_ha_operation': False, 'device_type': 'zfs', 'validate_importable': True}
[2017-05-10 11:50:47,488: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c54de1c8-f108-4323-931b-c34ec39d05d8
[2017-05-10 11:50:47,488: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c54de1c8-f108-4323-931b-c34ec39d05d8
[2017-05-10 11:50:47,542: INFO/job_scheduler] Job 69: RunJobThread.run
[2017-05-10 11:50:47,545: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/testfs-OST0000', 'device_type': 'zfs'}
[2017-05-10 11:50:47,555: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/testfs-OST0000', 'device_type': 'zfs'}
[2017-05-10 11:50:47,560: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/testfs-OST0000', 'device_type': 'zfs'}
[2017-05-10 11:50:47,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d000f8a7-6bf6-47a0-ad13-65d4f8e9be27
[2017-05-10 11:50:47,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d000f8a7-6bf6-47a0-ad13-65d4f8e9be27
[2017-05-10 11:50:47,594: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f0c33400-d59b-464b-8205-24da3865306b
[2017-05-10 11:50:47,594: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f0c33400-d59b-464b-8205-24da3865306b
[2017-05-10 11:50:47,607: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ea089f65-9f27-4c21-8571-e3eb75e9994e
[2017-05-10 11:50:47,607: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ea089f65-9f27-4c21-8571-e3eb75e9994e
[2017-05-10 11:50:48,688: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:48,688: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:48,689: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:48,689: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:48,689: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f0c33400-d59b-464b-8205-24da3865306b
[2017-05-10 11:50:48,689: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f0c33400-d59b-464b-8205-24da3865306b
[2017-05-10 11:50:48,708: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f0c33400-d59b-464b-8205-24da3865306b
[2017-05-10 11:50:48,708: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f0c33400-d59b-464b-8205-24da3865306b
[2017-05-10 11:50:48,983: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 11:50:48,999: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 11:50:49,638: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 11:50:49,655: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 11:50:49,750: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:49,750: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:49,751: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:49,751: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bf20f713-9838-43a4-a69d-5a9e215b2280
[2017-05-10 11:50:49,751: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ea089f65-9f27-4c21-8571-e3eb75e9994e
[2017-05-10 11:50:49,751: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ea089f65-9f27-4c21-8571-e3eb75e9994e
[2017-05-10 11:50:49,783: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ea089f65-9f27-4c21-8571-e3eb75e9994e
[2017-05-10 11:50:49,783: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ea089f65-9f27-4c21-8571-e3eb75e9994e
[2017-05-10 11:50:50,660: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:50:50,660: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:50:50,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:50:50,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6ffeca0f-ef18-4443-b998-7f9a1e2c4c41
[2017-05-10 11:50:50,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d000f8a7-6bf6-47a0-ad13-65d4f8e9be27
[2017-05-10 11:50:50,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d000f8a7-6bf6-47a0-ad13-65d4f8e9be27
[2017-05-10 11:50:50,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d000f8a7-6bf6-47a0-ad13-65d4f8e9be27
[2017-05-10 11:50:50,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d000f8a7-6bf6-47a0-ad13-65d4f8e9be27
[2017-05-10 11:50:50,692: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com import_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/testfs-OST0000', 'pacemaker_ha_operation': False, 'device_type': 'zfs', 'validate_importable': True}
[2017-05-10 11:50:50,718: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f7c51c70-a391-4b28-867c-20b9a0d8b987
[2017-05-10 11:50:50,718: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f7c51c70-a391-4b28-867c-20b9a0d8b987
[2017-05-10 11:50:53,653: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:53,653: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:53,654: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:53,654: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:50:53,654: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c54de1c8-f108-4323-931b-c34ec39d05d8
[2017-05-10 11:50:53,654: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c54de1c8-f108-4323-931b-c34ec39d05d8
[2017-05-10 11:50:53,662: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c54de1c8-f108-4323-931b-c34ec39d05d8
[2017-05-10 11:50:53,662: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c54de1c8-f108-4323-931b-c34ec39d05d8
[2017-05-10 11:50:53,663: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_target {'ha_label': u'testfs-OST0001_ec0738'}
[2017-05-10 11:50:53,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 587eb93c-3a7c-4bf6-b068-59ff60455670
[2017-05-10 11:50:53,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 587eb93c-3a7c-4bf6-b068-59ff60455670
[2017-05-10 11:50:56,917: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:56,917: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:56,917: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:56,917: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:50:56,917: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f7c51c70-a391-4b28-867c-20b9a0d8b987
[2017-05-10 11:50:56,917: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f7c51c70-a391-4b28-867c-20b9a0d8b987
[2017-05-10 11:50:56,944: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f7c51c70-a391-4b28-867c-20b9a0d8b987
[2017-05-10 11:50:56,944: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f7c51c70-a391-4b28-867c-20b9a0d8b987
[2017-05-10 11:50:56,945: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_target {'ha_label': u'testfs-OST0000_42cd9c'}
[2017-05-10 11:50:57,013: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 40c3f913-a407-437e-bd53-870d48f8a1aa
[2017-05-10 11:50:57,013: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 40c3f913-a407-437e-bd53-870d48f8a1aa
[2017-05-10 11:51:07,861: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:51:07,861: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:51:07,861: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:51:07,861: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5a9a0014-56e9-43db-a382-382e2c642d9d
[2017-05-10 11:51:07,861: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 587eb93c-3a7c-4bf6-b068-59ff60455670
[2017-05-10 11:51:07,861: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 587eb93c-3a7c-4bf6-b068-59ff60455670
[2017-05-10 11:51:07,883: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 587eb93c-3a7c-4bf6-b068-59ff60455670
[2017-05-10 11:51:07,883: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 587eb93c-3a7c-4bf6-b068-59ff60455670
[2017-05-10 11:51:07,884: INFO/job_scheduler] Job 70 finished 1 steps successfully
[2017-05-10 11:51:08,041: INFO/job] StatefulObject.set_state testfs-OST0001 unmounted->mounted (intentional=True) 139877555853840
[2017-05-10 11:51:08,080: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm8.lotus.hpdd.lab.intel.com at severity 20
[2017-05-10 11:51:08,097: INFO/job] Job 70: StateChangeJob complete, setting state mounted on testfs-OST0001
[2017-05-10 11:51:08,098: INFO/job] Job 70: not_deleted=True
[2017-05-10 11:51:08,098: INFO/job_scheduler] Job 70: complete_job: Updating cache
[2017-05-10 11:51:08,098: INFO/job_scheduler] Job 70: locked_item 139877555853840 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 11:51:08,103: INFO/job_scheduler] Job 70 completing (errored=False, cancelled=False)
[2017-05-10 11:51:08,182: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 11:51:08,777: INFO/job_scheduler] notify: Dropping update of 4 (testfs-OST0001) because it has been updated since
[2017-05-10 11:51:08,777: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 11:51:08,887: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 11:51:11,057: INFO/job_scheduler] notify: Dropping update of 4 (testfs-OST0001) because it has been updated since
[2017-05-10 11:51:11,057: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 11:51:11,074: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 11:51:21,280: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:51:21,280: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:51:21,281: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:51:21,281: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/bfe5a5ab-75be-4d6c-ba4d-749db53c5886
[2017-05-10 11:51:21,281: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 40c3f913-a407-437e-bd53-870d48f8a1aa
[2017-05-10 11:51:21,281: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 40c3f913-a407-437e-bd53-870d48f8a1aa
[2017-05-10 11:51:21,320: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 40c3f913-a407-437e-bd53-870d48f8a1aa
[2017-05-10 11:51:21,320: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 40c3f913-a407-437e-bd53-870d48f8a1aa
[2017-05-10 11:51:21,322: INFO/job_scheduler] Job 69 finished 1 steps successfully
[2017-05-10 11:51:21,548: INFO/job] StatefulObject.set_state testfs-OST0000 unmounted->mounted (intentional=True) 139877219988176
[2017-05-10 11:51:21,591: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 20
[2017-05-10 11:51:21,602: INFO/job] Job 69: StateChangeJob complete, setting state mounted on testfs-OST0000
[2017-05-10 11:51:21,602: INFO/job] Job 69: not_deleted=True
[2017-05-10 11:51:21,602: INFO/job_scheduler] Job 69: complete_job: Updating cache
[2017-05-10 11:51:21,602: INFO/job_scheduler] Job 69: locked_item 139877219988176 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 11:51:21,612: INFO/job_scheduler] Job 69 completing (errored=False, cancelled=False)
[2017-05-10 11:51:21,727: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 11:51:21,831: INFO/job_scheduler] Job 71: open DB connection during completion
[2017-05-10 11:51:21,840: INFO/job] StatefulObject.set_state testfs unavailable->available (intentional=True) 139877555873104
[2017-05-10 11:51:21,843: INFO/job] Job 71: StateChangeJob complete, setting state available on testfs
[2017-05-10 11:51:21,843: INFO/job] Job 71: not_deleted=True
[2017-05-10 11:51:21,844: INFO/job_scheduler] Job 71: complete_job: Updating cache
[2017-05-10 11:51:21,844: INFO/job_scheduler] Job 71: locked_item 139877555873104 <class 'chroma_core.models.filesystem.ManagedFilesystem'> False True
[2017-05-10 11:51:21,854: INFO/job_scheduler] Job 71 completing (errored=False, cancelled=False)
[2017-05-10 11:51:21,960: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 11:51:28,886: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 11:51:28,929: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 11:51:29,173: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 11:51:29,252: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:00:49,863: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:00:49,902: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:00:50,787: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:00:50,830: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:00:51,012: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:00:51,042: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:00:51,575: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:00:51,594: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:00:52,644: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:00:52,673: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:00:52,829: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:00:52,856: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:00:53,646: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:00:53,674: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:00:54,799: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:00:54,827: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:11:02,560: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:11:02,609: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:11:02,721: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:11:02,757: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:11:03,733: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:11:03,781: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:11:03,813: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:11:03,834: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:11:04,510: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:11:04,546: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:11:04,655: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:11:04,689: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:11:05,016: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:11:05,035: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:11:06,993: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:11:07,010: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:21:13,189: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:21:13,211: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:21:14,517: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:21:14,559: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:21:14,722: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:21:14,756: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:21:15,418: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:21:15,433: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:21:16,553: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:21:16,578: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:21:17,069: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:21:17,092: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:21:17,319: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:21:17,357: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:21:19,092: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:21:19,124: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:31:24,954: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:31:24,995: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:31:26,307: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:31:26,336: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:31:26,439: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:31:26,470: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:31:27,403: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:31:27,418: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:31:28,069: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:31:28,149: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:31:28,417: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:31:28,448: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:31:29,264: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:31:29,286: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:31:31,146: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:31:31,174: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:41:36,699: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:41:36,733: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:41:38,116: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:41:38,143: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:41:38,282: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:41:38,319: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:41:39,311: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:41:39,333: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:41:39,870: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:41:39,914: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:41:40,256: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:41:40,274: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:41:41,399: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:41:41,440: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:41:43,042: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:41:43,072: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:51:48,416: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:51:48,448: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:51:49,884: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:51:49,915: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:51:51,206: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:51:51,256: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:51:51,362: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:51:51,376: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:51:51,679: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:51:51,719: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:51:52,032: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:51:52,050: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:51:53,456: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:51:53,477: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:51:54,976: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 12:51:54,993: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:02:00,130: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:02:00,178: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:02:01,614: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:02:01,639: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:02:02,287: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:02:02,312: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:02:03,329: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:02:03,361: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:02:03,616: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:02:03,773: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:02:03,971: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:02:03,997: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:02:05,508: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:02:05,525: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:02:06,886: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:02:06,908: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:12:11,921: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:12:11,964: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:12:13,355: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:12:13,385: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:12:14,214: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:12:14,251: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:12:15,117: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:12:15,138: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:12:15,421: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:12:15,453: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:12:15,834: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:12:15,858: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:12:17,631: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:12:17,649: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:12:18,955: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:12:18,976: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:19:31,728: INFO/chroma_service] Stopping job_scheduler
[2017-05-10 13:19:31,728: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-05-10 13:19:31,728: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-05-10 13:19:31,728: INFO/queue] Stopping ServiceQueue agent_action_runner_rx
[2017-05-10 13:19:32,079: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-05-10 13:19:32,079: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-05-10 13:19:32,080: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-05-10 13:19:32,080: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-05-10 13:19:32,080: INFO/job_scheduler] Stopping...
[2017-05-10 13:19:32,080: INFO/queue] Stopping ServiceQueue job_scheduler_notifications
[2017-05-10 13:19:32,080: INFO/job_scheduler] Joining...
[2017-05-10 13:19:32,744: INFO/job_scheduler] Complete.
[2017-05-10 13:19:32,744: INFO/chroma_service] Joining job_scheduler
[2017-05-10 13:19:32,744: INFO/job_scheduler] Cancelling outstanding jobs...
[2017-05-10 13:19:32,750: ERROR/chroma_service] Rogue thread still running, exiting hard
[2017-05-10 13:19:32,750: ERROR/chroma_service] ['Thread-66094', 'Thread-66132', 'Thread-66129', 'Thread-66125', 'Thread-66131', 'Thread-66124', 'MainThread', 'Thread-66088', 'Thread-66130']
[2017-05-10 13:21:20,883: INFO/queue] Purged 0 messages from 'job_scheduler_notifications' queue
[2017-05-10 13:21:20,959: INFO/queue] Purged 0 messages from 'agent_action_runner_rx' queue
[2017-05-10 13:21:21,263: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: None/None
[2017-05-10 13:21:21,263: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: None/None
[2017-05-10 13:27:09,389: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:27:09,391: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 13:27:09,403: INFO/command_plan] add_jobs: created Job 1 (Test for host connectivity)
[2017-05-10 13:27:09,438: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 13:27:09,454: INFO/job_scheduler] Job 1: RunJobThread.run
[2017-05-10 13:27:09,515: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:27:09,516: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 13:27:09,528: INFO/command_plan] add_jobs: created Job 2 (Test for host connectivity)
[2017-05-10 13:27:09,565: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 13:27:09,573: INFO/job_scheduler] Job 2: RunJobThread.run
[2017-05-10 13:27:09,752: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:27:09,754: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 13:27:09,770: INFO/command_plan] add_jobs: created Job 3 (Test for host connectivity)
[2017-05-10 13:27:09,855: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 13:27:09,866: INFO/job_scheduler] Job 3: RunJobThread.run
[2017-05-10 13:27:09,945: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:27:09,947: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 13:27:09,972: INFO/command_plan] add_jobs: created Job 4 (Test for host connectivity)
[2017-05-10 13:27:10,021: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 13:27:10,044: INFO/job_scheduler] Job 4: RunJobThread.run
[2017-05-10 13:27:13,443: INFO/job_scheduler] Job 1 finished 1 steps successfully
[2017-05-10 13:27:13,511: INFO/job_scheduler] Job 1: complete_job: Updating cache
[2017-05-10 13:27:13,512: INFO/job_scheduler] Job 1 completing (errored=False, cancelled=False)
[2017-05-10 13:27:13,553: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 13:27:13,754: INFO/job_scheduler] Job 2 finished 1 steps successfully
[2017-05-10 13:27:13,759: INFO/job_scheduler] Job 3 finished 1 steps successfully
[2017-05-10 13:27:13,779: INFO/job_scheduler] Job 4 finished 1 steps successfully
[2017-05-10 13:27:13,795: INFO/job_scheduler] Job 2: complete_job: Updating cache
[2017-05-10 13:27:13,795: INFO/job_scheduler] Job 2 completing (errored=False, cancelled=False)
[2017-05-10 13:27:13,828: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-05-10 13:27:13,836: INFO/job_scheduler] Job 3: complete_job: Updating cache
[2017-05-10 13:27:13,837: INFO/job_scheduler] Job 3 completing (errored=False, cancelled=False)
[2017-05-10 13:27:13,895: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 13:27:13,928: INFO/job_scheduler] Job 4: complete_job: Updating cache
[2017-05-10 13:27:13,928: INFO/job_scheduler] Job 4 completing (errored=False, cancelled=False)
[2017-05-10 13:27:13,999: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:27:16,632: INFO/job_scheduler] Getting FQDN for 'lotus-32vm5': lotus-32vm5
lotus-32vm5.lotus.hpdd.lab.intel.com

[2017-05-10 13:27:16,662: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:27:16,666: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-1 transitioning from undeployed to unconfigured
[2017-05-10 13:27:16,691: INFO/command_plan] Created command 5 (Deploying agent to host lotus-32vm5.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-05-10 13:27:16,736: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 13:27:16,799: INFO/job_scheduler] Job 5: RunJobThread.run
[2017-05-10 13:27:17,669: INFO/job_scheduler] Getting FQDN for 'lotus-32vm6.lotus.hpdd.lab.intel.com': lotus-32vm6
lotus-32vm6.lotus.hpdd.lab.intel.com

[2017-05-10 13:27:17,709: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:27:17,712: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-2 transitioning from undeployed to unconfigured
[2017-05-10 13:27:17,734: INFO/command_plan] Created command 6 (Deploying agent to host lotus-32vm6.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-05-10 13:27:17,766: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 13:27:17,816: INFO/job_scheduler] Job 6: RunJobThread.run
[2017-05-10 13:27:17,828: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-05-10 13:27:18,378: INFO/job_scheduler] Getting FQDN for 'lotus-32vm7.lotus.hpdd.lab.intel.com': lotus-32vm7
lotus-32vm7.lotus.hpdd.lab.intel.com

[2017-05-10 13:27:18,399: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:27:18,402: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-3 transitioning from undeployed to unconfigured
[2017-05-10 13:27:18,419: INFO/command_plan] Created command 7 (Deploying agent to host lotus-32vm7.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-05-10 13:27:18,570: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 13:27:18,629: INFO/job_scheduler] Job 7: RunJobThread.run
[2017-05-10 13:27:19,293: INFO/job_scheduler] Getting FQDN for 'lotus-32vm8.lotus.hpdd.lab.intel.com': lotus-32vm8
lotus-32vm8.lotus.hpdd.lab.intel.com

[2017-05-10 13:27:19,325: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:27:19,326: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-4 transitioning from undeployed to unconfigured
[2017-05-10 13:27:19,347: INFO/command_plan] Created command 8 (Deploying agent to host lotus-32vm8.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-05-10 13:27:19,360: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 13:27:19,376: INFO/job_scheduler] Job 8: RunJobThread.run
[2017-05-10 13:27:19,457: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 13:27:19,461: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-10 13:27:19,461: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm5.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-10 13:27:19,476: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 13:27:19,565: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/11eaa142-6754-4836-9a61-6e98ec38b5da
[2017-05-10 13:27:19,565: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/11eaa142-6754-4836-9a61-6e98ec38b5da
[2017-05-10 13:27:19,570: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:27:19,570: INFO/job_scheduler]   Job 5 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-10 13:27:19,570: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 13:27:20,088: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 13:27:20,463: INFO/job_scheduler] Job 5 finished 1 steps successfully
[2017-05-10 13:27:20,465: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:27:20,466: INFO/job_scheduler]   Job 6 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-10 13:27:20,466: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 13:27:20,481: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-10 13:27:20,481: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._send: no session yet for lotus-32vm6.lotus.hpdd.lab.intel.com, 360 seconds remain
[2017-05-10 13:27:20,483: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/b65a3669-157d-4016-b9a9-00975edbbcf5
[2017-05-10 13:27:20,483: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/b65a3669-157d-4016-b9a9-00975edbbcf5
[2017-05-10 13:27:20,525: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 140430735405840
[2017-05-10 13:27:20,533: INFO/job] Job 5: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:20,533: INFO/job] Job 5: not_deleted=True
[2017-05-10 13:27:20,533: INFO/job_scheduler] Job 5: complete_job: Updating cache
[2017-05-10 13:27:20,533: INFO/job_scheduler] Job 5: locked_item 140430735405840 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 13:27:20,536: INFO/job_scheduler] Job 5 completing (errored=False, cancelled=False)
[2017-05-10 13:27:20,612: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-1
[2017-05-10 13:27:20,613: INFO/job_scheduler] _notify: Updating .boot_time of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com) from None to 2017-05-03 16:42:07+00:00
[2017-05-10 13:27:20,625: INFO/job_scheduler] _notify: Set boot_time=2017-05-03 16:42:07+00:00 on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-1) and saved
[2017-05-10 13:27:20,641: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 13:27:21,112: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:27:21,112: INFO/job_scheduler]   Job 7 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-10 13:27:21,112: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 13:27:21,164: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/935860bf-cc23-4c03-87c9-fbf8b7010b84
[2017-05-10 13:27:21,164: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/935860bf-cc23-4c03-87c9-fbf8b7010b84
[2017-05-10 13:27:21,173: INFO/job_scheduler] Job 7 finished 1 steps successfully
[2017-05-10 13:27:21,217: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 140430600492112
[2017-05-10 13:27:21,222: INFO/job] Job 7: StateChangeJob complete, setting state unconfigured on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:21,222: INFO/job] Job 7: not_deleted=True
[2017-05-10 13:27:21,223: INFO/job_scheduler] Job 7: complete_job: Updating cache
[2017-05-10 13:27:21,223: INFO/job_scheduler] Job 7: locked_item 140430600492112 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 13:27:21,227: INFO/job_scheduler] Job 7 completing (errored=False, cancelled=False)
[2017-05-10 13:27:21,262: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-3
[2017-05-10 13:27:21,262: INFO/job_scheduler] _notify: Updating .boot_time of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com) from None to 2017-04-25 07:09:31+00:00
[2017-05-10 13:27:21,267: INFO/job_scheduler] _notify: Set boot_time=2017-04-25 07:09:31+00:00 on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-3) and saved
[2017-05-10 13:27:21,279: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-05-10 13:27:21,484: INFO/job_scheduler] Job 6 finished 1 steps successfully
[2017-05-10 13:27:21,529: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 140430600504720
[2017-05-10 13:27:21,536: INFO/job] Job 6: StateChangeJob complete, setting state unconfigured on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:21,537: INFO/job] Job 6: not_deleted=True
[2017-05-10 13:27:21,537: INFO/job_scheduler] Job 6: complete_job: Updating cache
[2017-05-10 13:27:21,537: INFO/job_scheduler] Job 6: locked_item 140430600504720 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 13:27:21,541: INFO/job_scheduler] Job 6 completing (errored=False, cancelled=False)
[2017-05-10 13:27:21,717: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/ec5f6d0a-457f-46b0-83d2-fb4d246b092f
[2017-05-10 13:27:21,717: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/ec5f6d0a-457f-46b0-83d2-fb4d246b092f
[2017-05-10 13:27:21,740: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-2
[2017-05-10 13:27:21,740: INFO/job_scheduler] _notify: Updating .boot_time of item 2 (lotus-32vm6.lotus.hpdd.lab.intel.com) from None to 2017-04-25 07:09:19+00:00
[2017-05-10 13:27:21,747: INFO/job_scheduler] _notify: Set boot_time=2017-04-25 07:09:19+00:00 on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-2) and saved
[2017-05-10 13:27:21,753: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 13:27:21,757: INFO/job_scheduler] Job 8 finished 1 steps successfully
[2017-05-10 13:27:21,845: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:27:21,845: INFO/job_scheduler]   Job 8 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-10 13:27:21,846: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 13:27:21,894: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 140430600543376
[2017-05-10 13:27:21,904: INFO/job] Job 8: StateChangeJob complete, setting state unconfigured on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:21,904: INFO/job] Job 8: not_deleted=True
[2017-05-10 13:27:21,904: INFO/job_scheduler] Job 8: complete_job: Updating cache
[2017-05-10 13:27:21,905: INFO/job_scheduler] Job 8: locked_item 140430600543376 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 13:27:21,907: INFO/job_scheduler] Job 8 completing (errored=False, cancelled=False)
[2017-05-10 13:27:21,963: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-4
[2017-05-10 13:27:21,963: INFO/job_scheduler] _notify: Updating .boot_time of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com) from None to 2017-04-25 07:09:56+00:00
[2017-05-10 13:27:21,968: INFO/job_scheduler] _notify: Set boot_time=2017-04-25 07:09:56+00:00 on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-4) and saved
[2017-05-10 13:27:21,974: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:27:29,047: INFO/job_scheduler] _notify: Updating .properties of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-05-10 13:27:29,061: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-1) and saved
[2017-05-10 13:27:29,068: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:27:29,203: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:27:30,048: INFO/job_scheduler] _notify: Updating .properties of item 2 (lotus-32vm6.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-05-10 13:27:30,053: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-2) and saved
[2017-05-10 13:27:30,059: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:27:30,237: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:27:30,376: INFO/job_scheduler] _notify: Updating .properties of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-05-10 13:27:30,383: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-4) and saved
[2017-05-10 13:27:30,393: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:27:30,706: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:27:30,920: INFO/job_scheduler] _notify: Updating .properties of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-05-10 13:27:30,946: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-3) and saved
[2017-05-10 13:27:30,951: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:27:31,085: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:27:32,378: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:27:32,382: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-1 transitioning from unconfigured to managed
[2017-05-10 13:27:32,624: INFO/command_plan] Created command 9 (Setup managed host lotus-32vm5.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-05-10 13:27:32,725: INFO/job_scheduler] run_next: 1 runnable jobs of (11 pending, 0 tasked)
[2017-05-10 13:27:32,767: INFO/job_scheduler] Job 9: RunJobThread.run
[2017-05-10 13:27:32,767: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-05-10 13:27:32,800: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc aeb37932-4748-4834-b604-7d1df2f6e80f
[2017-05-10 13:27:32,800: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc aeb37932-4748-4834-b604-7d1df2f6e80f
[2017-05-10 13:27:33,142: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:27:33,147: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-2 transitioning from unconfigured to managed
[2017-05-10 13:27:33,292: INFO/command_plan] Created command 10 (Setup managed host lotus-32vm6.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-05-10 13:27:33,367: INFO/job_scheduler] run_next: 1 runnable jobs of (21 pending, 1 tasked)
[2017-05-10 13:27:33,426: INFO/job_scheduler] Job 20: RunJobThread.run
[2017-05-10 13:27:33,426: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-05-10 13:27:33,475: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 96a1e79f-fcf2-4a0a-9899-fc4f5cc5479f
[2017-05-10 13:27:33,475: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 96a1e79f-fcf2-4a0a-9899-fc4f5cc5479f
[2017-05-10 13:27:33,567: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:27:33,574: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-3 transitioning from unconfigured to managed
[2017-05-10 13:27:33,772: INFO/command_plan] Created command 11 (Setup managed host lotus-32vm7.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-05-10 13:27:33,839: INFO/job_scheduler] run_next: 1 runnable jobs of (31 pending, 2 tasked)
[2017-05-10 13:27:33,901: INFO/job_scheduler] Job 31: RunJobThread.run
[2017-05-10 13:27:33,901: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-05-10 13:27:33,919: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2db14eb2-50c0-45fc-896f-cbb226e268db
[2017-05-10 13:27:33,919: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2db14eb2-50c0-45fc-896f-cbb226e268db
[2017-05-10 13:27:33,947: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/11eaa142-6754-4836-9a61-6e98ec38b5da
[2017-05-10 13:27:33,947: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/11eaa142-6754-4836-9a61-6e98ec38b5da
[2017-05-10 13:27:33,948: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/11eaa142-6754-4836-9a61-6e98ec38b5da
[2017-05-10 13:27:33,948: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/11eaa142-6754-4836-9a61-6e98ec38b5da
[2017-05-10 13:27:33,948: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc aeb37932-4748-4834-b604-7d1df2f6e80f
[2017-05-10 13:27:33,948: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc aeb37932-4748-4834-b604-7d1df2f6e80f
[2017-05-10 13:27:33,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc aeb37932-4748-4834-b604-7d1df2f6e80f
[2017-05-10 13:27:33,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc aeb37932-4748-4834-b604-7d1df2f6e80f
[2017-05-10 13:27:33,976: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-10 13:27:33.976526+00:00:
[2017-05-10 13:27:33,976: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-10 13:27:33.976526+00:00:
[2017-05-10 13:27:33,976: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 13:27:33,976: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 13:27:33,990: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:27:33,993: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-4 transitioning from unconfigured to managed
[2017-05-10 13:27:34,015: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-10 13:27:34.015081+00:00:
[2017-05-10 13:27:34,015: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-10 13:27:34.015081+00:00:
[2017-05-10 13:27:34,015: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 13:27:34,015: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 13:27:34,029: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com restart_agent {}
[2017-05-10 13:27:34,064: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7deb5297-8433-4541-94a8-e3537984a6b5
[2017-05-10 13:27:34,064: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7deb5297-8433-4541-94a8-e3537984a6b5
[2017-05-10 13:27:34,122: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/11eaa142-6754-4836-9a61-6e98ec38b5da
[2017-05-10 13:27:34,122: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/11eaa142-6754-4836-9a61-6e98ec38b5da
[2017-05-10 13:27:34,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/11eaa142-6754-4836-9a61-6e98ec38b5da
[2017-05-10 13:27:34,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/11eaa142-6754-4836-9a61-6e98ec38b5da
[2017-05-10 13:27:34,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7deb5297-8433-4541-94a8-e3537984a6b5
[2017-05-10 13:27:34,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7deb5297-8433-4541-94a8-e3537984a6b5
[2017-05-10 13:27:34,144: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7deb5297-8433-4541-94a8-e3537984a6b5
[2017-05-10 13:27:34,144: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7deb5297-8433-4541-94a8-e3537984a6b5
[2017-05-10 13:27:34,144: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old 11eaa142-6754-4836-9a61-6e98ec38b5da)
[2017-05-10 13:27:34,144: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old 11eaa142-6754-4836-9a61-6e98ec38b5da)
[2017-05-10 13:27:34,189: INFO/command_plan] Created command 12 (Setup managed host lotus-32vm8.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-05-10 13:27:34,206: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:27:34,206: INFO/job_scheduler]   Job 9 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 13:27:34,207: INFO/job_scheduler]   Job 10 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:34,207: INFO/job_scheduler]   Job 11 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:34,207: INFO/job_scheduler]   Job 12 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:34,207: INFO/job_scheduler]   Job 13 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:34,207: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:34,207: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:27:34,208: INFO/job_scheduler] run_next: 1 runnable jobs of (41 pending, 3 tasked)
[2017-05-10 13:27:34,265: INFO/job_scheduler] Job 42: RunJobThread.run
[2017-05-10 13:27:34,266: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-05-10 13:27:34,295: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 830a45cb-89a8-4ae5-bb79-11ab1a5b3949
[2017-05-10 13:27:34,295: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 830a45cb-89a8-4ae5-bb79-11ab1a5b3949
[2017-05-10 13:27:34,317: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 13:27:34,319: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:27:34,320: INFO/job_scheduler]   Job 9 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 13:27:34,320: INFO/job_scheduler]   Job 10 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:34,320: INFO/job_scheduler]   Job 11 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:34,320: INFO/job_scheduler]   Job 12 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:34,320: INFO/job_scheduler]   Job 13 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:34,320: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:34,320: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:27:34,321: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 13:27:34,544: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/b65a3669-157d-4016-b9a9-00975edbbcf5
[2017-05-10 13:27:34,544: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/b65a3669-157d-4016-b9a9-00975edbbcf5
[2017-05-10 13:27:34,544: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/b65a3669-157d-4016-b9a9-00975edbbcf5
[2017-05-10 13:27:34,544: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/b65a3669-157d-4016-b9a9-00975edbbcf5
[2017-05-10 13:27:34,544: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 96a1e79f-fcf2-4a0a-9899-fc4f5cc5479f
[2017-05-10 13:27:34,544: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 96a1e79f-fcf2-4a0a-9899-fc4f5cc5479f
[2017-05-10 13:27:34,547: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 96a1e79f-fcf2-4a0a-9899-fc4f5cc5479f
[2017-05-10 13:27:34,547: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 96a1e79f-fcf2-4a0a-9899-fc4f5cc5479f
[2017-05-10 13:27:34,547: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-05-10 13:27:34.547835+00:00:
[2017-05-10 13:27:34,547: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-05-10 13:27:34.547835+00:00:
[2017-05-10 13:27:34,548: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 13:27:34,548: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 13:27:34,562: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-05-10 13:27:34.562676+00:00:
[2017-05-10 13:27:34,562: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-05-10 13:27:34.562676+00:00:
[2017-05-10 13:27:34,562: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 13:27:34,562: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 13:27:34,575: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:27:34,575: INFO/job_scheduler]   Job 20 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 13:27:34,576: INFO/job_scheduler]   Job 21 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:34,576: INFO/job_scheduler]   Job 22 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:34,576: INFO/job_scheduler]   Job 23 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:34,576: INFO/job_scheduler]   Job 24 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:34,576: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:34,576: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:27:34,577: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 13:27:34,588: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:27:34,588: INFO/job_scheduler]   Job 20 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 13:27:34,588: INFO/job_scheduler]   Job 21 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:34,588: INFO/job_scheduler]   Job 22 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:34,588: INFO/job_scheduler]   Job 23 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:34,589: INFO/job_scheduler]   Job 24 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:34,589: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:34,589: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:27:34,589: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 13:27:34,591: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com restart_agent {}
[2017-05-10 13:27:34,606: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4b0d12aa-5bcf-41d5-a27f-40240af99f14
[2017-05-10 13:27:34,606: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4b0d12aa-5bcf-41d5-a27f-40240af99f14
[2017-05-10 13:27:34,686: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/b65a3669-157d-4016-b9a9-00975edbbcf5
[2017-05-10 13:27:34,686: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/b65a3669-157d-4016-b9a9-00975edbbcf5
[2017-05-10 13:27:34,686: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/b65a3669-157d-4016-b9a9-00975edbbcf5
[2017-05-10 13:27:34,686: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/b65a3669-157d-4016-b9a9-00975edbbcf5
[2017-05-10 13:27:34,686: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4b0d12aa-5bcf-41d5-a27f-40240af99f14
[2017-05-10 13:27:34,686: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4b0d12aa-5bcf-41d5-a27f-40240af99f14
[2017-05-10 13:27:34,726: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4b0d12aa-5bcf-41d5-a27f-40240af99f14
[2017-05-10 13:27:34,726: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4b0d12aa-5bcf-41d5-a27f-40240af99f14
[2017-05-10 13:27:34,726: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm6.lotus.hpdd.lab.intel.com (old b65a3669-157d-4016-b9a9-00975edbbcf5)
[2017-05-10 13:27:34,726: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm6.lotus.hpdd.lab.intel.com (old b65a3669-157d-4016-b9a9-00975edbbcf5)
[2017-05-10 13:27:35,012: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/935860bf-cc23-4c03-87c9-fbf8b7010b84
[2017-05-10 13:27:35,012: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/935860bf-cc23-4c03-87c9-fbf8b7010b84
[2017-05-10 13:27:35,013: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/935860bf-cc23-4c03-87c9-fbf8b7010b84
[2017-05-10 13:27:35,013: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/935860bf-cc23-4c03-87c9-fbf8b7010b84
[2017-05-10 13:27:35,013: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2db14eb2-50c0-45fc-896f-cbb226e268db
[2017-05-10 13:27:35,013: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2db14eb2-50c0-45fc-896f-cbb226e268db
[2017-05-10 13:27:35,037: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2db14eb2-50c0-45fc-896f-cbb226e268db
[2017-05-10 13:27:35,037: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2db14eb2-50c0-45fc-896f-cbb226e268db
[2017-05-10 13:27:35,038: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-05-10 13:27:35.038247+00:00:
[2017-05-10 13:27:35,038: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-05-10 13:27:35.038247+00:00:
[2017-05-10 13:27:35,038: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 13:27:35,038: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 13:27:35,051: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:27:35,051: INFO/job_scheduler]   Job 31 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 13:27:35,051: INFO/job_scheduler]   Job 32 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:35,051: INFO/job_scheduler]   Job 33 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:35,052: INFO/job_scheduler]   Job 34 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:35,052: INFO/job_scheduler]   Job 35 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:35,052: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:35,052: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:27:35,052: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 13:27:35,057: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-05-10 13:27:35.057330+00:00:
[2017-05-10 13:27:35,057: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-05-10 13:27:35.057330+00:00:
[2017-05-10 13:27:35,057: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 13:27:35,057: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 13:27:35,070: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:27:35,070: INFO/job_scheduler]   Job 31 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 13:27:35,071: INFO/job_scheduler]   Job 32 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:35,071: INFO/job_scheduler]   Job 33 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:35,071: INFO/job_scheduler]   Job 34 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:35,071: INFO/job_scheduler]   Job 35 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:35,071: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:35,071: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:27:35,072: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 13:27:35,076: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com restart_agent {}
[2017-05-10 13:27:35,089: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cea2f959-d0e9-489d-a31d-35f45af05ed4
[2017-05-10 13:27:35,089: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cea2f959-d0e9-489d-a31d-35f45af05ed4
[2017-05-10 13:27:35,135: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/935860bf-cc23-4c03-87c9-fbf8b7010b84
[2017-05-10 13:27:35,135: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/935860bf-cc23-4c03-87c9-fbf8b7010b84
[2017-05-10 13:27:35,135: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/935860bf-cc23-4c03-87c9-fbf8b7010b84
[2017-05-10 13:27:35,135: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/935860bf-cc23-4c03-87c9-fbf8b7010b84
[2017-05-10 13:27:35,135: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cea2f959-d0e9-489d-a31d-35f45af05ed4
[2017-05-10 13:27:35,135: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cea2f959-d0e9-489d-a31d-35f45af05ed4
[2017-05-10 13:27:35,157: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cea2f959-d0e9-489d-a31d-35f45af05ed4
[2017-05-10 13:27:35,157: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cea2f959-d0e9-489d-a31d-35f45af05ed4
[2017-05-10 13:27:35,157: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm7.lotus.hpdd.lab.intel.com (old 935860bf-cc23-4c03-87c9-fbf8b7010b84)
[2017-05-10 13:27:35,157: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm7.lotus.hpdd.lab.intel.com (old 935860bf-cc23-4c03-87c9-fbf8b7010b84)
[2017-05-10 13:27:35,355: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/ec5f6d0a-457f-46b0-83d2-fb4d246b092f
[2017-05-10 13:27:35,355: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/ec5f6d0a-457f-46b0-83d2-fb4d246b092f
[2017-05-10 13:27:35,355: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/ec5f6d0a-457f-46b0-83d2-fb4d246b092f
[2017-05-10 13:27:35,355: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/ec5f6d0a-457f-46b0-83d2-fb4d246b092f
[2017-05-10 13:27:35,355: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 830a45cb-89a8-4ae5-bb79-11ab1a5b3949
[2017-05-10 13:27:35,355: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 830a45cb-89a8-4ae5-bb79-11ab1a5b3949
[2017-05-10 13:27:35,361: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 830a45cb-89a8-4ae5-bb79-11ab1a5b3949
[2017-05-10 13:27:35,361: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 830a45cb-89a8-4ae5-bb79-11ab1a5b3949
[2017-05-10 13:27:35,361: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-05-10 13:27:35.361676+00:00:
[2017-05-10 13:27:35,361: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-05-10 13:27:35.361676+00:00:
[2017-05-10 13:27:35,361: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 13:27:35,361: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 13:27:35,374: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:27:35,374: INFO/job_scheduler]   Job 42 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 13:27:35,374: INFO/job_scheduler]   Job 43 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:35,374: INFO/job_scheduler]   Job 44 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:35,374: INFO/job_scheduler]   Job 45 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:35,374: INFO/job_scheduler]   Job 46 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:35,374: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:35,375: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:27:35,375: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 13:27:35,376: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-05-10 13:27:35.376756+00:00:
[2017-05-10 13:27:35,376: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-05-10 13:27:35.376756+00:00:
[2017-05-10 13:27:35,376: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 13:27:35,376: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 13:27:35,389: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:27:35,389: INFO/job_scheduler]   Job 42 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 13:27:35,389: INFO/job_scheduler]   Job 43 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:35,389: INFO/job_scheduler]   Job 44 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:35,389: INFO/job_scheduler]   Job 45 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:35,389: INFO/job_scheduler]   Job 46 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:35,389: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:27:35,390: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:27:35,390: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 13:27:35,391: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com restart_agent {}
[2017-05-10 13:27:35,403: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9ae61fc2-bd12-48a6-bfe0-7d4857125174
[2017-05-10 13:27:35,403: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9ae61fc2-bd12-48a6-bfe0-7d4857125174
[2017-05-10 13:27:35,474: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/ec5f6d0a-457f-46b0-83d2-fb4d246b092f
[2017-05-10 13:27:35,474: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/ec5f6d0a-457f-46b0-83d2-fb4d246b092f
[2017-05-10 13:27:35,475: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/ec5f6d0a-457f-46b0-83d2-fb4d246b092f
[2017-05-10 13:27:35,475: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/ec5f6d0a-457f-46b0-83d2-fb4d246b092f
[2017-05-10 13:27:35,475: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9ae61fc2-bd12-48a6-bfe0-7d4857125174
[2017-05-10 13:27:35,475: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9ae61fc2-bd12-48a6-bfe0-7d4857125174
[2017-05-10 13:27:35,524: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9ae61fc2-bd12-48a6-bfe0-7d4857125174
[2017-05-10 13:27:35,524: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9ae61fc2-bd12-48a6-bfe0-7d4857125174
[2017-05-10 13:27:35,524: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm8.lotus.hpdd.lab.intel.com (old ec5f6d0a-457f-46b0-83d2-fb4d246b092f)
[2017-05-10 13:27:35,524: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm8.lotus.hpdd.lab.intel.com (old ec5f6d0a-457f-46b0-83d2-fb4d246b092f)
[2017-05-10 13:28:02,490: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/b65a3669-157d-4016-b9a9-00975edbbcf5
[2017-05-10 13:28:02,490: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/b65a3669-157d-4016-b9a9-00975edbbcf5
[2017-05-10 13:28:02,490: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session b65a3669-157d-4016-b9a9-00975edbbcf5 because session terminated
[2017-05-10 13:28:02,490: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session b65a3669-157d-4016-b9a9-00975edbbcf5 because session terminated
[2017-05-10 13:28:02,493: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:28:02,493: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:28:02,755: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm6.lotus.hpdd.lab.intel.com new bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:28:02,755: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm6.lotus.hpdd.lab.intel.com new bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:28:02,802: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 13:28:02,825: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc beb50ab4-f028-4a1c-9cff-8ba89786de2d
[2017-05-10 13:28:02,825: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc beb50ab4-f028-4a1c-9cff-8ba89786de2d
[2017-05-10 13:28:02,888: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:28:02,888: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:28:02,888: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:28:02,888: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:28:02,888: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc beb50ab4-f028-4a1c-9cff-8ba89786de2d
[2017-05-10 13:28:02,888: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc beb50ab4-f028-4a1c-9cff-8ba89786de2d
[2017-05-10 13:28:02,889: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc beb50ab4-f028-4a1c-9cff-8ba89786de2d
[2017-05-10 13:28:02,889: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc beb50ab4-f028-4a1c-9cff-8ba89786de2d
[2017-05-10 13:28:02,889: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-05-10 13:28:02,902: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 585617e9-9f6a-4be2-88ae-05a3e58ca7bb
[2017-05-10 13:28:02,902: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 585617e9-9f6a-4be2-88ae-05a3e58ca7bb
[2017-05-10 13:28:03,027: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/935860bf-cc23-4c03-87c9-fbf8b7010b84
[2017-05-10 13:28:03,027: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/935860bf-cc23-4c03-87c9-fbf8b7010b84
[2017-05-10 13:28:03,028: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 935860bf-cc23-4c03-87c9-fbf8b7010b84 because session terminated
[2017-05-10 13:28:03,028: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 935860bf-cc23-4c03-87c9-fbf8b7010b84 because session terminated
[2017-05-10 13:28:03,031: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:28:03,031: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:28:03,188: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm7.lotus.hpdd.lab.intel.com new 97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:28:03,188: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm7.lotus.hpdd.lab.intel.com new 97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:28:03,189: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 13:28:03,208: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ac58dff7-cfb4-4e0d-8f8a-0925155f68c2
[2017-05-10 13:28:03,208: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ac58dff7-cfb4-4e0d-8f8a-0925155f68c2
[2017-05-10 13:28:03,257: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:28:03,257: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:28:03,257: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:28:03,257: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:28:03,258: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ac58dff7-cfb4-4e0d-8f8a-0925155f68c2
[2017-05-10 13:28:03,258: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ac58dff7-cfb4-4e0d-8f8a-0925155f68c2
[2017-05-10 13:28:03,276: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ac58dff7-cfb4-4e0d-8f8a-0925155f68c2
[2017-05-10 13:28:03,276: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ac58dff7-cfb4-4e0d-8f8a-0925155f68c2
[2017-05-10 13:28:03,276: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-05-10 13:28:03,309: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8071984a-d602-42a4-9e50-783519e778f3
[2017-05-10 13:28:03,309: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8071984a-d602-42a4-9e50-783519e778f3
[2017-05-10 13:28:03,983: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/11eaa142-6754-4836-9a61-6e98ec38b5da
[2017-05-10 13:28:03,983: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/11eaa142-6754-4836-9a61-6e98ec38b5da
[2017-05-10 13:28:03,984: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 11eaa142-6754-4836-9a61-6e98ec38b5da because session terminated
[2017-05-10 13:28:03,984: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 11eaa142-6754-4836-9a61-6e98ec38b5da because session terminated
[2017-05-10 13:28:03,986: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:28:03,986: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:28:04,180: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:28:04,180: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:28:04,181: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 13:28:04,200: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4bcb2210-19a4-4839-88df-c6829e03d255
[2017-05-10 13:28:04,200: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4bcb2210-19a4-4839-88df-c6829e03d255
[2017-05-10 13:28:04,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:28:04,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:28:04,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:28:04,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:28:04,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4bcb2210-19a4-4839-88df-c6829e03d255
[2017-05-10 13:28:04,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4bcb2210-19a4-4839-88df-c6829e03d255
[2017-05-10 13:28:04,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4bcb2210-19a4-4839-88df-c6829e03d255
[2017-05-10 13:28:04,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4bcb2210-19a4-4839-88df-c6829e03d255
[2017-05-10 13:28:04,322: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-05-10 13:28:04,337: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 225b0fb6-f420-4ec1-b756-7657b9ef4b36
[2017-05-10 13:28:04,337: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 225b0fb6-f420-4ec1-b756-7657b9ef4b36
[2017-05-10 13:28:05,359: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/ec5f6d0a-457f-46b0-83d2-fb4d246b092f
[2017-05-10 13:28:05,359: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/ec5f6d0a-457f-46b0-83d2-fb4d246b092f
[2017-05-10 13:28:05,360: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session ec5f6d0a-457f-46b0-83d2-fb4d246b092f because session terminated
[2017-05-10 13:28:05,360: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session ec5f6d0a-457f-46b0-83d2-fb4d246b092f because session terminated
[2017-05-10 13:28:05,361: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:28:05,361: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:28:05,556: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm8.lotus.hpdd.lab.intel.com new 91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:28:05,556: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm8.lotus.hpdd.lab.intel.com new 91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:28:05,678: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 13:28:05,694: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 753248aa-183d-4909-ab82-21994615ffce
[2017-05-10 13:28:05,694: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 753248aa-183d-4909-ab82-21994615ffce
[2017-05-10 13:28:05,847: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:28:05,847: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:28:05,847: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:28:05,847: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:28:05,847: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 753248aa-183d-4909-ab82-21994615ffce
[2017-05-10 13:28:05,847: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 753248aa-183d-4909-ab82-21994615ffce
[2017-05-10 13:28:05,860: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 753248aa-183d-4909-ab82-21994615ffce
[2017-05-10 13:28:05,860: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 753248aa-183d-4909-ab82-21994615ffce
[2017-05-10 13:28:05,860: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-05-10 13:28:05,874: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 33dd3d03-5dad-438c-a614-649efc5ae237
[2017-05-10 13:28:05,874: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 33dd3d03-5dad-438c-a614-649efc5ae237
[2017-05-10 13:28:12,762: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:28:12,762: INFO/job_scheduler]   Job 20 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 13:28:12,762: INFO/job_scheduler]   Job 21 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:28:12,762: INFO/job_scheduler]   Job 22 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:28:12,763: INFO/job_scheduler]   Job 23 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:28:12,763: INFO/job_scheduler]   Job 24 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:28:12,763: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:28:12,763: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:28:12,763: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 13:28:13,439: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:28:13,439: INFO/job_scheduler]   Job 31 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 13:28:13,440: INFO/job_scheduler]   Job 32 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:28:13,440: INFO/job_scheduler]   Job 33 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:28:13,440: INFO/job_scheduler]   Job 34 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:28:13,440: INFO/job_scheduler]   Job 35 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:28:13,440: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:28:13,440: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:28:13,441: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 13:28:15,301: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:28:15,302: INFO/job_scheduler]   Job 9 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 13:28:15,302: INFO/job_scheduler]   Job 10 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:28:15,302: INFO/job_scheduler]   Job 11 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:28:15,302: INFO/job_scheduler]   Job 12 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:28:15,302: INFO/job_scheduler]   Job 13 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:28:15,302: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:28:15,302: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:28:15,303: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 13:28:15,808: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:28:15,808: INFO/job_scheduler]   Job 42 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 13:28:15,808: INFO/job_scheduler]   Job 43 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:28:15,808: INFO/job_scheduler]   Job 44 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:28:15,809: INFO/job_scheduler]   Job 45 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:28:15,809: INFO/job_scheduler]   Job 46 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:28:15,809: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:28:15,809: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:28:15,809: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-05-10 13:28:44,663: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:28:44,663: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:28:44,663: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:28:44,663: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:28:44,663: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 585617e9-9f6a-4be2-88ae-05a3e58ca7bb
[2017-05-10 13:28:44,663: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 585617e9-9f6a-4be2-88ae-05a3e58ca7bb
[2017-05-10 13:28:44,695: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 585617e9-9f6a-4be2-88ae-05a3e58ca7bb
[2017-05-10 13:28:44,695: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 585617e9-9f6a-4be2-88ae-05a3e58ca7bb
[2017-05-10 13:28:45,210: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com install_packages {'repos': [u'e2fsprogs', u'iml-agent', u'lustre', u'zfs'], 'packages': [u'chroma-agent-management', u'kernel-devel-3.10.0-514.10.2.el7_lustre', u'lustre-osd-zfs', u'lustre-osd-ldiskfs', u'lustre-modules', u'lustre', u'zfs']}
[2017-05-10 13:28:45,223: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 15a63ef8-3d4b-417e-90cd-c5a0b0f9326c
[2017-05-10 13:28:45,223: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 15a63ef8-3d4b-417e-90cd-c5a0b0f9326c
[2017-05-10 13:28:52,385: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:28:52,385: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:28:52,385: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:28:52,385: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:28:52,386: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 225b0fb6-f420-4ec1-b756-7657b9ef4b36
[2017-05-10 13:28:52,386: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 225b0fb6-f420-4ec1-b756-7657b9ef4b36
[2017-05-10 13:28:52,389: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 225b0fb6-f420-4ec1-b756-7657b9ef4b36
[2017-05-10 13:28:52,389: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 225b0fb6-f420-4ec1-b756-7657b9ef4b36
[2017-05-10 13:28:53,357: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com install_packages {'repos': [u'e2fsprogs', u'iml-agent', u'lustre', u'zfs'], 'packages': [u'chroma-agent-management', u'kernel-devel-3.10.0-514.10.2.el7_lustre', u'lustre-osd-zfs', u'lustre-osd-ldiskfs', u'lustre-modules', u'lustre', u'zfs']}
[2017-05-10 13:28:53,379: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2db6b5ae-264b-4d88-a35b-c6cfeadaeedc
[2017-05-10 13:28:53,379: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2db6b5ae-264b-4d88-a35b-c6cfeadaeedc
[2017-05-10 13:28:54,116: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:28:54,116: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:28:54,116: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:28:54,116: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:28:54,116: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8071984a-d602-42a4-9e50-783519e778f3
[2017-05-10 13:28:54,116: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8071984a-d602-42a4-9e50-783519e778f3
[2017-05-10 13:28:54,138: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8071984a-d602-42a4-9e50-783519e778f3
[2017-05-10 13:28:54,138: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8071984a-d602-42a4-9e50-783519e778f3
[2017-05-10 13:28:55,157: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com install_packages {'repos': [u'e2fsprogs', u'iml-agent', u'lustre', u'zfs'], 'packages': [u'chroma-agent-management', u'kernel-devel-3.10.0-514.10.2.el7_lustre', u'lustre-osd-zfs', u'lustre-osd-ldiskfs', u'lustre-modules', u'lustre', u'zfs']}
[2017-05-10 13:28:55,174: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 86052d56-1044-444a-a987-3514f9d6ec8f
[2017-05-10 13:28:55,174: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 86052d56-1044-444a-a987-3514f9d6ec8f
[2017-05-10 13:28:56,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:28:56,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:28:56,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:28:56,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:28:56,546: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 33dd3d03-5dad-438c-a614-649efc5ae237
[2017-05-10 13:28:56,546: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 33dd3d03-5dad-438c-a614-649efc5ae237
[2017-05-10 13:28:56,568: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 33dd3d03-5dad-438c-a614-649efc5ae237
[2017-05-10 13:28:56,568: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 33dd3d03-5dad-438c-a614-649efc5ae237
[2017-05-10 13:28:57,218: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com install_packages {'repos': [u'e2fsprogs', u'iml-agent', u'lustre', u'zfs'], 'packages': [u'chroma-agent-management', u'kernel-devel-3.10.0-514.10.2.el7_lustre', u'lustre-osd-zfs', u'lustre-osd-ldiskfs', u'lustre-modules', u'lustre', u'zfs']}
[2017-05-10 13:28:57,236: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 782dca2f-3219-4b79-8180-ae0026f8f803
[2017-05-10 13:28:57,236: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 782dca2f-3219-4b79-8180-ae0026f8f803
[2017-05-10 13:29:05,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:05,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:05,748: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:05,748: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:05,748: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 15a63ef8-3d4b-417e-90cd-c5a0b0f9326c
[2017-05-10 13:29:05,748: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 15a63ef8-3d4b-417e-90cd-c5a0b0f9326c
[2017-05-10 13:29:05,752: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 15a63ef8-3d4b-417e-90cd-c5a0b0f9326c
[2017-05-10 13:29:05,752: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 15a63ef8-3d4b-417e-90cd-c5a0b0f9326c
[2017-05-10 13:29:06,167: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com kernel_status {}
[2017-05-10 13:29:06,182: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e4de4dde-facf-4807-9f58-728d3826d1c5
[2017-05-10 13:29:06,182: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e4de4dde-facf-4807-9f58-728d3826d1c5
[2017-05-10 13:29:08,308: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:08,308: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:08,308: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:08,308: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:08,308: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e4de4dde-facf-4807-9f58-728d3826d1c5
[2017-05-10 13:29:08,308: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e4de4dde-facf-4807-9f58-728d3826d1c5
[2017-05-10 13:29:08,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e4de4dde-facf-4807-9f58-728d3826d1c5
[2017-05-10 13:29:08,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e4de4dde-facf-4807-9f58-728d3826d1c5
[2017-05-10 13:29:08,351: INFO/job_scheduler] Job 20 finished 4 steps successfully
[2017-05-10 13:29:08,434: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 140430465890000
[2017-05-10 13:29:08,440: INFO/job] Job 20: StateChangeJob complete, setting state packages_installed on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:29:08,440: INFO/job] Job 20: not_deleted=True
[2017-05-10 13:29:08,440: INFO/job_scheduler] Job 20: complete_job: Updating cache
[2017-05-10 13:29:08,440: INFO/job_scheduler] Job 20: locked_item 140430600507216 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 13:29:08,444: INFO/job_scheduler] Job 20 completing (errored=False, cancelled=False)
[2017-05-10 13:29:08,483: INFO/job_scheduler] run_next: 4 runnable jobs of (40 pending, 3 tasked)
[2017-05-10 13:29:08,537: INFO/job_scheduler] Job 21: RunJobThread.run
[2017-05-10 13:29:08,538: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-05-10 13:29:08,546: INFO/job_scheduler] Job 22: RunJobThread.run
[2017-05-10 13:29:08,552: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b5deea49-6986-4858-85dd-51fd3b094add
[2017-05-10 13:29:08,552: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b5deea49-6986-4858-85dd-51fd3b094add
[2017-05-10 13:29:08,558: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-05-10 13:29:08,568: INFO/job_scheduler] Job 23: RunJobThread.run
[2017-05-10 13:29:08,572: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f553e262-255e-4a36-8ec1-4bd8e8d9dd00
[2017-05-10 13:29:08,572: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f553e262-255e-4a36-8ec1-4bd8e8d9dd00
[2017-05-10 13:29:08,569: INFO/job_scheduler] Job 24: open DB connection during completion
[2017-05-10 13:29:08,575: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-05-10 13:29:08,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 632f8c89-7f9a-4c1f-8e2c-61e7324fdee9
[2017-05-10 13:29:08,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 632f8c89-7f9a-4c1f-8e2c-61e7324fdee9
[2017-05-10 13:29:08,694: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 140430600555280
[2017-05-10 13:29:08,706: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 13:29:08,718: INFO/job] Job 24: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:29:08,718: INFO/job] Job 24: not_deleted=True
[2017-05-10 13:29:08,718: INFO/job_scheduler] Job 24: complete_job: Updating cache
[2017-05-10 13:29:08,718: INFO/job_scheduler] Job 24: locked_item 140430600555280 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:29:08,722: INFO/job_scheduler] Job 24 completing (errored=False, cancelled=False)
[2017-05-10 13:29:08,821: INFO/job_scheduler] run_next: 1 runnable jobs of (36 pending, 6 tasked)
[2017-05-10 13:29:08,854: INFO/job_scheduler] Job 25: RunJobThread.run
[2017-05-10 13:29:08,854: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com load_lnet {}
[2017-05-10 13:29:08,875: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cf48e80b-cd0a-449f-9426-1e7449db85c1
[2017-05-10 13:29:08,875: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cf48e80b-cd0a-449f-9426-1e7449db85c1
[2017-05-10 13:29:08,990: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:08,990: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:08,990: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:08,990: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:08,990: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cf48e80b-cd0a-449f-9426-1e7449db85c1
[2017-05-10 13:29:08,990: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cf48e80b-cd0a-449f-9426-1e7449db85c1
[2017-05-10 13:29:09,017: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cf48e80b-cd0a-449f-9426-1e7449db85c1
[2017-05-10 13:29:09,017: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cf48e80b-cd0a-449f-9426-1e7449db85c1
[2017-05-10 13:29:09,018: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 13:29:09,068: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 367a686e-360a-41b3-a0a5-949360f4f1ac
[2017-05-10 13:29:09,068: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 367a686e-360a-41b3-a0a5-949360f4f1ac
[2017-05-10 13:29:10,161: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:10,161: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:10,161: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:10,161: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:10,161: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 367a686e-360a-41b3-a0a5-949360f4f1ac
[2017-05-10 13:29:10,161: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 367a686e-360a-41b3-a0a5-949360f4f1ac
[2017-05-10 13:29:10,187: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 367a686e-360a-41b3-a0a5-949360f4f1ac
[2017-05-10 13:29:10,187: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 367a686e-360a-41b3-a0a5-949360f4f1ac
[2017-05-10 13:29:10,922: INFO/job_scheduler] Job 25 finished 2 steps successfully
[2017-05-10 13:29:11,044: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 140430600555280
[2017-05-10 13:29:11,053: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 13:29:11,056: INFO/job] Job 25: StateChangeJob complete, setting state lnet_down on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:29:11,057: INFO/job] Job 25: not_deleted=True
[2017-05-10 13:29:11,057: INFO/job_scheduler] Job 25: complete_job: Updating cache
[2017-05-10 13:29:11,057: INFO/job_scheduler] Job 25: locked_item 140430600555280 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:29:11,066: INFO/job_scheduler] Job 25 completing (errored=False, cancelled=False)
[2017-05-10 13:29:11,134: INFO/job_scheduler] run_next: 1 runnable jobs of (35 pending, 6 tasked)
[2017-05-10 13:29:11,138: INFO/job_scheduler] Job 28: RunJobThread.run
[2017-05-10 13:29:11,139: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_lnet {}
[2017-05-10 13:29:11,155: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c24cee1c-b817-4a9e-ad68-4ce01ab553b6
[2017-05-10 13:29:11,155: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c24cee1c-b817-4a9e-ad68-4ce01ab553b6
[2017-05-10 13:29:11,272: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:11,272: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:11,272: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:11,272: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:11,273: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c24cee1c-b817-4a9e-ad68-4ce01ab553b6
[2017-05-10 13:29:11,273: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c24cee1c-b817-4a9e-ad68-4ce01ab553b6
[2017-05-10 13:29:11,281: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c24cee1c-b817-4a9e-ad68-4ce01ab553b6
[2017-05-10 13:29:11,281: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c24cee1c-b817-4a9e-ad68-4ce01ab553b6
[2017-05-10 13:29:11,282: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 13:29:11,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f892bda5-3f48-48de-a12e-27dd998f19ad
[2017-05-10 13:29:11,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f892bda5-3f48-48de-a12e-27dd998f19ad
[2017-05-10 13:29:11,407: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:11,407: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:11,407: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:11,407: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:11,407: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f892bda5-3f48-48de-a12e-27dd998f19ad
[2017-05-10 13:29:11,407: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f892bda5-3f48-48de-a12e-27dd998f19ad
[2017-05-10 13:29:11,441: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f892bda5-3f48-48de-a12e-27dd998f19ad
[2017-05-10 13:29:11,441: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f892bda5-3f48-48de-a12e-27dd998f19ad
[2017-05-10 13:29:11,718: INFO/job_scheduler] Job 28 finished 2 steps successfully
[2017-05-10 13:29:11,776: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 140430600555280
[2017-05-10 13:29:11,781: INFO/job] Job 28: StateChangeJob complete, setting state lnet_up on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:29:11,781: INFO/job] Job 28: not_deleted=True
[2017-05-10 13:29:11,781: INFO/job_scheduler] Job 28: complete_job: Updating cache
[2017-05-10 13:29:11,781: INFO/job_scheduler] Job 28: locked_item 140430600555280 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:29:11,785: INFO/job_scheduler] Job 28 completing (errored=False, cancelled=False)
[2017-05-10 13:29:11,813: INFO/job_scheduler] run_next: 0 runnable jobs of (34 pending, 6 tasked)
[2017-05-10 13:29:13,746: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:13,746: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:13,746: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:13,746: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:13,746: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f553e262-255e-4a36-8ec1-4bd8e8d9dd00
[2017-05-10 13:29:13,746: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f553e262-255e-4a36-8ec1-4bd8e8d9dd00
[2017-05-10 13:29:13,792: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f553e262-255e-4a36-8ec1-4bd8e8d9dd00
[2017-05-10 13:29:13,792: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f553e262-255e-4a36-8ec1-4bd8e8d9dd00
[2017-05-10 13:29:13,793: INFO/job_scheduler] Job 22 finished 1 steps successfully
[2017-05-10 13:29:14,102: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 140430600473040
[2017-05-10 13:29:14,106: INFO/job] Job 22: StateChangeJob complete, setting state configured on lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-10 13:29:14,106: INFO/job] Job 22: not_deleted=True
[2017-05-10 13:29:14,106: INFO/job_scheduler] Job 22: complete_job: Updating cache
[2017-05-10 13:29:14,106: INFO/job_scheduler] Job 22: locked_item 140430600473040 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-10 13:29:14,109: INFO/job_scheduler] Job 22 completing (errored=False, cancelled=False)
[2017-05-10 13:29:14,191: INFO/job_scheduler] run_next: 0 runnable jobs of (34 pending, 5 tasked)
[2017-05-10 13:29:15,081: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:15,081: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:15,082: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:15,082: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:15,082: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2db6b5ae-264b-4d88-a35b-c6cfeadaeedc
[2017-05-10 13:29:15,082: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2db6b5ae-264b-4d88-a35b-c6cfeadaeedc
[2017-05-10 13:29:15,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2db6b5ae-264b-4d88-a35b-c6cfeadaeedc
[2017-05-10 13:29:15,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2db6b5ae-264b-4d88-a35b-c6cfeadaeedc
[2017-05-10 13:29:15,680: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com kernel_status {}
[2017-05-10 13:29:15,703: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4b3b273f-2e16-4f4b-959d-cc85de63bbd8
[2017-05-10 13:29:15,703: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4b3b273f-2e16-4f4b-959d-cc85de63bbd8
[2017-05-10 13:29:15,810: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:15,810: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:15,811: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:15,811: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:15,811: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b5deea49-6986-4858-85dd-51fd3b094add
[2017-05-10 13:29:15,811: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b5deea49-6986-4858-85dd-51fd3b094add
[2017-05-10 13:29:15,829: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b5deea49-6986-4858-85dd-51fd3b094add
[2017-05-10 13:29:15,829: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b5deea49-6986-4858-85dd-51fd3b094add
[2017-05-10 13:29:15,830: INFO/job_scheduler] Job 21 finished 1 steps successfully
[2017-05-10 13:29:15,908: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 140430735406160
[2017-05-10 13:29:15,913: INFO/job] Job 21: StateChangeJob complete, setting state configured on lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-10 13:29:15,913: INFO/job] Job 21: not_deleted=True
[2017-05-10 13:29:15,913: INFO/job_scheduler] Job 21: complete_job: Updating cache
[2017-05-10 13:29:15,913: INFO/job_scheduler] Job 21: locked_item 140430735406160 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-10 13:29:15,916: INFO/job_scheduler] Job 21 completing (errored=False, cancelled=False)
[2017-05-10 13:29:15,940: INFO/job_scheduler] run_next: 0 runnable jobs of (34 pending, 4 tasked)
[2017-05-10 13:29:17,820: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:17,820: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:17,820: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:17,820: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:17,821: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4b3b273f-2e16-4f4b-959d-cc85de63bbd8
[2017-05-10 13:29:17,821: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4b3b273f-2e16-4f4b-959d-cc85de63bbd8
[2017-05-10 13:29:17,825: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4b3b273f-2e16-4f4b-959d-cc85de63bbd8
[2017-05-10 13:29:17,825: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4b3b273f-2e16-4f4b-959d-cc85de63bbd8
[2017-05-10 13:29:17,826: INFO/job_scheduler] Job 9 finished 4 steps successfully
[2017-05-10 13:29:17,905: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 140430600534096
[2017-05-10 13:29:17,911: INFO/job] Job 9: StateChangeJob complete, setting state packages_installed on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:29:17,911: INFO/job] Job 9: not_deleted=True
[2017-05-10 13:29:17,911: INFO/job_scheduler] Job 9: complete_job: Updating cache
[2017-05-10 13:29:17,911: INFO/job_scheduler] Job 9: locked_item 140430600504272 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 13:29:17,915: INFO/job_scheduler] Job 9 completing (errored=False, cancelled=False)
[2017-05-10 13:29:17,939: INFO/job_scheduler] run_next: 4 runnable jobs of (34 pending, 3 tasked)
[2017-05-10 13:29:18,006: INFO/job_scheduler] Job 10: RunJobThread.run
[2017-05-10 13:29:18,008: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-05-10 13:29:18,049: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2ab43b01-e8b1-4510-8ae3-ff7ed6370db0
[2017-05-10 13:29:18,049: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2ab43b01-e8b1-4510-8ae3-ff7ed6370db0
[2017-05-10 13:29:18,059: INFO/job_scheduler] Job 11: RunJobThread.run
[2017-05-10 13:29:18,059: INFO/job_scheduler] Job 12: open DB connection during completion
[2017-05-10 13:29:18,061: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-05-10 13:29:18,079: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 73290a37-83d9-4cb1-ae09-bbf7b0049e64
[2017-05-10 13:29:18,079: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 73290a37-83d9-4cb1-ae09-bbf7b0049e64
[2017-05-10 13:29:18,439: INFO/job_scheduler] Job 13: RunJobThread.run
[2017-05-10 13:29:18,440: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-05-10 13:29:18,466: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 09000ac7-0541-42b4-9196-956cf6b934e9
[2017-05-10 13:29:18,466: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 09000ac7-0541-42b4-9196-956cf6b934e9
[2017-05-10 13:29:18,488: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 140430465896336
[2017-05-10 13:29:18,496: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 13:29:18,514: INFO/job] Job 12: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:29:18,514: INFO/job] Job 12: not_deleted=True
[2017-05-10 13:29:18,514: INFO/job_scheduler] Job 12: complete_job: Updating cache
[2017-05-10 13:29:18,514: INFO/job_scheduler] Job 12: locked_item 140430465896336 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:29:18,524: INFO/job_scheduler] Job 12 completing (errored=False, cancelled=False)
[2017-05-10 13:29:18,564: INFO/job_scheduler] run_next: 1 runnable jobs of (30 pending, 6 tasked)
[2017-05-10 13:29:18,594: INFO/job_scheduler] Job 14: RunJobThread.run
[2017-05-10 13:29:18,594: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com load_lnet {}
[2017-05-10 13:29:18,627: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 69979e55-d907-4ede-a300-32e7383608cc
[2017-05-10 13:29:18,627: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 69979e55-d907-4ede-a300-32e7383608cc
[2017-05-10 13:29:19,737: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:19,737: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:19,738: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:19,738: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:19,738: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 69979e55-d907-4ede-a300-32e7383608cc
[2017-05-10 13:29:19,738: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 69979e55-d907-4ede-a300-32e7383608cc
[2017-05-10 13:29:19,741: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:19,741: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:19,742: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:19,742: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:19,742: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 782dca2f-3219-4b79-8180-ae0026f8f803
[2017-05-10 13:29:19,742: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 782dca2f-3219-4b79-8180-ae0026f8f803
[2017-05-10 13:29:19,742: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 69979e55-d907-4ede-a300-32e7383608cc
[2017-05-10 13:29:19,742: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 69979e55-d907-4ede-a300-32e7383608cc
[2017-05-10 13:29:19,743: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 13:29:19,759: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 59a6b099-931d-4df2-8dae-4564d76e05a1
[2017-05-10 13:29:19,759: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 59a6b099-931d-4df2-8dae-4564d76e05a1
[2017-05-10 13:29:19,767: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 782dca2f-3219-4b79-8180-ae0026f8f803
[2017-05-10 13:29:19,767: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 782dca2f-3219-4b79-8180-ae0026f8f803
[2017-05-10 13:29:19,816: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:19,816: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:19,816: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:19,816: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:19,816: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 59a6b099-931d-4df2-8dae-4564d76e05a1
[2017-05-10 13:29:19,816: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 59a6b099-931d-4df2-8dae-4564d76e05a1
[2017-05-10 13:29:19,829: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 59a6b099-931d-4df2-8dae-4564d76e05a1
[2017-05-10 13:29:19,829: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 59a6b099-931d-4df2-8dae-4564d76e05a1
[2017-05-10 13:29:20,410: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com kernel_status {}
[2017-05-10 13:29:20,433: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3061cff5-feaf-4245-87b4-b2a3a59397af
[2017-05-10 13:29:20,433: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3061cff5-feaf-4245-87b4-b2a3a59397af
[2017-05-10 13:29:20,472: INFO/job_scheduler] Job 14 finished 2 steps successfully
[2017-05-10 13:29:20,481: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 140430465896336
[2017-05-10 13:29:20,496: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 13:29:20,515: INFO/job] Job 14: StateChangeJob complete, setting state lnet_down on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:29:20,515: INFO/job] Job 14: not_deleted=True
[2017-05-10 13:29:20,515: INFO/job_scheduler] Job 14: complete_job: Updating cache
[2017-05-10 13:29:20,515: INFO/job_scheduler] Job 14: locked_item 140430465896336 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:29:20,519: INFO/job_scheduler] Job 14 completing (errored=False, cancelled=False)
[2017-05-10 13:29:20,554: INFO/job_scheduler] run_next: 1 runnable jobs of (29 pending, 6 tasked)
[2017-05-10 13:29:20,559: INFO/job_scheduler] Job 17: RunJobThread.run
[2017-05-10 13:29:20,560: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_lnet {}
[2017-05-10 13:29:20,577: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8979da42-f2f1-4027-a5ff-b402889ccd14
[2017-05-10 13:29:20,577: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8979da42-f2f1-4027-a5ff-b402889ccd14
[2017-05-10 13:29:20,658: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:20,658: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:20,658: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:20,658: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:20,658: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 86052d56-1044-444a-a987-3514f9d6ec8f
[2017-05-10 13:29:20,658: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 86052d56-1044-444a-a987-3514f9d6ec8f
[2017-05-10 13:29:20,659: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:20,659: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:20,659: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:20,659: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:20,659: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8979da42-f2f1-4027-a5ff-b402889ccd14
[2017-05-10 13:29:20,659: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8979da42-f2f1-4027-a5ff-b402889ccd14
[2017-05-10 13:29:20,675: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 86052d56-1044-444a-a987-3514f9d6ec8f
[2017-05-10 13:29:20,675: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 86052d56-1044-444a-a987-3514f9d6ec8f
[2017-05-10 13:29:20,700: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8979da42-f2f1-4027-a5ff-b402889ccd14
[2017-05-10 13:29:20,700: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8979da42-f2f1-4027-a5ff-b402889ccd14
[2017-05-10 13:29:20,701: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 13:29:20,764: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 40ff3cb5-3037-48fc-b179-a74985157c81
[2017-05-10 13:29:20,764: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 40ff3cb5-3037-48fc-b179-a74985157c81
[2017-05-10 13:29:20,865: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:20,865: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:20,865: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:20,865: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:20,865: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 40ff3cb5-3037-48fc-b179-a74985157c81
[2017-05-10 13:29:20,865: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 40ff3cb5-3037-48fc-b179-a74985157c81
[2017-05-10 13:29:20,889: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 40ff3cb5-3037-48fc-b179-a74985157c81
[2017-05-10 13:29:20,889: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 40ff3cb5-3037-48fc-b179-a74985157c81
[2017-05-10 13:29:21,383: INFO/job_scheduler] Job 17 finished 2 steps successfully
[2017-05-10 13:29:21,436: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 140430465896336
[2017-05-10 13:29:21,443: INFO/job] Job 17: StateChangeJob complete, setting state lnet_up on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:29:21,443: INFO/job] Job 17: not_deleted=True
[2017-05-10 13:29:21,443: INFO/job_scheduler] Job 17: complete_job: Updating cache
[2017-05-10 13:29:21,443: INFO/job_scheduler] Job 17: locked_item 140430465896336 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:29:21,453: INFO/job_scheduler] Job 17 completing (errored=False, cancelled=False)
[2017-05-10 13:29:21,494: INFO/job_scheduler] run_next: 0 runnable jobs of (28 pending, 6 tasked)
[2017-05-10 13:29:21,776: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com kernel_status {}
[2017-05-10 13:29:21,794: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b3e66ec6-3dc2-4f56-9edd-7bb7e3576354
[2017-05-10 13:29:21,794: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b3e66ec6-3dc2-4f56-9edd-7bb7e3576354
[2017-05-10 13:29:22,524: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:22,524: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:22,524: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:22,524: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:22,524: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3061cff5-feaf-4245-87b4-b2a3a59397af
[2017-05-10 13:29:22,524: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3061cff5-feaf-4245-87b4-b2a3a59397af
[2017-05-10 13:29:22,548: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3061cff5-feaf-4245-87b4-b2a3a59397af
[2017-05-10 13:29:22,548: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3061cff5-feaf-4245-87b4-b2a3a59397af
[2017-05-10 13:29:22,549: INFO/job_scheduler] Job 42 finished 4 steps successfully
[2017-05-10 13:29:22,619: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 140430600664656
[2017-05-10 13:29:22,628: INFO/job] Job 42: StateChangeJob complete, setting state packages_installed on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:29:22,628: INFO/job] Job 42: not_deleted=True
[2017-05-10 13:29:22,629: INFO/job_scheduler] Job 42: complete_job: Updating cache
[2017-05-10 13:29:22,629: INFO/job_scheduler] Job 42: locked_item 140430465989648 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 13:29:22,632: INFO/job_scheduler] Job 42 completing (errored=False, cancelled=False)
[2017-05-10 13:29:22,653: INFO/job_scheduler] run_next: 4 runnable jobs of (28 pending, 5 tasked)
[2017-05-10 13:29:22,707: INFO/job_scheduler] Job 43: RunJobThread.run
[2017-05-10 13:29:22,709: INFO/job_scheduler] Job 44: open DB connection during completion
[2017-05-10 13:29:22,711: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-05-10 13:29:22,726: INFO/job_scheduler] Job 45: RunJobThread.run
[2017-05-10 13:29:22,727: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-05-10 13:29:22,732: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 54a54440-b9c0-4c18-8cf1-b8a8d1a6a05a
[2017-05-10 13:29:22,732: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 54a54440-b9c0-4c18-8cf1-b8a8d1a6a05a
[2017-05-10 13:29:22,746: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8fa0d864-bd29-4496-bfcd-2efe78f6e26a
[2017-05-10 13:29:22,746: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8fa0d864-bd29-4496-bfcd-2efe78f6e26a
[2017-05-10 13:29:22,761: INFO/job_scheduler] Job 46: RunJobThread.run
[2017-05-10 13:29:22,762: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-05-10 13:29:22,783: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 18d561b2-df50-45d2-b68c-5559fb8a3256
[2017-05-10 13:29:22,783: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 18d561b2-df50-45d2-b68c-5559fb8a3256
[2017-05-10 13:29:22,786: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 140430600554768
[2017-05-10 13:29:22,818: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 13:29:22,848: INFO/job] Job 44: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:29:22,848: INFO/job] Job 44: not_deleted=True
[2017-05-10 13:29:22,848: INFO/job_scheduler] Job 44: complete_job: Updating cache
[2017-05-10 13:29:22,848: INFO/job_scheduler] Job 44: locked_item 140430600554768 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:29:22,860: INFO/job_scheduler] Job 44 completing (errored=False, cancelled=False)
[2017-05-10 13:29:22,908: INFO/job_scheduler] run_next: 1 runnable jobs of (24 pending, 8 tasked)
[2017-05-10 13:29:22,934: INFO/job_scheduler] Job 48: RunJobThread.run
[2017-05-10 13:29:22,935: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com load_lnet {}
[2017-05-10 13:29:22,973: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 095eab9b-2c0c-421b-837a-0524792cf20d
[2017-05-10 13:29:22,973: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 095eab9b-2c0c-421b-837a-0524792cf20d
[2017-05-10 13:29:24,083: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:24,083: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:24,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:24,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:24,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 095eab9b-2c0c-421b-837a-0524792cf20d
[2017-05-10 13:29:24,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 095eab9b-2c0c-421b-837a-0524792cf20d
[2017-05-10 13:29:24,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 095eab9b-2c0c-421b-837a-0524792cf20d
[2017-05-10 13:29:24,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 095eab9b-2c0c-421b-837a-0524792cf20d
[2017-05-10 13:29:24,088: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 13:29:24,106: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ffb60360-76ef-462d-b424-e465f85fc114
[2017-05-10 13:29:24,106: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ffb60360-76ef-462d-b424-e465f85fc114
[2017-05-10 13:29:24,202: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:24,202: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:24,203: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:24,203: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:24,203: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ffb60360-76ef-462d-b424-e465f85fc114
[2017-05-10 13:29:24,203: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ffb60360-76ef-462d-b424-e465f85fc114
[2017-05-10 13:29:24,230: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ffb60360-76ef-462d-b424-e465f85fc114
[2017-05-10 13:29:24,230: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ffb60360-76ef-462d-b424-e465f85fc114
[2017-05-10 13:29:24,240: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:24,240: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:24,240: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:24,240: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:24,240: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 73290a37-83d9-4cb1-ae09-bbf7b0049e64
[2017-05-10 13:29:24,240: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 73290a37-83d9-4cb1-ae09-bbf7b0049e64
[2017-05-10 13:29:24,266: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 73290a37-83d9-4cb1-ae09-bbf7b0049e64
[2017-05-10 13:29:24,266: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 73290a37-83d9-4cb1-ae09-bbf7b0049e64
[2017-05-10 13:29:24,267: INFO/job_scheduler] Job 11 finished 1 steps successfully
[2017-05-10 13:29:24,473: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 140430465892624
[2017-05-10 13:29:24,477: INFO/job] Job 11: StateChangeJob complete, setting state configured on lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-10 13:29:24,478: INFO/job] Job 11: not_deleted=True
[2017-05-10 13:29:24,478: INFO/job_scheduler] Job 11: complete_job: Updating cache
[2017-05-10 13:29:24,478: INFO/job_scheduler] Job 11: locked_item 140430465892624 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-10 13:29:24,481: INFO/job_scheduler] Job 11 completing (errored=False, cancelled=False)
[2017-05-10 13:29:24,501: INFO/job_scheduler] run_next: 0 runnable jobs of (23 pending, 8 tasked)
[2017-05-10 13:29:24,674: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:24,674: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:24,674: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:24,674: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:24,674: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 09000ac7-0541-42b4-9196-956cf6b934e9
[2017-05-10 13:29:24,674: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 09000ac7-0541-42b4-9196-956cf6b934e9
[2017-05-10 13:29:24,704: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 09000ac7-0541-42b4-9196-956cf6b934e9
[2017-05-10 13:29:24,704: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 09000ac7-0541-42b4-9196-956cf6b934e9
[2017-05-10 13:29:24,704: INFO/job_scheduler] Job 13 finished 1 steps successfully
[2017-05-10 13:29:24,812: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 140430735406416
[2017-05-10 13:29:24,815: INFO/job] Job 13: StateChangeJob complete, setting state configured on lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-10 13:29:24,815: INFO/job] Job 13: not_deleted=True
[2017-05-10 13:29:24,816: INFO/job_scheduler] Job 13: complete_job: Updating cache
[2017-05-10 13:29:24,816: INFO/job_scheduler] Job 13: locked_item 140430735406416 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-10 13:29:24,818: INFO/job_scheduler] Job 13 completing (errored=False, cancelled=False)
[2017-05-10 13:29:24,861: INFO/job_scheduler] run_next: 0 runnable jobs of (23 pending, 7 tasked)
[2017-05-10 13:29:24,901: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:24,901: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:24,901: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:24,901: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:24,901: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b3e66ec6-3dc2-4f56-9edd-7bb7e3576354
[2017-05-10 13:29:24,901: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b3e66ec6-3dc2-4f56-9edd-7bb7e3576354
[2017-05-10 13:29:24,924: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b3e66ec6-3dc2-4f56-9edd-7bb7e3576354
[2017-05-10 13:29:24,924: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b3e66ec6-3dc2-4f56-9edd-7bb7e3576354
[2017-05-10 13:29:24,925: INFO/job_scheduler] Job 31 finished 4 steps successfully
[2017-05-10 13:29:24,935: INFO/job_scheduler] Job 48 finished 2 steps successfully
[2017-05-10 13:29:25,010: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 140430600635280
[2017-05-10 13:29:25,015: INFO/job] Job 31: StateChangeJob complete, setting state packages_installed on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:29:25,015: INFO/job] Job 31: not_deleted=True
[2017-05-10 13:29:25,015: INFO/job_scheduler] Job 31: complete_job: Updating cache
[2017-05-10 13:29:25,015: INFO/job_scheduler] Job 31: locked_item 140430465870032 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 13:29:25,032: INFO/job_scheduler] Job 31 completing (errored=False, cancelled=False)
[2017-05-10 13:29:25,076: INFO/job_scheduler] run_next: 4 runnable jobs of (23 pending, 6 tasked)
[2017-05-10 13:29:25,131: INFO/job_scheduler] Job 32: RunJobThread.run
[2017-05-10 13:29:25,133: INFO/job_scheduler] Job 33: RunJobThread.run
[2017-05-10 13:29:25,134: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-05-10 13:29:25,143: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-05-10 13:29:25,161: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 008c07c6-1445-4286-a929-d5c5410d2f24
[2017-05-10 13:29:25,161: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 008c07c6-1445-4286-a929-d5c5410d2f24
[2017-05-10 13:29:25,170: INFO/job_scheduler] Job 34: RunJobThread.run
[2017-05-10 13:29:25,171: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-05-10 13:29:25,171: INFO/job_scheduler] Job 35: open DB connection during completion
[2017-05-10 13:29:25,206: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 140430600554768
[2017-05-10 13:29:25,218: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 13:29:25,302: INFO/job] Job 48: StateChangeJob complete, setting state lnet_down on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:29:25,303: INFO/job] Job 48: not_deleted=True
[2017-05-10 13:29:25,303: INFO/job_scheduler] Job 48: complete_job: Updating cache
[2017-05-10 13:29:25,303: INFO/job_scheduler] Job 48: locked_item 140430600554768 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:29:25,311: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 28483274-79ea-480b-a43e-d0c004f4b1cd
[2017-05-10 13:29:25,311: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 28483274-79ea-480b-a43e-d0c004f4b1cd
[2017-05-10 13:29:25,326: INFO/job_scheduler] Job 48 completing (errored=False, cancelled=False)
[2017-05-10 13:29:25,334: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cec8628e-69ac-4f86-a248-6261ff40b8d7
[2017-05-10 13:29:25,334: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cec8628e-69ac-4f86-a248-6261ff40b8d7
[2017-05-10 13:29:25,364: INFO/job_scheduler] run_next: 1 runnable jobs of (19 pending, 9 tasked)
[2017-05-10 13:29:25,413: INFO/job_scheduler] Job 49: RunJobThread.run
[2017-05-10 13:29:25,414: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_lnet {}
[2017-05-10 13:29:25,440: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 739c5ed9-a236-4af3-aaf2-dbdba1c5b150
[2017-05-10 13:29:25,440: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 739c5ed9-a236-4af3-aaf2-dbdba1c5b150
[2017-05-10 13:29:25,502: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 140430465888400
[2017-05-10 13:29:25,510: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 13:29:25,524: INFO/job] Job 35: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:29:25,524: INFO/job] Job 35: not_deleted=True
[2017-05-10 13:29:25,524: INFO/job_scheduler] Job 35: complete_job: Updating cache
[2017-05-10 13:29:25,524: INFO/job_scheduler] Job 35: locked_item 140430465888400 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:29:25,529: INFO/job_scheduler] Job 35 completing (errored=False, cancelled=False)
[2017-05-10 13:29:25,558: INFO/job_scheduler] run_next: 1 runnable jobs of (18 pending, 9 tasked)
[2017-05-10 13:29:25,574: INFO/job_scheduler] Job 36: RunJobThread.run
[2017-05-10 13:29:25,575: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com load_lnet {}
[2017-05-10 13:29:25,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2d836431-cfcf-4b61-8bdd-68097df8b6c9
[2017-05-10 13:29:25,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2d836431-cfcf-4b61-8bdd-68097df8b6c9
[2017-05-10 13:29:25,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:25,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:25,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:25,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:25,600: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 739c5ed9-a236-4af3-aaf2-dbdba1c5b150
[2017-05-10 13:29:25,600: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 739c5ed9-a236-4af3-aaf2-dbdba1c5b150
[2017-05-10 13:29:25,614: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 739c5ed9-a236-4af3-aaf2-dbdba1c5b150
[2017-05-10 13:29:25,614: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 739c5ed9-a236-4af3-aaf2-dbdba1c5b150
[2017-05-10 13:29:25,614: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 13:29:25,668: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d394ade6-974c-43eb-bba9-78e5793ef098
[2017-05-10 13:29:25,668: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d394ade6-974c-43eb-bba9-78e5793ef098
[2017-05-10 13:29:25,700: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:25,700: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:25,700: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:25,700: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:25,700: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2d836431-cfcf-4b61-8bdd-68097df8b6c9
[2017-05-10 13:29:25,700: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2d836431-cfcf-4b61-8bdd-68097df8b6c9
[2017-05-10 13:29:25,726: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2d836431-cfcf-4b61-8bdd-68097df8b6c9
[2017-05-10 13:29:25,726: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2d836431-cfcf-4b61-8bdd-68097df8b6c9
[2017-05-10 13:29:25,727: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 13:29:25,768: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 83a302ce-0529-4e95-a6b1-3c3908c2ac3c
[2017-05-10 13:29:25,768: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 83a302ce-0529-4e95-a6b1-3c3908c2ac3c
[2017-05-10 13:29:26,828: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:26,828: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:26,828: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:26,828: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:26,828: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d394ade6-974c-43eb-bba9-78e5793ef098
[2017-05-10 13:29:26,828: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d394ade6-974c-43eb-bba9-78e5793ef098
[2017-05-10 13:29:26,830: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:26,830: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:26,830: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:26,830: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:26,830: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 83a302ce-0529-4e95-a6b1-3c3908c2ac3c
[2017-05-10 13:29:26,830: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 83a302ce-0529-4e95-a6b1-3c3908c2ac3c
[2017-05-10 13:29:26,833: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 83a302ce-0529-4e95-a6b1-3c3908c2ac3c
[2017-05-10 13:29:26,833: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 83a302ce-0529-4e95-a6b1-3c3908c2ac3c
[2017-05-10 13:29:26,835: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d394ade6-974c-43eb-bba9-78e5793ef098
[2017-05-10 13:29:26,835: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d394ade6-974c-43eb-bba9-78e5793ef098
[2017-05-10 13:29:27,474: INFO/job_scheduler] Job 36 finished 2 steps successfully
[2017-05-10 13:29:27,541: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 140430465888400
[2017-05-10 13:29:27,548: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 13:29:27,567: INFO/job] Job 36: StateChangeJob complete, setting state lnet_down on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:29:27,567: INFO/job] Job 36: not_deleted=True
[2017-05-10 13:29:27,567: INFO/job_scheduler] Job 36: complete_job: Updating cache
[2017-05-10 13:29:27,568: INFO/job_scheduler] Job 36: locked_item 140430465888400 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:29:27,577: INFO/job_scheduler] Job 36 completing (errored=False, cancelled=False)
[2017-05-10 13:29:27,626: INFO/job_scheduler] run_next: 1 runnable jobs of (17 pending, 9 tasked)
[2017-05-10 13:29:27,659: INFO/job_scheduler] Job 39: RunJobThread.run
[2017-05-10 13:29:27,659: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_lnet {}
[2017-05-10 13:29:27,686: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 825f634a-fae1-4719-89c7-31fd20a501f0
[2017-05-10 13:29:27,686: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 825f634a-fae1-4719-89c7-31fd20a501f0
[2017-05-10 13:29:27,802: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:27,802: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:27,802: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:27,802: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:27,802: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 825f634a-fae1-4719-89c7-31fd20a501f0
[2017-05-10 13:29:27,802: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 825f634a-fae1-4719-89c7-31fd20a501f0
[2017-05-10 13:29:27,830: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 825f634a-fae1-4719-89c7-31fd20a501f0
[2017-05-10 13:29:27,830: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 825f634a-fae1-4719-89c7-31fd20a501f0
[2017-05-10 13:29:27,831: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 13:29:27,852: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 77a63471-ed0e-43d3-a1d2-57e282cbd477
[2017-05-10 13:29:27,852: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 77a63471-ed0e-43d3-a1d2-57e282cbd477
[2017-05-10 13:29:27,900: INFO/job_scheduler] Job 49 finished 2 steps successfully
[2017-05-10 13:29:27,938: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:27,938: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:27,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:27,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:27,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 77a63471-ed0e-43d3-a1d2-57e282cbd477
[2017-05-10 13:29:27,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 77a63471-ed0e-43d3-a1d2-57e282cbd477
[2017-05-10 13:29:27,959: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:27,959: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:27,959: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:27,959: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:27,959: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 54a54440-b9c0-4c18-8cf1-b8a8d1a6a05a
[2017-05-10 13:29:27,959: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 54a54440-b9c0-4c18-8cf1-b8a8d1a6a05a
[2017-05-10 13:29:27,969: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 54a54440-b9c0-4c18-8cf1-b8a8d1a6a05a
[2017-05-10 13:29:27,969: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 54a54440-b9c0-4c18-8cf1-b8a8d1a6a05a
[2017-05-10 13:29:27,969: INFO/job_scheduler] Job 43 finished 1 steps successfully
[2017-05-10 13:29:27,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 77a63471-ed0e-43d3-a1d2-57e282cbd477
[2017-05-10 13:29:27,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 77a63471-ed0e-43d3-a1d2-57e282cbd477
[2017-05-10 13:29:28,006: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 140430600554768
[2017-05-10 13:29:28,013: INFO/job] Job 49: StateChangeJob complete, setting state lnet_up on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:29:28,013: INFO/job] Job 49: not_deleted=True
[2017-05-10 13:29:28,014: INFO/job_scheduler] Job 49: complete_job: Updating cache
[2017-05-10 13:29:28,014: INFO/job_scheduler] Job 49: locked_item 140430600554768 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:29:28,017: INFO/job_scheduler] Job 49 completing (errored=False, cancelled=False)
[2017-05-10 13:29:28,034: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 9 tasked)
[2017-05-10 13:29:28,089: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:28,089: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:28,089: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:28,089: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:29:28,090: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 18d561b2-df50-45d2-b68c-5559fb8a3256
[2017-05-10 13:29:28,090: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 18d561b2-df50-45d2-b68c-5559fb8a3256
[2017-05-10 13:29:28,113: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 18d561b2-df50-45d2-b68c-5559fb8a3256
[2017-05-10 13:29:28,113: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 18d561b2-df50-45d2-b68c-5559fb8a3256
[2017-05-10 13:29:28,113: INFO/job_scheduler] Job 46 finished 1 steps successfully
[2017-05-10 13:29:28,169: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 140430600554896
[2017-05-10 13:29:28,172: INFO/job] Job 43: StateChangeJob complete, setting state configured on lotus-32vm8.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-10 13:29:28,172: INFO/job] Job 43: not_deleted=True
[2017-05-10 13:29:28,172: INFO/job_scheduler] Job 43: complete_job: Updating cache
[2017-05-10 13:29:28,173: INFO/job_scheduler] Job 43: locked_item 140430600554896 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-10 13:29:28,175: INFO/job_scheduler] Job 43 completing (errored=False, cancelled=False)
[2017-05-10 13:29:28,211: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 8 tasked)
[2017-05-10 13:29:28,374: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 140430600635728
[2017-05-10 13:29:28,378: INFO/job] Job 46: StateChangeJob complete, setting state configured on lotus-32vm8.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-10 13:29:28,378: INFO/job] Job 46: not_deleted=True
[2017-05-10 13:29:28,378: INFO/job_scheduler] Job 46: complete_job: Updating cache
[2017-05-10 13:29:28,378: INFO/job_scheduler] Job 46: locked_item 140430600635728 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-10 13:29:28,381: INFO/job_scheduler] Job 46 completing (errored=False, cancelled=False)
[2017-05-10 13:29:28,386: INFO/job_scheduler] Job 39 finished 2 steps successfully
[2017-05-10 13:29:28,421: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 7 tasked)
[2017-05-10 13:29:28,486: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 140430465888400
[2017-05-10 13:29:28,491: INFO/job] Job 39: StateChangeJob complete, setting state lnet_up on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:29:28,491: INFO/job] Job 39: not_deleted=True
[2017-05-10 13:29:28,491: INFO/job_scheduler] Job 39: complete_job: Updating cache
[2017-05-10 13:29:28,492: INFO/job_scheduler] Job 39: locked_item 140430465888400 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:29:28,494: INFO/job_scheduler] Job 39 completing (errored=False, cancelled=False)
[2017-05-10 13:29:28,538: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 6 tasked)
[2017-05-10 13:29:30,339: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:30,339: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:30,339: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:30,339: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:30,339: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 008c07c6-1445-4286-a929-d5c5410d2f24
[2017-05-10 13:29:30,339: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 008c07c6-1445-4286-a929-d5c5410d2f24
[2017-05-10 13:29:30,378: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 008c07c6-1445-4286-a929-d5c5410d2f24
[2017-05-10 13:29:30,378: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 008c07c6-1445-4286-a929-d5c5410d2f24
[2017-05-10 13:29:30,379: INFO/job_scheduler] Job 33 finished 1 steps successfully
[2017-05-10 13:29:30,475: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 140430465869264
[2017-05-10 13:29:30,479: INFO/job] Job 33: StateChangeJob complete, setting state configured on lotus-32vm7.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-10 13:29:30,479: INFO/job] Job 33: not_deleted=True
[2017-05-10 13:29:30,479: INFO/job_scheduler] Job 33: complete_job: Updating cache
[2017-05-10 13:29:30,479: INFO/job_scheduler] Job 33: locked_item 140430465869264 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-10 13:29:30,484: INFO/job_scheduler] Job 33 completing (errored=False, cancelled=False)
[2017-05-10 13:29:30,516: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 5 tasked)
[2017-05-10 13:29:30,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:30,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:30,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:30,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:30,560: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 28483274-79ea-480b-a43e-d0c004f4b1cd
[2017-05-10 13:29:30,560: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 28483274-79ea-480b-a43e-d0c004f4b1cd
[2017-05-10 13:29:30,584: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 28483274-79ea-480b-a43e-d0c004f4b1cd
[2017-05-10 13:29:30,584: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 28483274-79ea-480b-a43e-d0c004f4b1cd
[2017-05-10 13:29:30,585: INFO/job_scheduler] Job 32 finished 1 steps successfully
[2017-05-10 13:29:30,775: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 140430600638416
[2017-05-10 13:29:30,780: INFO/job] Job 32: StateChangeJob complete, setting state configured on lotus-32vm7.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-10 13:29:30,780: INFO/job] Job 32: not_deleted=True
[2017-05-10 13:29:30,780: INFO/job_scheduler] Job 32: complete_job: Updating cache
[2017-05-10 13:29:30,780: INFO/job_scheduler] Job 32: locked_item 140430600638416 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-10 13:29:30,787: INFO/job_scheduler] Job 32 completing (errored=False, cancelled=False)
[2017-05-10 13:29:30,849: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 4 tasked)
[2017-05-10 13:29:35,095: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:35,095: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:35,095: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:35,095: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:35,096: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2ab43b01-e8b1-4510-8ae3-ff7ed6370db0
[2017-05-10 13:29:35,096: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2ab43b01-e8b1-4510-8ae3-ff7ed6370db0
[2017-05-10 13:29:35,096: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2ab43b01-e8b1-4510-8ae3-ff7ed6370db0
[2017-05-10 13:29:35,096: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2ab43b01-e8b1-4510-8ae3-ff7ed6370db0
[2017-05-10 13:29:35,097: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.224', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-05-10 13:29:35,113: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ef501a0c-25d1-4fe2-93e9-97e7ee533478
[2017-05-10 13:29:35,113: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ef501a0c-25d1-4fe2-93e9-97e7ee533478
[2017-05-10 13:29:35,214: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:35,214: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:35,214: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:35,214: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:35,214: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ef501a0c-25d1-4fe2-93e9-97e7ee533478
[2017-05-10 13:29:35,214: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ef501a0c-25d1-4fe2-93e9-97e7ee533478
[2017-05-10 13:29:35,235: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ef501a0c-25d1-4fe2-93e9-97e7ee533478
[2017-05-10 13:29:35,235: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ef501a0c-25d1-4fe2-93e9-97e7ee533478
[2017-05-10 13:29:35,253: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 53491, 'pcs_password': 'WMZGHPX1tfMtzycbr3uq'}
[2017-05-10 13:29:35,268: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9004b2db-7f06-4b57-a57d-7b4c7631e70a
[2017-05-10 13:29:35,268: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9004b2db-7f06-4b57-a57d-7b4c7631e70a
[2017-05-10 13:29:40,984: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:40,984: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:40,984: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:40,984: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:40,984: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cec8628e-69ac-4f86-a248-6261ff40b8d7
[2017-05-10 13:29:40,984: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cec8628e-69ac-4f86-a248-6261ff40b8d7
[2017-05-10 13:29:41,005: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cec8628e-69ac-4f86-a248-6261ff40b8d7
[2017-05-10 13:29:41,005: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cec8628e-69ac-4f86-a248-6261ff40b8d7
[2017-05-10 13:29:41,006: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.226', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-05-10 13:29:41,022: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3cec10f7-4fa0-4af3-a39f-72bf72aece50
[2017-05-10 13:29:41,022: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3cec10f7-4fa0-4af3-a39f-72bf72aece50
[2017-05-10 13:29:41,105: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:41,105: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:41,105: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:41,105: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:41,105: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3cec10f7-4fa0-4af3-a39f-72bf72aece50
[2017-05-10 13:29:41,105: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3cec10f7-4fa0-4af3-a39f-72bf72aece50
[2017-05-10 13:29:41,137: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3cec10f7-4fa0-4af3-a39f-72bf72aece50
[2017-05-10 13:29:41,137: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3cec10f7-4fa0-4af3-a39f-72bf72aece50
[2017-05-10 13:29:41,151: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 51853, 'pcs_password': 'n2tvO2nwh7YiT0625JuI'}
[2017-05-10 13:29:41,164: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f0405819-dcbd-4457-9e43-31eb24198431
[2017-05-10 13:29:41,164: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f0405819-dcbd-4457-9e43-31eb24198431
[2017-05-10 13:29:47,413: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:47,413: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:47,413: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:47,413: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:47,413: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9004b2db-7f06-4b57-a57d-7b4c7631e70a
[2017-05-10 13:29:47,413: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9004b2db-7f06-4b57-a57d-7b4c7631e70a
[2017-05-10 13:29:47,438: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9004b2db-7f06-4b57-a57d-7b4c7631e70a
[2017-05-10 13:29:47,438: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9004b2db-7f06-4b57-a57d-7b4c7631e70a
[2017-05-10 13:29:47,439: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': True, 'mcast_port': 53491, 'new_node_fqdn': u'lotus-32vm5.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'WMZGHPX1tfMtzycbr3uq'}
[2017-05-10 13:29:47,453: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc af650096-5aaa-4860-be60-c59d615a59fd
[2017-05-10 13:29:47,453: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc af650096-5aaa-4860-be60-c59d615a59fd
[2017-05-10 13:29:52,849: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:52,849: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:52,849: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:52,849: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:52,849: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 632f8c89-7f9a-4c1f-8e2c-61e7324fdee9
[2017-05-10 13:29:52,849: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 632f8c89-7f9a-4c1f-8e2c-61e7324fdee9
[2017-05-10 13:29:52,858: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 632f8c89-7f9a-4c1f-8e2c-61e7324fdee9
[2017-05-10 13:29:52,858: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 632f8c89-7f9a-4c1f-8e2c-61e7324fdee9
[2017-05-10 13:29:52,858: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.225', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-05-10 13:29:52,880: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ca97649d-29f4-4a77-91f4-c1307bc01c0f
[2017-05-10 13:29:52,880: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ca97649d-29f4-4a77-91f4-c1307bc01c0f
[2017-05-10 13:29:52,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:52,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:52,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:52,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:29:52,977: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ca97649d-29f4-4a77-91f4-c1307bc01c0f
[2017-05-10 13:29:52,977: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ca97649d-29f4-4a77-91f4-c1307bc01c0f
[2017-05-10 13:29:52,997: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ca97649d-29f4-4a77-91f4-c1307bc01c0f
[2017-05-10 13:29:52,997: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ca97649d-29f4-4a77-91f4-c1307bc01c0f
[2017-05-10 13:29:53,304: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:53,304: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:53,304: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:53,304: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:53,304: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f0405819-dcbd-4457-9e43-31eb24198431
[2017-05-10 13:29:53,304: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f0405819-dcbd-4457-9e43-31eb24198431
[2017-05-10 13:29:53,333: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f0405819-dcbd-4457-9e43-31eb24198431
[2017-05-10 13:29:53,333: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f0405819-dcbd-4457-9e43-31eb24198431
[2017-05-10 13:29:53,334: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': True, 'mcast_port': 51853, 'new_node_fqdn': u'lotus-32vm7.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'n2tvO2nwh7YiT0625JuI'}
[2017-05-10 13:29:53,348: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7fa7fee0-f254-44a4-9fef-ae5bcaf12910
[2017-05-10 13:29:53,348: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7fa7fee0-f254-44a4-9fef-ae5bcaf12910
[2017-05-10 13:29:53,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:53,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:53,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:53,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:29:53,527: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc af650096-5aaa-4860-be60-c59d615a59fd
[2017-05-10 13:29:53,527: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc af650096-5aaa-4860-be60-c59d615a59fd
[2017-05-10 13:29:53,573: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc af650096-5aaa-4860-be60-c59d615a59fd
[2017-05-10 13:29:53,573: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc af650096-5aaa-4860-be60-c59d615a59fd
[2017-05-10 13:29:53,574: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 13:29:53.574396+00:00:
[2017-05-10 13:29:53,574: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 13:29:53.574396+00:00:
[2017-05-10 13:29:53,574: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->53491
[2017-05-10 13:29:53,574: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->53491
[2017-05-10 13:29:53,586: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 13:29:53,586: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 13:29:53,617: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 53491, 'pcs_password': 'XIIcKreYDC4u7zsliJVm'}
[2017-05-10 13:29:53,633: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:29:53,634: INFO/job_scheduler]   Job 10 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-05-10 13:29:53,635: INFO/job_scheduler]   Job 15 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-05-10 13:29:53,635: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:29:53,636: INFO/job_scheduler] Job 10 finished 1 steps successfully
[2017-05-10 13:29:53,635: INFO/job_scheduler]   Job 18 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:29:53,637: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 4 tasked)
[2017-05-10 13:29:53,646: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f6acf45a-d2ef-4275-af2c-c4f67f366f3b
[2017-05-10 13:29:53,646: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f6acf45a-d2ef-4275-af2c-c4f67f366f3b
[2017-05-10 13:29:53,653: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 140430600505488
[2017-05-10 13:29:53,668: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-10 13:29:53,682: INFO/job] Job 10: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:29:53,685: INFO/job] Job 10: not_deleted=True
[2017-05-10 13:29:53,693: INFO/job_scheduler] Job 10: complete_job: Updating cache
[2017-05-10 13:29:53,694: INFO/job_scheduler] Job 10: locked_item 140430600505488 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 13:29:53,704: INFO/job_scheduler] Job 10 completing (errored=False, cancelled=False)
[2017-05-10 13:29:53,742: INFO/job_scheduler] run_next: 1 runnable jobs of (16 pending, 3 tasked)
[2017-05-10 13:29:53,751: INFO/job_scheduler] Job 15: RunJobThread.run
[2017-05-10 13:29:53,751: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-05-10 13:29:53,766: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 82dd99e1-5393-4aa6-837a-40cb62fbd28c
[2017-05-10 13:29:53,766: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 82dd99e1-5393-4aa6-837a-40cb62fbd28c
[2017-05-10 13:29:58,673: INFO/job_scheduler] run_next: 0 runnable jobs of (15 pending, 4 tasked)
[2017-05-10 13:29:58,696: INFO/job_scheduler] run_next: 0 runnable jobs of (15 pending, 4 tasked)
[2017-05-10 13:29:59,449: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:59,449: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:59,450: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:59,450: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:29:59,450: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7fa7fee0-f254-44a4-9fef-ae5bcaf12910
[2017-05-10 13:29:59,450: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7fa7fee0-f254-44a4-9fef-ae5bcaf12910
[2017-05-10 13:29:59,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7fa7fee0-f254-44a4-9fef-ae5bcaf12910
[2017-05-10 13:29:59,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7fa7fee0-f254-44a4-9fef-ae5bcaf12910
[2017-05-10 13:29:59,468: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 13:29:59.468325+00:00:
[2017-05-10 13:29:59,468: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 13:29:59.468325+00:00:
[2017-05-10 13:29:59,468: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->51853
[2017-05-10 13:29:59,468: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->51853
[2017-05-10 13:29:59,470: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 13:29:59,470: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 13:29:59,483: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:29:59,484: INFO/job_scheduler] Job 34 finished 1 steps successfully
[2017-05-10 13:29:59,485: INFO/job_scheduler]   Job 34 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-05-10 13:29:59,485: INFO/job_scheduler]   Job 37 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-05-10 13:29:59,485: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:29:59,486: INFO/job_scheduler]   Job 40 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:29:59,486: INFO/job_scheduler] run_next: 0 runnable jobs of (15 pending, 4 tasked)
[2017-05-10 13:29:59,633: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 140430465895632
[2017-05-10 13:29:59,647: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-10 13:29:59,666: INFO/job] Job 34: StateChangeJob complete, setting state stopped on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:29:59,667: INFO/job] Job 34: not_deleted=True
[2017-05-10 13:29:59,667: INFO/job_scheduler] Job 34: complete_job: Updating cache
[2017-05-10 13:29:59,667: INFO/job_scheduler] Job 34: locked_item 140430465895632 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 13:29:59,673: INFO/job_scheduler] Job 34 completing (errored=False, cancelled=False)
[2017-05-10 13:29:59,721: INFO/job_scheduler] run_next: 1 runnable jobs of (15 pending, 3 tasked)
[2017-05-10 13:29:59,759: INFO/job_scheduler] Job 37: RunJobThread.run
[2017-05-10 13:29:59,760: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-05-10 13:29:59,779: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4f072c6b-f29d-4e2b-a98d-046b4eb5b3d1
[2017-05-10 13:29:59,779: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4f072c6b-f29d-4e2b-a98d-046b4eb5b3d1
[2017-05-10 13:30:01,061: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:30:01,061: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:30:01,062: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:30:01,062: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:30:01,062: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8fa0d864-bd29-4496-bfcd-2efe78f6e26a
[2017-05-10 13:30:01,062: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8fa0d864-bd29-4496-bfcd-2efe78f6e26a
[2017-05-10 13:30:01,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8fa0d864-bd29-4496-bfcd-2efe78f6e26a
[2017-05-10 13:30:01,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8fa0d864-bd29-4496-bfcd-2efe78f6e26a
[2017-05-10 13:30:01,089: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.227', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-05-10 13:30:01,107: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 813cf759-c2e8-4255-af32-131aadd2e1de
[2017-05-10 13:30:01,107: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 813cf759-c2e8-4255-af32-131aadd2e1de
[2017-05-10 13:30:01,180: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:30:01,180: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:30:01,180: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:30:01,180: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:30:01,180: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 813cf759-c2e8-4255-af32-131aadd2e1de
[2017-05-10 13:30:01,180: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 813cf759-c2e8-4255-af32-131aadd2e1de
[2017-05-10 13:30:01,223: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 813cf759-c2e8-4255-af32-131aadd2e1de
[2017-05-10 13:30:01,223: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 813cf759-c2e8-4255-af32-131aadd2e1de
[2017-05-10 13:30:01,238: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 51853, 'pcs_password': 'hGp9Dx90Hb2pUakRjOB8'}
[2017-05-10 13:30:01,252: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bf495a12-905e-42da-bf22-80fa471dde6a
[2017-05-10 13:30:01,252: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bf495a12-905e-42da-bf22-80fa471dde6a
[2017-05-10 13:30:01,905: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:30:01,905: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:30:01,905: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:30:01,905: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:30:01,905: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 82dd99e1-5393-4aa6-837a-40cb62fbd28c
[2017-05-10 13:30:01,905: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 82dd99e1-5393-4aa6-837a-40cb62fbd28c
[2017-05-10 13:30:01,937: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 82dd99e1-5393-4aa6-837a-40cb62fbd28c
[2017-05-10 13:30:01,937: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 82dd99e1-5393-4aa6-837a-40cb62fbd28c
[2017-05-10 13:30:01,937: INFO/job_scheduler] Job 15 finished 1 steps successfully
[2017-05-10 13:30:01,979: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 140430600505488
[2017-05-10 13:30:02,036: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 13:30:02,064: INFO/job] Job 15: StateChangeJob complete, setting state started on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:30:02,064: INFO/job] Job 15: not_deleted=True
[2017-05-10 13:30:02,064: INFO/job_scheduler] Job 15: complete_job: Updating cache
[2017-05-10 13:30:02,064: INFO/job_scheduler] Job 15: locked_item 140430600505488 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 13:30:02,072: INFO/job_scheduler] Job 15 completing (errored=False, cancelled=False)
[2017-05-10 13:30:02,133: INFO/job_scheduler] run_next: 1 runnable jobs of (14 pending, 3 tasked)
[2017-05-10 13:30:02,174: INFO/job_scheduler] Job 16: RunJobThread.run
[2017-05-10 13:30:02,176: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 13:30:02,218: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1db20609-4dae-43b0-988c-6b2696b16c77
[2017-05-10 13:30:02,218: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1db20609-4dae-43b0-988c-6b2696b16c77
[2017-05-10 13:30:05,783: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:30:05,783: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:30:05,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:30:05,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:30:05,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f6acf45a-d2ef-4275-af2c-c4f67f366f3b
[2017-05-10 13:30:05,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f6acf45a-d2ef-4275-af2c-c4f67f366f3b
[2017-05-10 13:30:05,821: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f6acf45a-d2ef-4275-af2c-c4f67f366f3b
[2017-05-10 13:30:05,821: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f6acf45a-d2ef-4275-af2c-c4f67f366f3b
[2017-05-10 13:30:05,822: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': False, 'mcast_port': 53491, 'new_node_fqdn': u'lotus-32vm6.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'XIIcKreYDC4u7zsliJVm'}
[2017-05-10 13:30:05,837: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 76cbce98-c385-469e-a0a0-7be91e01262e
[2017-05-10 13:30:05,837: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 76cbce98-c385-469e-a0a0-7be91e01262e
[2017-05-10 13:30:07,871: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:30:07,871: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:30:07,871: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:30:07,871: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:30:07,871: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4f072c6b-f29d-4e2b-a98d-046b4eb5b3d1
[2017-05-10 13:30:07,871: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4f072c6b-f29d-4e2b-a98d-046b4eb5b3d1
[2017-05-10 13:30:07,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4f072c6b-f29d-4e2b-a98d-046b4eb5b3d1
[2017-05-10 13:30:07,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4f072c6b-f29d-4e2b-a98d-046b4eb5b3d1
[2017-05-10 13:30:07,898: INFO/job_scheduler] Job 37 finished 1 steps successfully
[2017-05-10 13:30:08,013: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 140430465895632
[2017-05-10 13:30:08,045: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 13:30:08,063: INFO/job] Job 37: StateChangeJob complete, setting state started on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:30:08,065: INFO/job] Job 37: not_deleted=True
[2017-05-10 13:30:08,065: INFO/job_scheduler] Job 37: complete_job: Updating cache
[2017-05-10 13:30:08,066: INFO/job_scheduler] Job 37: locked_item 140430465895632 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 13:30:08,075: INFO/job_scheduler] Job 37 completing (errored=False, cancelled=False)
[2017-05-10 13:30:08,114: INFO/job_scheduler] run_next: 1 runnable jobs of (13 pending, 3 tasked)
[2017-05-10 13:30:08,141: INFO/job_scheduler] Job 38: RunJobThread.run
[2017-05-10 13:30:08,143: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 13:30:08,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a54639c3-e2bc-4928-9c26-996ebc045f8c
[2017-05-10 13:30:08,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a54639c3-e2bc-4928-9c26-996ebc045f8c
[2017-05-10 13:30:08,583: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-10 13:30:08,601: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-10 13:30:08,724: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-10 13:30:08,762: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-10 13:30:09,336: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:30:09,336: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:30:09,337: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:30:09,337: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:30:09,337: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1db20609-4dae-43b0-988c-6b2696b16c77
[2017-05-10 13:30:09,337: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1db20609-4dae-43b0-988c-6b2696b16c77
[2017-05-10 13:30:09,386: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1db20609-4dae-43b0-988c-6b2696b16c77
[2017-05-10 13:30:09,386: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1db20609-4dae-43b0-988c-6b2696b16c77
[2017-05-10 13:30:09,387: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-05-10 13:30:09,408: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0ede9090-865b-4c0a-af84-e4fb2089208f
[2017-05-10 13:30:09,408: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0ede9090-865b-4c0a-af84-e4fb2089208f
[2017-05-10 13:30:11,954: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:30:11,954: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:30:11,954: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:30:11,954: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:30:11,954: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 76cbce98-c385-469e-a0a0-7be91e01262e
[2017-05-10 13:30:11,954: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 76cbce98-c385-469e-a0a0-7be91e01262e
[2017-05-10 13:30:12,005: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 76cbce98-c385-469e-a0a0-7be91e01262e
[2017-05-10 13:30:12,005: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 76cbce98-c385-469e-a0a0-7be91e01262e
[2017-05-10 13:30:12,006: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 13:30:12.006271+00:00:
[2017-05-10 13:30:12,006: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 13:30:12.006271+00:00:
[2017-05-10 13:30:12,006: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->53491
[2017-05-10 13:30:12,006: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->53491
[2017-05-10 13:30:12,008: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 13:30:12,008: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 13:30:12,023: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:30:12,023: INFO/job_scheduler]   Job 23 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-05-10 13:30:12,023: INFO/job_scheduler]   Job 26 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-05-10 13:30:12,023: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:30:12,023: INFO/job_scheduler]   Job 29 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:30:12,024: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-05-10 13:30:12,025: INFO/job_scheduler] Job 23 finished 1 steps successfully
[2017-05-10 13:30:12,143: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 140430600525456
[2017-05-10 13:30:12,151: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-10 13:30:12,159: INFO/job] Job 23: StateChangeJob complete, setting state stopped on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:30:12,159: INFO/job] Job 23: not_deleted=True
[2017-05-10 13:30:12,160: INFO/job_scheduler] Job 23: complete_job: Updating cache
[2017-05-10 13:30:12,161: INFO/job_scheduler] Job 23: locked_item 140430600525456 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 13:30:12,175: INFO/job_scheduler] Job 23 completing (errored=False, cancelled=False)
[2017-05-10 13:30:12,252: INFO/job_scheduler] run_next: 1 runnable jobs of (12 pending, 3 tasked)
[2017-05-10 13:30:12,286: INFO/job_scheduler] Job 26: RunJobThread.run
[2017-05-10 13:30:12,287: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-05-10 13:30:12,324: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ea03d0e8-826c-43f7-87f6-493cdceeb512
[2017-05-10 13:30:12,324: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ea03d0e8-826c-43f7-87f6-493cdceeb512
[2017-05-10 13:30:13,368: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:30:13,368: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:30:13,369: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:30:13,369: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:30:13,369: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bf495a12-905e-42da-bf22-80fa471dde6a
[2017-05-10 13:30:13,369: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bf495a12-905e-42da-bf22-80fa471dde6a
[2017-05-10 13:30:13,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bf495a12-905e-42da-bf22-80fa471dde6a
[2017-05-10 13:30:13,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bf495a12-905e-42da-bf22-80fa471dde6a
[2017-05-10 13:30:13,382: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': False, 'mcast_port': 51853, 'new_node_fqdn': u'lotus-32vm8.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'hGp9Dx90Hb2pUakRjOB8'}
[2017-05-10 13:30:13,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ba6a92e8-0950-43c8-9b0b-c70b4d80489b
[2017-05-10 13:30:13,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ba6a92e8-0950-43c8-9b0b-c70b4d80489b
[2017-05-10 13:30:15,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:30:15,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:30:15,316: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:30:15,316: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:30:15,316: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a54639c3-e2bc-4928-9c26-996ebc045f8c
[2017-05-10 13:30:15,316: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a54639c3-e2bc-4928-9c26-996ebc045f8c
[2017-05-10 13:30:15,321: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a54639c3-e2bc-4928-9c26-996ebc045f8c
[2017-05-10 13:30:15,321: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a54639c3-e2bc-4928-9c26-996ebc045f8c
[2017-05-10 13:30:15,324: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-05-10 13:30:15,344: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c83e98ba-71c0-4ec0-943d-1ad524dc1aa6
[2017-05-10 13:30:15,344: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c83e98ba-71c0-4ec0-943d-1ad524dc1aa6
[2017-05-10 13:30:17,193: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 4 tasked)
[2017-05-10 13:30:17,210: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 4 tasked)
[2017-05-10 13:30:18,766: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 4 tasked)
[2017-05-10 13:30:18,785: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 4 tasked)
[2017-05-10 13:30:20,468: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:30:20,468: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:30:20,468: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:30:20,468: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:30:20,468: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ea03d0e8-826c-43f7-87f6-493cdceeb512
[2017-05-10 13:30:20,468: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ea03d0e8-826c-43f7-87f6-493cdceeb512
[2017-05-10 13:30:20,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:30:20,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:30:20,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:30:20,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:30:20,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ba6a92e8-0950-43c8-9b0b-c70b4d80489b
[2017-05-10 13:30:20,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ba6a92e8-0950-43c8-9b0b-c70b4d80489b
[2017-05-10 13:30:20,502: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ea03d0e8-826c-43f7-87f6-493cdceeb512
[2017-05-10 13:30:20,502: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ea03d0e8-826c-43f7-87f6-493cdceeb512
[2017-05-10 13:30:20,503: INFO/job_scheduler] Job 26 finished 1 steps successfully
[2017-05-10 13:30:20,516: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ba6a92e8-0950-43c8-9b0b-c70b4d80489b
[2017-05-10 13:30:20,516: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ba6a92e8-0950-43c8-9b0b-c70b4d80489b
[2017-05-10 13:30:20,516: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 13:30:20.516725+00:00:
[2017-05-10 13:30:20,516: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 13:30:20.516725+00:00:
[2017-05-10 13:30:20,517: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->51853
[2017-05-10 13:30:20,517: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->51853
[2017-05-10 13:30:20,519: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 13:30:20,519: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 13:30:20,534: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:30:20,534: INFO/job_scheduler]   Job 45 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-05-10 13:30:20,534: INFO/job_scheduler]   Job 47 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-05-10 13:30:20,535: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:30:20,535: INFO/job_scheduler] Job 45 finished 1 steps successfully
[2017-05-10 13:30:20,536: INFO/job_scheduler]   Job 51 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:30:20,537: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 4 tasked)
[2017-05-10 13:30:20,602: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 140430600525456
[2017-05-10 13:30:20,633: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm6.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 13:30:20,645: INFO/job] Job 26: StateChangeJob complete, setting state started on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:30:20,645: INFO/job] Job 26: not_deleted=True
[2017-05-10 13:30:20,645: INFO/job_scheduler] Job 26: complete_job: Updating cache
[2017-05-10 13:30:20,646: INFO/job_scheduler] Job 26: locked_item 140430600525456 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 13:30:20,650: INFO/job_scheduler] Job 26 completing (errored=False, cancelled=False)
[2017-05-10 13:30:20,694: INFO/job_scheduler] run_next: 1 runnable jobs of (11 pending, 3 tasked)
[2017-05-10 13:30:20,740: INFO/job_scheduler] Job 27: RunJobThread.run
[2017-05-10 13:30:20,741: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 13:30:20,756: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b3a6f005-d87b-4de8-9172-186b2e50d71e
[2017-05-10 13:30:20,756: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b3a6f005-d87b-4de8-9172-186b2e50d71e
[2017-05-10 13:30:20,945: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 140430600493840
[2017-05-10 13:30:20,953: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-10 13:30:20,974: INFO/job] Job 45: StateChangeJob complete, setting state stopped on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:30:20,974: INFO/job] Job 45: not_deleted=True
[2017-05-10 13:30:20,974: INFO/job_scheduler] Job 45: complete_job: Updating cache
[2017-05-10 13:30:20,974: INFO/job_scheduler] Job 45: locked_item 140430600493840 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 13:30:20,982: INFO/job_scheduler] Job 45 completing (errored=False, cancelled=False)
[2017-05-10 13:30:21,048: INFO/job_scheduler] run_next: 1 runnable jobs of (10 pending, 3 tasked)
[2017-05-10 13:30:21,109: INFO/job_scheduler] Job 47: RunJobThread.run
[2017-05-10 13:30:21,110: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-05-10 13:30:21,173: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 711952ff-4b1f-49de-ba31-40a3dbd47af2
[2017-05-10 13:30:21,173: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 711952ff-4b1f-49de-ba31-40a3dbd47af2
[2017-05-10 13:30:27,204: INFO/job_scheduler] run_next: 0 runnable jobs of (9 pending, 4 tasked)
[2017-05-10 13:30:27,221: INFO/job_scheduler] run_next: 0 runnable jobs of (9 pending, 4 tasked)
[2017-05-10 13:30:27,249: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:30:27,249: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:30:27,250: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:30:27,250: INFO/job_scheduler] run_next: 0 runnable jobs of (9 pending, 4 tasked)
[2017-05-10 13:30:27,272: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:30:27,273: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:30:27,273: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:30:27,273: INFO/job_scheduler] run_next: 0 runnable jobs of (9 pending, 4 tasked)
[2017-05-10 13:30:27,854: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:30:27,854: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:30:27,855: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:30:27,855: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:30:27,855: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b3a6f005-d87b-4de8-9172-186b2e50d71e
[2017-05-10 13:30:27,855: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b3a6f005-d87b-4de8-9172-186b2e50d71e
[2017-05-10 13:30:27,873: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b3a6f005-d87b-4de8-9172-186b2e50d71e
[2017-05-10 13:30:27,873: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b3a6f005-d87b-4de8-9172-186b2e50d71e
[2017-05-10 13:30:27,874: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-05-10 13:30:27,897: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fc667c22-ba46-45ed-b70a-e22e5339f0c5
[2017-05-10 13:30:27,897: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fc667c22-ba46-45ed-b70a-e22e5339f0c5
[2017-05-10 13:30:29,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:30:29,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:30:29,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:30:29,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:30:29,272: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 711952ff-4b1f-49de-ba31-40a3dbd47af2
[2017-05-10 13:30:29,272: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 711952ff-4b1f-49de-ba31-40a3dbd47af2
[2017-05-10 13:30:29,291: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 711952ff-4b1f-49de-ba31-40a3dbd47af2
[2017-05-10 13:30:29,291: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 711952ff-4b1f-49de-ba31-40a3dbd47af2
[2017-05-10 13:30:29,292: INFO/job_scheduler] Job 47 finished 1 steps successfully
[2017-05-10 13:30:29,339: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 140430600493840
[2017-05-10 13:30:29,375: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm8.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 13:30:29,387: INFO/job] Job 47: StateChangeJob complete, setting state started on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:30:29,387: INFO/job] Job 47: not_deleted=True
[2017-05-10 13:30:29,387: INFO/job_scheduler] Job 47: complete_job: Updating cache
[2017-05-10 13:30:29,388: INFO/job_scheduler] Job 47: locked_item 140430600493840 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 13:30:29,392: INFO/job_scheduler] Job 47 completing (errored=False, cancelled=False)
[2017-05-10 13:30:29,423: INFO/job_scheduler] run_next: 1 runnable jobs of (9 pending, 3 tasked)
[2017-05-10 13:30:29,468: INFO/job_scheduler] Job 50: RunJobThread.run
[2017-05-10 13:30:29,469: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 13:30:29,487: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 69ae86f0-396c-40b8-8255-14b618d51b02
[2017-05-10 13:30:29,487: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 69ae86f0-396c-40b8-8255-14b618d51b02
[2017-05-10 13:30:31,172: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:31,187: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:36,597: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:30:36,597: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:30:36,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:30:36,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:30:36,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 69ae86f0-396c-40b8-8255-14b618d51b02
[2017-05-10 13:30:36,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 69ae86f0-396c-40b8-8255-14b618d51b02
[2017-05-10 13:30:36,604: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 69ae86f0-396c-40b8-8255-14b618d51b02
[2017-05-10 13:30:36,604: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 69ae86f0-396c-40b8-8255-14b618d51b02
[2017-05-10 13:30:36,606: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-05-10 13:30:36,637: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 12d094d8-894b-4aa2-bc61-3675afdc8914
[2017-05-10 13:30:36,637: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 12d094d8-894b-4aa2-bc61-3675afdc8914
[2017-05-10 13:30:37,242: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:37,259: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:38,794: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:38,809: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:38,839: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:30:38,839: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:30:38,839: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:30:38,839: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:38,859: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:30:38,859: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:30:38,859: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:30:38,859: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:41,218: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:41,235: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:41,284: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:30:41,284: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:30:41,284: INFO/job_scheduler]   Job 40 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:30:41,284: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:41,296: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:30:41,296: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:30:41,296: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:30:41,297: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:41,317: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:30:41,317: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:30:41,318: INFO/job_scheduler]   Job 51 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:30:41,318: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:41,332: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:30:41,332: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:30:41,332: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:30:41,332: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:47,300: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:47,315: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:47,341: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:30:47,348: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:30:47,348: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:30:47,348: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:47,361: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:30:47,361: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:30:47,361: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:30:47,361: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:48,695: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:48,718: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:48,750: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:30:48,750: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:30:48,751: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:30:48,751: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:48,778: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:30:48,779: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:30:48,779: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:30:48,779: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:48,866: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:30:48,867: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:30:48,867: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:30:48,867: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:48,886: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:30:48,886: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:30:48,887: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:30:48,887: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:51,213: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:51,228: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:51,256: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:30:51,257: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:30:51,257: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:30:51,257: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:51,275: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:30:51,275: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:30:51,276: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:30:51,276: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:57,320: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:30:57,320: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:30:57,321: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:30:57,321: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:57,343: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:30:57,343: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:30:57,343: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:30:57,343: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:58,720: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:30:58,720: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:30:58,720: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:30:58,720: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:58,742: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:30:58,742: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:30:58,742: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:30:58,742: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:58,898: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:30:58,899: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:30:58,899: INFO/job_scheduler]   Job 18 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:30:58,899: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:58,924: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:30:58,924: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:30:58,924: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:30:58,925: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:59,003: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:30:59,003: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:30:59,003: INFO/job_scheduler]   Job 29 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:30:59,004: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:30:59,019: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:30:59,020: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:30:59,020: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:30:59,020: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:31:01,252: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:01,252: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:31:01,253: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:01,253: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:31:01,271: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:01,272: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:31:01,272: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:01,272: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:31:07,478: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:31:07,479: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:07,479: INFO/job_scheduler]   Job 18 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:07,479: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:31:07,495: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:07,495: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:31:07,495: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:07,495: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:31:07,517: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:31:07,517: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:07,517: INFO/job_scheduler]   Job 29 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:07,518: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:31:07,534: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:07,534: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:31:07,534: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:07,534: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:31:08,784: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:31:08,784: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:08,784: INFO/job_scheduler]   Job 40 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:08,785: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:31:08,799: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:08,799: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:31:08,799: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:08,799: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:31:08,845: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:31:08,845: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:08,845: INFO/job_scheduler]   Job 51 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:08,846: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:31:08,868: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:08,868: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:31:08,868: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:08,869: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:31:08,943: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:31:08,943: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:08,943: INFO/job_scheduler]   Job 18 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:08,943: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:31:08,967: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:08,967: INFO/job_scheduler]   Job 16 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:31:08,967: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:08,967: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:31:09,000: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:31:09,000: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:09,000: INFO/job_scheduler]   Job 29 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:09,001: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:31:09,036: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:09,036: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:31:09,036: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:09,036: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:31:09,705: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:31:09,705: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:31:09,706: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:31:09,706: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:31:09,706: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0ede9090-865b-4c0a-af84-e4fb2089208f
[2017-05-10 13:31:09,706: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0ede9090-865b-4c0a-af84-e4fb2089208f
[2017-05-10 13:31:09,744: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0ede9090-865b-4c0a-af84-e4fb2089208f
[2017-05-10 13:31:09,744: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0ede9090-865b-4c0a-af84-e4fb2089208f
[2017-05-10 13:31:09,747: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-10 13:31:09,770: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7e2ec375-00f4-45ca-8c9c-157134de73d5
[2017-05-10 13:31:09,770: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7e2ec375-00f4-45ca-8c9c-157134de73d5
[2017-05-10 13:31:11,314: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:31:11,314: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:11,314: INFO/job_scheduler]   Job 40 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:11,314: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:31:11,331: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:11,331: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:31:11,331: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:11,331: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:31:11,351: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:31:11,351: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:11,351: INFO/job_scheduler]   Job 51 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:11,351: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:31:11,367: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:11,367: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:31:11,367: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:11,367: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-05-10 13:31:16,902: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:31:16,902: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:31:16,902: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:31:16,902: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:31:16,902: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7e2ec375-00f4-45ca-8c9c-157134de73d5
[2017-05-10 13:31:16,902: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7e2ec375-00f4-45ca-8c9c-157134de73d5
[2017-05-10 13:31:16,936: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7e2ec375-00f4-45ca-8c9c-157134de73d5
[2017-05-10 13:31:16,936: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7e2ec375-00f4-45ca-8c9c-157134de73d5
[2017-05-10 13:31:16,937: INFO/job_scheduler] Job 16 finished 3 steps successfully
[2017-05-10 13:31:17,062: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 140430465894544
[2017-05-10 13:31:17,071: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-10 13:31:17,084: INFO/job] Job 16: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 13:31:17,085: INFO/job] Job 16: not_deleted=True
[2017-05-10 13:31:17,085: INFO/job_scheduler] Job 16: complete_job: Updating cache
[2017-05-10 13:31:17,085: INFO/job_scheduler] Job 16: locked_item 140430465894544 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 13:31:17,088: INFO/job_scheduler] Job 16 completing (errored=False, cancelled=False)
[2017-05-10 13:31:17,171: INFO/job_scheduler] run_next: 1 runnable jobs of (8 pending, 3 tasked)
[2017-05-10 13:31:17,187: INFO/job_scheduler] Job 18: RunJobThread.run
[2017-05-10 13:31:17,188: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 13:31:17,203: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 10e947c1-38e2-4617-bc01-041c647cf0b1
[2017-05-10 13:31:17,203: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 10e947c1-38e2-4617-bc01-041c647cf0b1
[2017-05-10 13:31:17,525: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:31:17,525: INFO/job_scheduler]   Job 18 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:17,525: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-05-10 13:31:17,547: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:17,547: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:17,547: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-05-10 13:31:17,563: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:31:17,564: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:17,564: INFO/job_scheduler]   Job 29 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:17,564: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-05-10 13:31:17,577: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:17,578: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:31:17,578: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:17,578: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-05-10 13:31:18,786: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:31:18,786: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:31:18,786: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:31:18,786: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:31:18,787: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c83e98ba-71c0-4ec0-943d-1ad524dc1aa6
[2017-05-10 13:31:18,787: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c83e98ba-71c0-4ec0-943d-1ad524dc1aa6
[2017-05-10 13:31:18,799: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c83e98ba-71c0-4ec0-943d-1ad524dc1aa6
[2017-05-10 13:31:18,799: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c83e98ba-71c0-4ec0-943d-1ad524dc1aa6
[2017-05-10 13:31:18,801: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-10 13:31:18,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a8592c35-e66d-4ff3-be09-f50aa2a5f8b7
[2017-05-10 13:31:18,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a8592c35-e66d-4ff3-be09-f50aa2a5f8b7
[2017-05-10 13:31:18,853: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:31:18,854: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:18,854: INFO/job_scheduler]   Job 40 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:18,854: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-05-10 13:31:18,886: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:18,886: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:31:18,886: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:18,887: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-05-10 13:31:18,914: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:31:18,914: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:18,915: INFO/job_scheduler]   Job 51 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:18,915: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-05-10 13:31:18,930: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:18,930: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:31:18,930: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:18,930: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-05-10 13:31:21,350: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:31:21,350: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:21,351: INFO/job_scheduler]   Job 40 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:21,351: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-05-10 13:31:21,367: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:21,368: INFO/job_scheduler]   Job 38 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:31:21,368: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:21,368: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-05-10 13:31:21,386: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:31:21,386: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:21,386: INFO/job_scheduler]   Job 51 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:21,386: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-05-10 13:31:21,399: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:21,399: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:31:21,399: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:21,412: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-05-10 13:31:22,250: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:31:22,250: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:31:22,250: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:31:22,250: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:31:22,250: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fc667c22-ba46-45ed-b70a-e22e5339f0c5
[2017-05-10 13:31:22,250: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fc667c22-ba46-45ed-b70a-e22e5339f0c5
[2017-05-10 13:31:22,295: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fc667c22-ba46-45ed-b70a-e22e5339f0c5
[2017-05-10 13:31:22,295: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fc667c22-ba46-45ed-b70a-e22e5339f0c5
[2017-05-10 13:31:22,297: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-10 13:31:22,311: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8e647f63-dd7b-40d3-bd2d-8bd495ad8480
[2017-05-10 13:31:22,311: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8e647f63-dd7b-40d3-bd2d-8bd495ad8480
[2017-05-10 13:31:24,323: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:31:24,323: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:31:24,323: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:31:24,323: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:31:24,323: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 10e947c1-38e2-4617-bc01-041c647cf0b1
[2017-05-10 13:31:24,323: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 10e947c1-38e2-4617-bc01-041c647cf0b1
[2017-05-10 13:31:24,371: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 10e947c1-38e2-4617-bc01-041c647cf0b1
[2017-05-10 13:31:24,371: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 10e947c1-38e2-4617-bc01-041c647cf0b1
[2017-05-10 13:31:24,372: INFO/job_scheduler] Job 18 finished 1 steps successfully
[2017-05-10 13:31:24,444: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 140430465894544
[2017-05-10 13:31:24,478: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 13:31:24,497: INFO/job] Job 18: StateChangeJob complete, setting state started on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 13:31:24,498: INFO/job] Job 18: not_deleted=True
[2017-05-10 13:31:24,500: INFO/job_scheduler] Job 18: complete_job: Updating cache
[2017-05-10 13:31:24,500: INFO/job_scheduler] Job 18: locked_item 140430465894544 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 13:31:24,504: INFO/job_scheduler] Job 18 completing (errored=False, cancelled=False)
[2017-05-10 13:31:24,525: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-1
[2017-05-10 13:31:24,526: INFO/job_scheduler] _notify: Updating .mcast_port of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 53491
[2017-05-10 13:31:24,559: INFO/job_scheduler] _notify: Set mcast_port=53491 on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-05-10 13:31:24,561: INFO/job_scheduler] _notify: Updating .network_interfaces of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-05-10 13:31:24,571: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-05-10 13:31:24,578: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-10 13:31:24,583: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-05-10 13:31:24,602: INFO/job_scheduler] run_next: 1 runnable jobs of (7 pending, 3 tasked)
[2017-05-10 13:31:24,625: INFO/job_scheduler] Job 19: RunJobThread.run
[2017-05-10 13:31:24,628: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-05-10 13:31:24,644: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7135732f-6837-4194-aac1-7f099a11c1e0
[2017-05-10 13:31:24,644: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7135732f-6837-4194-aac1-7f099a11c1e0
[2017-05-10 13:31:25,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:31:25,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:31:25,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:31:25,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:31:25,909: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a8592c35-e66d-4ff3-be09-f50aa2a5f8b7
[2017-05-10 13:31:25,909: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a8592c35-e66d-4ff3-be09-f50aa2a5f8b7
[2017-05-10 13:31:25,957: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a8592c35-e66d-4ff3-be09-f50aa2a5f8b7
[2017-05-10 13:31:25,957: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a8592c35-e66d-4ff3-be09-f50aa2a5f8b7
[2017-05-10 13:31:25,958: INFO/job_scheduler] Job 38 finished 3 steps successfully
[2017-05-10 13:31:26,019: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 140430465889360
[2017-05-10 13:31:26,027: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-10 13:31:26,043: INFO/job] Job 38: StateChangeJob complete, setting state stopped on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 13:31:26,043: INFO/job] Job 38: not_deleted=True
[2017-05-10 13:31:26,043: INFO/job_scheduler] Job 38: complete_job: Updating cache
[2017-05-10 13:31:26,043: INFO/job_scheduler] Job 38: locked_item 140430465889360 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 13:31:26,046: INFO/job_scheduler] Job 38 completing (errored=False, cancelled=False)
[2017-05-10 13:31:26,077: INFO/job_scheduler] run_next: 1 runnable jobs of (6 pending, 3 tasked)
[2017-05-10 13:31:26,105: INFO/job_scheduler] Job 40: RunJobThread.run
[2017-05-10 13:31:26,106: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 13:31:26,125: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2a26144b-99fe-4881-9401-e730adf914d1
[2017-05-10 13:31:26,125: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2a26144b-99fe-4881-9401-e730adf914d1
[2017-05-10 13:31:27,633: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-10 13:31:27,647: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-10 13:31:28,964: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:28,964: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:28,964: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-10 13:31:28,985: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:31:28,985: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:28,986: INFO/job_scheduler]   Job 29 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:28,986: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-10 13:31:29,001: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:29,001: INFO/job_scheduler]   Job 27 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:31:29,002: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:29,002: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-10 13:31:30,405: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:31:30,405: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:31:30,405: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:31:30,405: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:31:30,406: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8e647f63-dd7b-40d3-bd2d-8bd495ad8480
[2017-05-10 13:31:30,406: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8e647f63-dd7b-40d3-bd2d-8bd495ad8480
[2017-05-10 13:31:30,428: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8e647f63-dd7b-40d3-bd2d-8bd495ad8480
[2017-05-10 13:31:30,428: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8e647f63-dd7b-40d3-bd2d-8bd495ad8480
[2017-05-10 13:31:30,429: INFO/job_scheduler] Job 27 finished 3 steps successfully
[2017-05-10 13:31:30,494: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 140430600471824
[2017-05-10 13:31:30,501: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-10 13:31:30,504: INFO/job] Job 27: StateChangeJob complete, setting state stopped on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 13:31:30,505: INFO/job] Job 27: not_deleted=True
[2017-05-10 13:31:30,505: INFO/job_scheduler] Job 27: complete_job: Updating cache
[2017-05-10 13:31:30,507: INFO/job_scheduler] Job 27: locked_item 140430600471824 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 13:31:30,513: INFO/job_scheduler] Job 27 completing (errored=False, cancelled=False)
[2017-05-10 13:31:30,586: INFO/job_scheduler] run_next: 1 runnable jobs of (5 pending, 3 tasked)
[2017-05-10 13:31:30,608: INFO/job_scheduler] Job 29: RunJobThread.run
[2017-05-10 13:31:30,608: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 13:31:30,664: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b65a6c4c-724b-40f9-b5ab-f14ffc394616
[2017-05-10 13:31:30,664: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b65a6c4c-724b-40f9-b5ab-f14ffc394616
[2017-05-10 13:31:30,983: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:31:30,983: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:31:30,984: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:31:30,984: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:31:30,984: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 12d094d8-894b-4aa2-bc61-3675afdc8914
[2017-05-10 13:31:30,984: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 12d094d8-894b-4aa2-bc61-3675afdc8914
[2017-05-10 13:31:31,030: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 12d094d8-894b-4aa2-bc61-3675afdc8914
[2017-05-10 13:31:31,030: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 12d094d8-894b-4aa2-bc61-3675afdc8914
[2017-05-10 13:31:31,032: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-10 13:31:31,045: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 64f0355b-a314-4473-ae4b-5759511675da
[2017-05-10 13:31:31,045: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 64f0355b-a314-4473-ae4b-5759511675da
[2017-05-10 13:31:31,338: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:31:31,339: INFO/job_scheduler]   Job 40 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:31,339: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 4 tasked)
[2017-05-10 13:31:31,355: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:31,355: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:31,355: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 4 tasked)
[2017-05-10 13:31:31,375: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:31:31,376: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:31,376: INFO/job_scheduler]   Job 51 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:31,376: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 4 tasked)
[2017-05-10 13:31:31,391: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:31,391: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:31:31,391: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:31,391: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 4 tasked)
[2017-05-10 13:31:33,238: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:31:33,238: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:31:33,238: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:31:33,238: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:31:33,239: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2a26144b-99fe-4881-9401-e730adf914d1
[2017-05-10 13:31:33,239: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2a26144b-99fe-4881-9401-e730adf914d1
[2017-05-10 13:31:33,240: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2a26144b-99fe-4881-9401-e730adf914d1
[2017-05-10 13:31:33,240: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2a26144b-99fe-4881-9401-e730adf914d1
[2017-05-10 13:31:33,241: INFO/job_scheduler] Job 40 finished 1 steps successfully
[2017-05-10 13:31:33,297: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 140430465889360
[2017-05-10 13:31:33,340: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 13:31:33,353: INFO/job] Job 40: StateChangeJob complete, setting state started on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 13:31:33,353: INFO/job] Job 40: not_deleted=True
[2017-05-10 13:31:33,353: INFO/job_scheduler] Job 40: complete_job: Updating cache
[2017-05-10 13:31:33,353: INFO/job_scheduler] Job 40: locked_item 140430465889360 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 13:31:33,360: INFO/job_scheduler] Job 40 completing (errored=False, cancelled=False)
[2017-05-10 13:31:33,384: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-3
[2017-05-10 13:31:33,384: INFO/job_scheduler] _notify: Updating .mcast_port of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 51853
[2017-05-10 13:31:33,389: INFO/job_scheduler] _notify: Set mcast_port=51853 on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-3) and saved
[2017-05-10 13:31:33,390: INFO/job_scheduler] _notify: Updating .network_interfaces of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-05-10 13:31:33,414: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-3) and saved
[2017-05-10 13:31:33,427: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-10 13:31:33,432: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-3) and saved
[2017-05-10 13:31:33,455: INFO/job_scheduler] run_next: 1 runnable jobs of (4 pending, 3 tasked)
[2017-05-10 13:31:33,483: INFO/job_scheduler] Job 41: RunJobThread.run
[2017-05-10 13:31:33,485: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-05-10 13:31:33,524: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6fcd799a-0900-472e-8e28-13af0c1ead48
[2017-05-10 13:31:33,524: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6fcd799a-0900-472e-8e28-13af0c1ead48
[2017-05-10 13:31:37,615: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 4 tasked)
[2017-05-10 13:31:37,634: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 4 tasked)
[2017-05-10 13:31:37,665: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:37,665: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:37,665: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 4 tasked)
[2017-05-10 13:31:37,687: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:31:37,688: INFO/job_scheduler]   Job 29 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:37,688: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 4 tasked)
[2017-05-10 13:31:37,707: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:37,707: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:37,707: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 4 tasked)
[2017-05-10 13:31:37,774: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:31:37,774: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:31:37,774: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:31:37,774: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:31:37,775: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b65a6c4c-724b-40f9-b5ab-f14ffc394616
[2017-05-10 13:31:37,775: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b65a6c4c-724b-40f9-b5ab-f14ffc394616
[2017-05-10 13:31:37,793: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b65a6c4c-724b-40f9-b5ab-f14ffc394616
[2017-05-10 13:31:37,793: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b65a6c4c-724b-40f9-b5ab-f14ffc394616
[2017-05-10 13:31:37,793: INFO/job_scheduler] Job 29 finished 1 steps successfully
[2017-05-10 13:31:37,855: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 140430600471824
[2017-05-10 13:31:37,893: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm6.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 13:31:37,921: INFO/job] Job 29: StateChangeJob complete, setting state started on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 13:31:37,921: INFO/job] Job 29: not_deleted=True
[2017-05-10 13:31:37,921: INFO/job_scheduler] Job 29: complete_job: Updating cache
[2017-05-10 13:31:37,921: INFO/job_scheduler] Job 29: locked_item 140430600471824 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 13:31:37,937: INFO/job_scheduler] Job 29 completing (errored=False, cancelled=False)
[2017-05-10 13:31:37,975: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-2
[2017-05-10 13:31:37,975: INFO/job_scheduler] _notify: Updating .mcast_port of item 2 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 53491
[2017-05-10 13:31:37,983: INFO/job_scheduler] _notify: Set mcast_port=53491 on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-2) and saved
[2017-05-10 13:31:37,984: INFO/job_scheduler] _notify: Updating .network_interfaces of item 2 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-05-10 13:31:37,998: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-2) and saved
[2017-05-10 13:31:38,004: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 2 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-10 13:31:38,014: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-2) and saved
[2017-05-10 13:31:38,023: INFO/job_scheduler] run_next: 1 runnable jobs of (3 pending, 3 tasked)
[2017-05-10 13:31:38,057: INFO/job_scheduler] Job 30: RunJobThread.run
[2017-05-10 13:31:38,060: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-05-10 13:31:38,096: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 84794e19-dca1-4a33-800a-7878f3916809
[2017-05-10 13:31:38,096: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 84794e19-dca1-4a33-800a-7878f3916809
[2017-05-10 13:31:38,960: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:38,960: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:38,961: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-05-10 13:31:38,991: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:31:38,991: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:38,991: INFO/job_scheduler]   Job 51 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:31:38,991: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-05-10 13:31:39,019: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:39,020: INFO/job_scheduler]   Job 50 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:31:39,020: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:39,020: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-05-10 13:31:39,057: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:39,058: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:39,058: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-05-10 13:31:39,082: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:39,082: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:39,082: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-05-10 13:31:39,191: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:31:39,191: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:31:39,192: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:31:39,192: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:31:39,192: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 64f0355b-a314-4473-ae4b-5759511675da
[2017-05-10 13:31:39,192: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 64f0355b-a314-4473-ae4b-5759511675da
[2017-05-10 13:31:39,227: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 64f0355b-a314-4473-ae4b-5759511675da
[2017-05-10 13:31:39,227: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 64f0355b-a314-4473-ae4b-5759511675da
[2017-05-10 13:31:39,228: INFO/job_scheduler] Job 50 finished 3 steps successfully
[2017-05-10 13:31:39,410: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 140430600556112
[2017-05-10 13:31:39,422: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-10 13:31:39,435: INFO/job] Job 50: StateChangeJob complete, setting state stopped on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 13:31:39,437: INFO/job] Job 50: not_deleted=True
[2017-05-10 13:31:39,438: INFO/job_scheduler] Job 50: complete_job: Updating cache
[2017-05-10 13:31:39,438: INFO/job_scheduler] Job 50: locked_item 140430600556112 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 13:31:39,458: INFO/job_scheduler] Job 50 completing (errored=False, cancelled=False)
[2017-05-10 13:31:39,493: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 3 tasked)
[2017-05-10 13:31:39,534: INFO/job_scheduler] Job 51: RunJobThread.run
[2017-05-10 13:31:39,535: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 13:31:39,572: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fedc7033-4b60-44cc-8882-c7aa4860239f
[2017-05-10 13:31:39,572: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fedc7033-4b60-44cc-8882-c7aa4860239f
[2017-05-10 13:31:46,701: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:31:46,701: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:31:46,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:31:46,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:31:46,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fedc7033-4b60-44cc-8882-c7aa4860239f
[2017-05-10 13:31:46,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fedc7033-4b60-44cc-8882-c7aa4860239f
[2017-05-10 13:31:46,739: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fedc7033-4b60-44cc-8882-c7aa4860239f
[2017-05-10 13:31:46,739: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fedc7033-4b60-44cc-8882-c7aa4860239f
[2017-05-10 13:31:46,740: INFO/job_scheduler] Job 51 finished 1 steps successfully
[2017-05-10 13:31:46,792: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 140430600556112
[2017-05-10 13:31:46,851: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm8.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 13:31:46,866: INFO/job] Job 51: StateChangeJob complete, setting state started on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 13:31:46,866: INFO/job] Job 51: not_deleted=True
[2017-05-10 13:31:46,867: INFO/job_scheduler] Job 51: complete_job: Updating cache
[2017-05-10 13:31:46,867: INFO/job_scheduler] Job 51: locked_item 140430600556112 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 13:31:46,872: INFO/job_scheduler] Job 51 completing (errored=False, cancelled=False)
[2017-05-10 13:31:46,897: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-4
[2017-05-10 13:31:46,906: INFO/job_scheduler] _notify: Updating .mcast_port of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 51853
[2017-05-10 13:31:46,910: INFO/job_scheduler] _notify: Set mcast_port=51853 on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-4) and saved
[2017-05-10 13:31:46,911: INFO/job_scheduler] _notify: Updating .network_interfaces of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-05-10 13:31:46,920: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-4) and saved
[2017-05-10 13:31:46,927: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-10 13:31:46,932: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-4) and saved
[2017-05-10 13:31:46,955: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 13:31:47,075: INFO/job_scheduler] Job 52: RunJobThread.run
[2017-05-10 13:31:47,077: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-05-10 13:31:47,104: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 57fe5259-e4d1-4bcf-b9b8-17baa017191d
[2017-05-10 13:31:47,104: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 57fe5259-e4d1-4bcf-b9b8-17baa017191d
[2017-05-10 13:31:47,619: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:47,620: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:47,620: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 13:31:47,637: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:47,637: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:47,638: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 13:31:48,998: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:48,998: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:48,998: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 13:31:49,020: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:49,021: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:49,021: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 13:31:49,063: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:49,063: INFO/job_scheduler]   Job 19 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:49,063: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 13:31:49,082: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:49,082: INFO/job_scheduler]   Job 30 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:49,082: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-05-10 13:31:50,964: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:31:50,964: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:31:50,965: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:31:50,965: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:31:50,965: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7135732f-6837-4194-aac1-7f099a11c1e0
[2017-05-10 13:31:50,965: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7135732f-6837-4194-aac1-7f099a11c1e0
[2017-05-10 13:31:50,983: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7135732f-6837-4194-aac1-7f099a11c1e0
[2017-05-10 13:31:50,983: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7135732f-6837-4194-aac1-7f099a11c1e0
[2017-05-10 13:31:50,984: INFO/job_scheduler] Job 19 finished 1 steps successfully
[2017-05-10 13:31:51,423: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 140430600504272
[2017-05-10 13:31:51,430: INFO/job] Job 19: StateChangeJob complete, setting state managed on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:31:51,430: INFO/job] Job 19: not_deleted=True
[2017-05-10 13:31:51,430: INFO/job_scheduler] Job 19: complete_job: Updating cache
[2017-05-10 13:31:51,430: INFO/job_scheduler] Job 19: locked_item 140430600504272 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 13:31:51,433: INFO/job_scheduler] Job 19 completing (errored=False, cancelled=False)
[2017-05-10 13:31:51,495: INFO/job_scheduler] Replaying 4 buffered notifications for ManagedHost-1
[2017-05-10 13:31:51,504: INFO/job_scheduler] _notify: Updating .immutable_state of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com) from True to False
[2017-05-10 13:31:51,513: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-1) and saved
[2017-05-10 13:31:51,536: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7fb87405d350> to [2]
[2017-05-10 13:31:51,563: INFO/job_scheduler] _notify: Set ha_cluster_peers=[2] on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-1) and saved
[2017-05-10 13:31:51,699: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 13:31:51,700: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:51,700: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:51,700: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 13:31:51,702: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:51,702: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:51,702: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 13:31:58,843: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:31:58,843: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:31:58,844: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:31:58,844: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:31:58,844: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6fcd799a-0900-472e-8e28-13af0c1ead48
[2017-05-10 13:31:58,844: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6fcd799a-0900-472e-8e28-13af0c1ead48
[2017-05-10 13:31:58,844: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6fcd799a-0900-472e-8e28-13af0c1ead48
[2017-05-10 13:31:58,844: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6fcd799a-0900-472e-8e28-13af0c1ead48
[2017-05-10 13:31:58,846: INFO/job_scheduler] Job 41 finished 1 steps successfully
[2017-05-10 13:31:59,045: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:59,046: INFO/job_scheduler]   Job 41 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:59,046: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 13:31:59,075: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:31:59,075: INFO/job_scheduler]   Job 52 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:31:59,075: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-05-10 13:31:59,646: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 140430465870032
[2017-05-10 13:31:59,652: INFO/job] Job 41: StateChangeJob complete, setting state managed on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:31:59,652: INFO/job] Job 41: not_deleted=True
[2017-05-10 13:31:59,652: INFO/job_scheduler] Job 41: complete_job: Updating cache
[2017-05-10 13:31:59,652: INFO/job_scheduler] Job 41: locked_item 140430465870032 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 13:31:59,656: INFO/job_scheduler] Job 41 completing (errored=False, cancelled=False)
[2017-05-10 13:31:59,735: INFO/job_scheduler] Replaying 4 buffered notifications for ManagedHost-3
[2017-05-10 13:31:59,748: INFO/job_scheduler] _notify: Updating .immutable_state of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com) from True to False
[2017-05-10 13:31:59,763: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-3) and saved
[2017-05-10 13:31:59,795: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7fb874037850> to [4]
[2017-05-10 13:31:59,828: INFO/job_scheduler] _notify: Set ha_cluster_peers=[4] on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-3) and saved
[2017-05-10 13:31:59,961: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-05-10 13:32:05,440: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:32:05,440: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:32:05,440: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:32:05,440: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:32:05,440: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 84794e19-dca1-4a33-800a-7878f3916809
[2017-05-10 13:32:05,440: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 84794e19-dca1-4a33-800a-7878f3916809
[2017-05-10 13:32:05,474: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 84794e19-dca1-4a33-800a-7878f3916809
[2017-05-10 13:32:05,474: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 84794e19-dca1-4a33-800a-7878f3916809
[2017-05-10 13:32:05,476: INFO/job_scheduler] Job 30 finished 1 steps successfully
[2017-05-10 13:32:06,024: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 140430600507216
[2017-05-10 13:32:06,029: INFO/job] Job 30: StateChangeJob complete, setting state managed on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:32:06,029: INFO/job] Job 30: not_deleted=True
[2017-05-10 13:32:06,029: INFO/job_scheduler] Job 30: complete_job: Updating cache
[2017-05-10 13:32:06,030: INFO/job_scheduler] Job 30: locked_item 140430600507216 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 13:32:06,032: INFO/job_scheduler] Job 30 completing (errored=False, cancelled=False)
[2017-05-10 13:32:06,129: INFO/job_scheduler] Replaying 4 buffered notifications for ManagedHost-2
[2017-05-10 13:32:06,171: INFO/job_scheduler] _notify: Updating .immutable_state of item 2 (lotus-32vm6.lotus.hpdd.lab.intel.com) from True to False
[2017-05-10 13:32:06,188: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-2) and saved
[2017-05-10 13:32:06,215: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 2 (lotus-32vm6.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7fb8840a3dd0> to [1]
[2017-05-10 13:32:06,238: INFO/job_scheduler] _notify: Set ha_cluster_peers=[1] on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-2) and saved
[2017-05-10 13:32:06,377: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 13:32:12,468: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:32:12,468: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:32:12,468: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:32:12,468: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:32:12,468: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 57fe5259-e4d1-4bcf-b9b8-17baa017191d
[2017-05-10 13:32:12,468: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 57fe5259-e4d1-4bcf-b9b8-17baa017191d
[2017-05-10 13:32:12,489: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 57fe5259-e4d1-4bcf-b9b8-17baa017191d
[2017-05-10 13:32:12,489: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 57fe5259-e4d1-4bcf-b9b8-17baa017191d
[2017-05-10 13:32:12,491: INFO/job_scheduler] Job 52 finished 1 steps successfully
[2017-05-10 13:32:12,807: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 140430465989648
[2017-05-10 13:32:12,814: INFO/job] Job 52: StateChangeJob complete, setting state managed on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:32:12,814: INFO/job] Job 52: not_deleted=True
[2017-05-10 13:32:12,815: INFO/job_scheduler] Job 52: complete_job: Updating cache
[2017-05-10 13:32:12,815: INFO/job_scheduler] Job 52: locked_item 140430465989648 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 13:32:12,818: INFO/job_scheduler] Job 52 completing (errored=False, cancelled=False)
[2017-05-10 13:32:12,904: INFO/job_scheduler] Replaying 4 buffered notifications for ManagedHost-4
[2017-05-10 13:32:12,922: INFO/job_scheduler] _notify: Updating .immutable_state of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com) from True to False
[2017-05-10 13:32:12,937: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-4) and saved
[2017-05-10 13:32:12,972: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7fb874054e50> to [3]
[2017-05-10 13:32:13,000: INFO/job_scheduler] _notify: Set ha_cluster_peers=[3] on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-4) and saved
[2017-05-10 13:32:13,151: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:32:32,042: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:32:32,044: INFO/command_plan] set_state: <class 'chroma_core.models.filesystem.ManagedFilesystem'>-1 transitioning from unavailable to available
[2017-05-10 13:32:32,470: INFO/command_plan] Created command 13 (Creating filesystem testfs) with 17 jobs
[2017-05-10 13:32:32,538: INFO/job_scheduler] run_next: 4 runnable jobs of (17 pending, 0 tasked)
[2017-05-10 13:32:33,192: INFO/job_scheduler] Job 53: RunJobThread.run
[2017-05-10 13:32:33,198: INFO/job_scheduler] Job 54: RunJobThread.run
[2017-05-10 13:32:33,206: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1', 'device_type': 'zfs'}
[2017-05-10 13:32:33,216: INFO/job_scheduler] Job 55: RunJobThread.run
[2017-05-10 13:32:33,218: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux'}
[2017-05-10 13:32:33,226: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1', 'device_type': 'zfs'}
[2017-05-10 13:32:33,229: INFO/job_scheduler] Job 56: RunJobThread.run
[2017-05-10 13:32:33,231: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3', 'device_type': 'zfs'}
[2017-05-10 13:32:33,234: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux'}
[2017-05-10 13:32:33,245: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3b0df835-199a-4d39-aa24-29c3dd889fe8
[2017-05-10 13:32:33,245: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3b0df835-199a-4d39-aa24-29c3dd889fe8
[2017-05-10 13:32:33,254: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5', 'device_type': 'zfs'}
[2017-05-10 13:32:33,254: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux'}
[2017-05-10 13:32:33,257: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3', 'device_type': 'zfs'}
[2017-05-10 13:32:33,258: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1', 'device_type': 'zfs'}
[2017-05-10 13:32:33,262: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 74cefed3-f5f2-47f7-9185-31d819a4c059
[2017-05-10 13:32:33,262: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 74cefed3-f5f2-47f7-9185-31d819a4c059
[2017-05-10 13:32:33,263: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3', 'device_type': 'zfs'}
[2017-05-10 13:32:33,275: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5', 'device_type': 'zfs'}
[2017-05-10 13:32:33,279: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e5e0d45b-f0e8-4da4-83d0-8ed409b1ba5d
[2017-05-10 13:32:33,279: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e5e0d45b-f0e8-4da4-83d0-8ed409b1ba5d
[2017-05-10 13:32:33,280: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5', 'device_type': 'zfs'}
[2017-05-10 13:32:33,313: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a5148445-212f-4dcd-8038-03c3c302a0c9
[2017-05-10 13:32:33,313: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a5148445-212f-4dcd-8038-03c3c302a0c9
[2017-05-10 13:32:33,345: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 24f62e1b-2bf5-4e79-abd4-c6fb8422c8cc
[2017-05-10 13:32:33,345: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 24f62e1b-2bf5-4e79-abd4-c6fb8422c8cc
[2017-05-10 13:32:33,379: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8e9f8df1-7dde-4daf-ade4-5325a792239b
[2017-05-10 13:32:33,379: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8e9f8df1-7dde-4daf-ade4-5325a792239b
[2017-05-10 13:32:33,411: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 52767f09-391d-44cd-b921-1d779563f679
[2017-05-10 13:32:33,411: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 52767f09-391d-44cd-b921-1d779563f679
[2017-05-10 13:32:33,441: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a7a22c34-d8e7-49c2-a137-49f6255dc618
[2017-05-10 13:32:33,441: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a7a22c34-d8e7-49c2-a137-49f6255dc618
[2017-05-10 13:32:33,457: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0561c5f5-6628-4ac5-abe5-ee9f1720a32a
[2017-05-10 13:32:33,457: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0561c5f5-6628-4ac5-abe5-ee9f1720a32a
[2017-05-10 13:32:33,457: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:32:33,457: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:32:33,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:32:33,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:32:33,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 74cefed3-f5f2-47f7-9185-31d819a4c059
[2017-05-10 13:32:33,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 74cefed3-f5f2-47f7-9185-31d819a4c059
[2017-05-10 13:32:33,459: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:32:33,459: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:32:33,459: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:32:33,459: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:32:33,459: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 24f62e1b-2bf5-4e79-abd4-c6fb8422c8cc
[2017-05-10 13:32:33,459: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 24f62e1b-2bf5-4e79-abd4-c6fb8422c8cc
[2017-05-10 13:32:33,480: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 24f62e1b-2bf5-4e79-abd4-c6fb8422c8cc
[2017-05-10 13:32:33,480: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 24f62e1b-2bf5-4e79-abd4-c6fb8422c8cc
[2017-05-10 13:32:33,493: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 74cefed3-f5f2-47f7-9185-31d819a4c059
[2017-05-10 13:32:33,493: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 74cefed3-f5f2-47f7-9185-31d819a4c059
[2017-05-10 13:32:33,496: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 833b07ba-d081-4377-acbd-5794f20f8244
[2017-05-10 13:32:33,496: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 833b07ba-d081-4377-acbd-5794f20f8244
[2017-05-10 13:32:33,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 71885345-d137-4e20-bc6c-fb6e2ec70d62
[2017-05-10 13:32:33,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 71885345-d137-4e20-bc6c-fb6e2ec70d62
[2017-05-10 13:32:33,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:32:33,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:32:33,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:32:33,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:32:33,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 52767f09-391d-44cd-b921-1d779563f679
[2017-05-10 13:32:33,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 52767f09-391d-44cd-b921-1d779563f679
[2017-05-10 13:32:33,556: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 41614756-17ef-4deb-a220-a97e2e5714ad
[2017-05-10 13:32:33,556: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 41614756-17ef-4deb-a220-a97e2e5714ad
[2017-05-10 13:32:33,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:32:33,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:32:33,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:32:33,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:32:33,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0561c5f5-6628-4ac5-abe5-ee9f1720a32a
[2017-05-10 13:32:33,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0561c5f5-6628-4ac5-abe5-ee9f1720a32a
[2017-05-10 13:32:33,587: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 52767f09-391d-44cd-b921-1d779563f679
[2017-05-10 13:32:33,587: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 52767f09-391d-44cd-b921-1d779563f679
[2017-05-10 13:32:33,587: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0561c5f5-6628-4ac5-abe5-ee9f1720a32a
[2017-05-10 13:32:33,587: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0561c5f5-6628-4ac5-abe5-ee9f1720a32a
[2017-05-10 13:32:33,591: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com import_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'pacemaker_ha_operation': False, 'device_type': 'linux', 'validate_importable': False}
[2017-05-10 13:32:33,629: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9f03c2c1-5f10-4c93-91b4-127f440bfed3
[2017-05-10 13:32:33,629: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9f03c2c1-5f10-4c93-91b4-127f440bfed3
[2017-05-10 13:32:33,736: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:32:33,736: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:32:33,736: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:32:33,736: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:32:33,737: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9f03c2c1-5f10-4c93-91b4-127f440bfed3
[2017-05-10 13:32:33,737: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9f03c2c1-5f10-4c93-91b4-127f440bfed3
[2017-05-10 13:32:33,753: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9f03c2c1-5f10-4c93-91b4-127f440bfed3
[2017-05-10 13:32:33,753: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9f03c2c1-5f10-4c93-91b4-127f440bfed3
[2017-05-10 13:32:33,754: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com check_block_device {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux'}
[2017-05-10 13:32:33,766: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6c3abfa6-c290-4dcf-81ba-aa3bd8b2bf1c
[2017-05-10 13:32:33,766: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6c3abfa6-c290-4dcf-81ba-aa3bd8b2bf1c
[2017-05-10 13:32:34,345: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:32:34,345: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:32:34,345: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:32:34,345: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:32:34,345: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3b0df835-199a-4d39-aa24-29c3dd889fe8
[2017-05-10 13:32:34,345: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3b0df835-199a-4d39-aa24-29c3dd889fe8
[2017-05-10 13:32:34,360: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3b0df835-199a-4d39-aa24-29c3dd889fe8
[2017-05-10 13:32:34,360: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3b0df835-199a-4d39-aa24-29c3dd889fe8
[2017-05-10 13:32:34,513: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:32:34,513: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:32:34,513: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:32:34,513: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:32:34,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a7a22c34-d8e7-49c2-a137-49f6255dc618
[2017-05-10 13:32:34,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a7a22c34-d8e7-49c2-a137-49f6255dc618
[2017-05-10 13:32:34,560: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a7a22c34-d8e7-49c2-a137-49f6255dc618
[2017-05-10 13:32:34,560: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a7a22c34-d8e7-49c2-a137-49f6255dc618
[2017-05-10 13:32:34,684: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:32:34,684: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:32:34,685: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:32:34,685: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:32:34,685: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 71885345-d137-4e20-bc6c-fb6e2ec70d62
[2017-05-10 13:32:34,685: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 71885345-d137-4e20-bc6c-fb6e2ec70d62
[2017-05-10 13:32:34,705: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 71885345-d137-4e20-bc6c-fb6e2ec70d62
[2017-05-10 13:32:34,705: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 71885345-d137-4e20-bc6c-fb6e2ec70d62
[2017-05-10 13:32:34,858: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:32:34,858: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:32:34,858: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:32:34,858: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:32:34,858: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6c3abfa6-c290-4dcf-81ba-aa3bd8b2bf1c
[2017-05-10 13:32:34,858: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6c3abfa6-c290-4dcf-81ba-aa3bd8b2bf1c
[2017-05-10 13:32:34,881: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6c3abfa6-c290-4dcf-81ba-aa3bd8b2bf1c
[2017-05-10 13:32:34,881: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6c3abfa6-c290-4dcf-81ba-aa3bd8b2bf1c
[2017-05-10 13:32:34,881: INFO/job] Job 54 PreFormatCheck: Filesystem found: type 'ext4'
[2017-05-10 13:32:34,881: ERROR/job_scheduler] Job 54 step 1 encountered an agent error: Filesystem found: type 'ext4'
[2017-05-10 13:32:34,993: INFO/job_scheduler] Job 54: complete_job: Updating cache
[2017-05-10 13:32:34,993: INFO/job_scheduler] Job 54: locked_item 140430197188176 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:32:34,997: INFO/job_scheduler] Job 54: locked_item 140430197617744 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-10 13:32:34,999: INFO/job_scheduler] Job 54: locked_item 140430197617936 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-10 13:32:35,003: INFO/job_scheduler] Job 54 completing (errored=True, cancelled=False)
[2017-05-10 13:32:35,069: INFO/job_scheduler] run_next: 0 runnable jobs of (13 pending, 3 tasked)
[2017-05-10 13:32:35,387: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:32:35,387: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:32:35,388: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:32:35,388: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:32:35,388: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a5148445-212f-4dcd-8038-03c3c302a0c9
[2017-05-10 13:32:35,388: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a5148445-212f-4dcd-8038-03c3c302a0c9
[2017-05-10 13:32:35,390: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a5148445-212f-4dcd-8038-03c3c302a0c9
[2017-05-10 13:32:35,390: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a5148445-212f-4dcd-8038-03c3c302a0c9
[2017-05-10 13:32:35,394: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:32:35,394: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:32:35,394: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:32:35,394: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:32:35,395: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e5e0d45b-f0e8-4da4-83d0-8ed409b1ba5d
[2017-05-10 13:32:35,395: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e5e0d45b-f0e8-4da4-83d0-8ed409b1ba5d
[2017-05-10 13:32:35,408: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e5e0d45b-f0e8-4da4-83d0-8ed409b1ba5d
[2017-05-10 13:32:35,408: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e5e0d45b-f0e8-4da4-83d0-8ed409b1ba5d
[2017-05-10 13:32:35,460: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:32:35,460: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:32:35,460: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:32:35,460: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:32:35,460: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8e9f8df1-7dde-4daf-ade4-5325a792239b
[2017-05-10 13:32:35,460: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8e9f8df1-7dde-4daf-ade4-5325a792239b
[2017-05-10 13:32:35,510: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8e9f8df1-7dde-4daf-ade4-5325a792239b
[2017-05-10 13:32:35,510: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8e9f8df1-7dde-4daf-ade4-5325a792239b
[2017-05-10 13:32:35,511: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com import_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1', 'pacemaker_ha_operation': False, 'device_type': 'zfs', 'validate_importable': False}
[2017-05-10 13:32:35,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7486a304-489c-4ce9-bdeb-434d11248b53
[2017-05-10 13:32:35,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7486a304-489c-4ce9-bdeb-434d11248b53
[2017-05-10 13:32:36,601: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:32:36,601: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:32:36,601: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:32:36,601: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:32:36,601: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 833b07ba-d081-4377-acbd-5794f20f8244
[2017-05-10 13:32:36,601: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 833b07ba-d081-4377-acbd-5794f20f8244
[2017-05-10 13:32:36,617: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 833b07ba-d081-4377-acbd-5794f20f8244
[2017-05-10 13:32:36,617: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 833b07ba-d081-4377-acbd-5794f20f8244
[2017-05-10 13:32:36,620: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com import_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5', 'pacemaker_ha_operation': False, 'device_type': 'zfs', 'validate_importable': False}
[2017-05-10 13:32:36,635: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b4dc6389-2bfe-49d9-8889-bc1d07efdb99
[2017-05-10 13:32:36,635: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b4dc6389-2bfe-49d9-8889-bc1d07efdb99
[2017-05-10 13:32:37,633: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:32:37,633: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:32:37,633: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:32:37,633: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:32:37,633: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7486a304-489c-4ce9-bdeb-434d11248b53
[2017-05-10 13:32:37,633: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7486a304-489c-4ce9-bdeb-434d11248b53
[2017-05-10 13:32:37,671: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7486a304-489c-4ce9-bdeb-434d11248b53
[2017-05-10 13:32:37,671: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7486a304-489c-4ce9-bdeb-434d11248b53
[2017-05-10 13:32:37,672: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com check_block_device {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1', 'device_type': 'zfs'}
[2017-05-10 13:32:37,685: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 49d70203-9fa9-487b-ab8f-e567703e2fe3
[2017-05-10 13:32:37,685: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 49d70203-9fa9-487b-ab8f-e567703e2fe3
[2017-05-10 13:32:37,774: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:32:37,774: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:32:37,774: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:32:37,774: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:32:37,774: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 49d70203-9fa9-487b-ab8f-e567703e2fe3
[2017-05-10 13:32:37,774: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 49d70203-9fa9-487b-ab8f-e567703e2fe3
[2017-05-10 13:32:37,803: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 49d70203-9fa9-487b-ab8f-e567703e2fe3
[2017-05-10 13:32:37,803: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 49d70203-9fa9-487b-ab8f-e567703e2fe3
[2017-05-10 13:32:37,805: INFO/job] testfs-OST0000 passed pre-format check, allowing subsequent reformats
[2017-05-10 13:32:37,856: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com format_target {'index': 0, 'backfstype': 'zfs', 'target_name': u'testfs-OST0000', 'mgsnode': ((u'10.14.81.224@tcp0',), (u'10.14.81.225@tcp0',)), 'target_types': 'ost', 'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1', 'device_type': 'zfs', 'failnode': [u'10.14.81.227@tcp0'], 'fsname': u'testfs'}
[2017-05-10 13:32:37,878: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc aa2b6b5e-3468-49b3-8a2e-175ed33e790d
[2017-05-10 13:32:37,878: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc aa2b6b5e-3468-49b3-8a2e-175ed33e790d
[2017-05-10 13:32:39,726: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:32:39,726: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:32:39,726: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:32:39,726: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:32:39,727: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b4dc6389-2bfe-49d9-8889-bc1d07efdb99
[2017-05-10 13:32:39,727: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b4dc6389-2bfe-49d9-8889-bc1d07efdb99
[2017-05-10 13:32:39,763: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b4dc6389-2bfe-49d9-8889-bc1d07efdb99
[2017-05-10 13:32:39,763: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b4dc6389-2bfe-49d9-8889-bc1d07efdb99
[2017-05-10 13:32:39,764: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com check_block_device {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5', 'device_type': 'zfs'}
[2017-05-10 13:32:39,790: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dfbc3c1c-4e7e-4e68-aed4-093dce5194d8
[2017-05-10 13:32:39,790: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dfbc3c1c-4e7e-4e68-aed4-093dce5194d8
[2017-05-10 13:32:40,524: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:32:40,524: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:32:40,524: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:32:40,524: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:32:40,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 41614756-17ef-4deb-a220-a97e2e5714ad
[2017-05-10 13:32:40,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 41614756-17ef-4deb-a220-a97e2e5714ad
[2017-05-10 13:32:40,561: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 41614756-17ef-4deb-a220-a97e2e5714ad
[2017-05-10 13:32:40,561: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 41614756-17ef-4deb-a220-a97e2e5714ad
[2017-05-10 13:32:40,562: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com import_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3', 'pacemaker_ha_operation': False, 'device_type': 'zfs', 'validate_importable': False}
[2017-05-10 13:32:40,578: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 62b291ef-1c90-4a7d-9024-10a860aeeafc
[2017-05-10 13:32:40,578: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 62b291ef-1c90-4a7d-9024-10a860aeeafc
[2017-05-10 13:32:40,911: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:32:40,911: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:32:40,911: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:32:40,911: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:32:40,912: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dfbc3c1c-4e7e-4e68-aed4-093dce5194d8
[2017-05-10 13:32:40,912: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dfbc3c1c-4e7e-4e68-aed4-093dce5194d8
[2017-05-10 13:32:40,963: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dfbc3c1c-4e7e-4e68-aed4-093dce5194d8
[2017-05-10 13:32:40,963: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dfbc3c1c-4e7e-4e68-aed4-093dce5194d8
[2017-05-10 13:32:40,964: INFO/job] MGS passed pre-format check, allowing subsequent reformats
[2017-05-10 13:32:40,999: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com format_target {'backfstype': 'zfs', 'target_name': u'MGS', 'target_types': 'mgs', 'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5', 'device_type': 'zfs', 'failnode': [u'10.14.81.225@tcp0']}
[2017-05-10 13:32:41,050: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1846dd06-9f68-481b-ab00-be7d0ce81cb1
[2017-05-10 13:32:41,050: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1846dd06-9f68-481b-ab00-be7d0ce81cb1
[2017-05-10 13:32:43,671: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:32:43,671: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:32:43,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:32:43,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:32:43,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 62b291ef-1c90-4a7d-9024-10a860aeeafc
[2017-05-10 13:32:43,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 62b291ef-1c90-4a7d-9024-10a860aeeafc
[2017-05-10 13:32:43,706: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 62b291ef-1c90-4a7d-9024-10a860aeeafc
[2017-05-10 13:32:43,706: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 62b291ef-1c90-4a7d-9024-10a860aeeafc
[2017-05-10 13:32:43,707: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com check_block_device {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3', 'device_type': 'zfs'}
[2017-05-10 13:32:43,728: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1495ba56-2f40-431f-beb0-cb4ff762df15
[2017-05-10 13:32:43,728: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1495ba56-2f40-431f-beb0-cb4ff762df15
[2017-05-10 13:32:44,810: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:32:44,810: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:32:44,810: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:32:44,810: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:32:44,811: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1495ba56-2f40-431f-beb0-cb4ff762df15
[2017-05-10 13:32:44,811: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1495ba56-2f40-431f-beb0-cb4ff762df15
[2017-05-10 13:32:44,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1495ba56-2f40-431f-beb0-cb4ff762df15
[2017-05-10 13:32:44,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1495ba56-2f40-431f-beb0-cb4ff762df15
[2017-05-10 13:32:44,861: INFO/job] testfs-OST0001 passed pre-format check, allowing subsequent reformats
[2017-05-10 13:32:44,916: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com format_target {'index': 1, 'backfstype': 'zfs', 'target_name': u'testfs-OST0001', 'mgsnode': ((u'10.14.81.224@tcp0',), (u'10.14.81.225@tcp0',)), 'target_types': 'ost', 'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3', 'device_type': 'zfs', 'failnode': [u'10.14.81.226@tcp0'], 'fsname': u'testfs'}
[2017-05-10 13:32:44,956: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8519bc57-5ce5-44b5-8e8e-8c241cec5d41
[2017-05-10 13:32:44,956: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8519bc57-5ce5-44b5-8e8e-8c241cec5d41
[2017-05-10 13:32:48,034: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:32:48,034: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:32:48,034: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:32:48,034: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:32:48,035: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc aa2b6b5e-3468-49b3-8a2e-175ed33e790d
[2017-05-10 13:32:48,035: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc aa2b6b5e-3468-49b3-8a2e-175ed33e790d
[2017-05-10 13:32:48,052: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc aa2b6b5e-3468-49b3-8a2e-175ed33e790d
[2017-05-10 13:32:48,052: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc aa2b6b5e-3468-49b3-8a2e-175ed33e790d
[2017-05-10 13:32:48,179: INFO/job] Updating mtm volume_nodes for target testfs-OST0000
[2017-05-10 13:32:50,584: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:32:50,584: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:32:50,584: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:32:50,584: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:32:50,585: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1846dd06-9f68-481b-ab00-be7d0ce81cb1
[2017-05-10 13:32:50,585: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1846dd06-9f68-481b-ab00-be7d0ce81cb1
[2017-05-10 13:32:50,624: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1846dd06-9f68-481b-ab00-be7d0ce81cb1
[2017-05-10 13:32:50,624: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1846dd06-9f68-481b-ab00-be7d0ce81cb1
[2017-05-10 13:32:50,652: INFO/job] Updating mtm volume_nodes for target MGS
[2017-05-10 13:32:52,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:32:52,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:32:52,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:32:52,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:32:52,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8519bc57-5ce5-44b5-8e8e-8c241cec5d41
[2017-05-10 13:32:52,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8519bc57-5ce5-44b5-8e8e-8c241cec5d41
[2017-05-10 13:32:52,126: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8519bc57-5ce5-44b5-8e8e-8c241cec5d41
[2017-05-10 13:32:52,126: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8519bc57-5ce5-44b5-8e8e-8c241cec5d41
[2017-05-10 13:32:52,162: INFO/job] Updating mtm volume_nodes for target testfs-OST0001
[2017-05-10 13:33:33,245: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:33:33,247: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 13:33:33,279: INFO/command_plan] add_jobs: created Job 70 (Trigger plugin poll for linux plugins)
[2017-05-10 13:33:33,315: INFO/job_scheduler] run_next: 1 runnable jobs of (14 pending, 3 tasked)
[2017-05-10 13:33:33,336: INFO/job_scheduler] Job 70: RunJobThread.run
[2017-05-10 13:33:33,360: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:33:33,395: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:33:33,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f1f78355-f61a-4a9c-9de4-4252d66f690b
[2017-05-10 13:33:33,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f1f78355-f61a-4a9c-9de4-4252d66f690b
[2017-05-10 13:33:33,403: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:33:33,437: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6bd15abf-036d-42b6-93f9-937491918fcf
[2017-05-10 13:33:33,437: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6bd15abf-036d-42b6-93f9-937491918fcf
[2017-05-10 13:33:33,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 68c48dba-f334-4391-b4dd-17aa9d7bc6ee
[2017-05-10 13:33:33,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 68c48dba-f334-4391-b4dd-17aa9d7bc6ee
[2017-05-10 13:33:33,498: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:33:33,498: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:33:33,498: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:33:33,498: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:33:33,498: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6bd15abf-036d-42b6-93f9-937491918fcf
[2017-05-10 13:33:33,498: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6bd15abf-036d-42b6-93f9-937491918fcf
[2017-05-10 13:33:33,501: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:33:33,501: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:33:33,501: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:33:33,501: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:33:33,501: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f1f78355-f61a-4a9c-9de4-4252d66f690b
[2017-05-10 13:33:33,501: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f1f78355-f61a-4a9c-9de4-4252d66f690b
[2017-05-10 13:33:33,527: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6bd15abf-036d-42b6-93f9-937491918fcf
[2017-05-10 13:33:33,527: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6bd15abf-036d-42b6-93f9-937491918fcf
[2017-05-10 13:33:33,527: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:33:33,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f1f78355-f61a-4a9c-9de4-4252d66f690b
[2017-05-10 13:33:33,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f1f78355-f61a-4a9c-9de4-4252d66f690b
[2017-05-10 13:33:33,528: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:33:33,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:33:33,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:33:33,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:33:33,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:33:33,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 68c48dba-f334-4391-b4dd-17aa9d7bc6ee
[2017-05-10 13:33:33,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 68c48dba-f334-4391-b4dd-17aa9d7bc6ee
[2017-05-10 13:33:33,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 68c48dba-f334-4391-b4dd-17aa9d7bc6ee
[2017-05-10 13:33:33,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 68c48dba-f334-4391-b4dd-17aa9d7bc6ee
[2017-05-10 13:33:33,549: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:33:33,549: INFO/job_scheduler] Job 70 finished 1 steps successfully
[2017-05-10 13:33:33,598: INFO/job_scheduler] Job 70: complete_job: Updating cache
[2017-05-10 13:33:33,598: INFO/job_scheduler] Job 70 completing (errored=False, cancelled=False)
[2017-05-10 13:33:33,660: INFO/job_scheduler] run_next: 0 runnable jobs of (13 pending, 3 tasked)
[2017-05-10 13:33:34,963: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:33:34,963: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 13:33:34,988: INFO/command_plan] add_jobs: created Job 71 (Trigger plugin poll for linux plugins)
[2017-05-10 13:33:35,034: INFO/job_scheduler] run_next: 1 runnable jobs of (14 pending, 3 tasked)
[2017-05-10 13:33:35,057: INFO/job_scheduler] Job 71: RunJobThread.run
[2017-05-10 13:33:35,075: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:33:35,122: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:33:35,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c85f6dc2-1686-4c1f-bb6d-d60302520a2e
[2017-05-10 13:33:35,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c85f6dc2-1686-4c1f-bb6d-d60302520a2e
[2017-05-10 13:33:35,152: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:33:35,164: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 00121ed7-f12f-4ae7-a209-8e2d2e020ee3
[2017-05-10 13:33:35,164: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 00121ed7-f12f-4ae7-a209-8e2d2e020ee3
[2017-05-10 13:33:35,205: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc addb3bd9-fc68-413c-9697-7ff38a2620eb
[2017-05-10 13:33:35,205: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc addb3bd9-fc68-413c-9697-7ff38a2620eb
[2017-05-10 13:33:35,221: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:33:35,221: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:33:35,221: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:33:35,221: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:33:35,221: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c85f6dc2-1686-4c1f-bb6d-d60302520a2e
[2017-05-10 13:33:35,221: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c85f6dc2-1686-4c1f-bb6d-d60302520a2e
[2017-05-10 13:33:35,260: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c85f6dc2-1686-4c1f-bb6d-d60302520a2e
[2017-05-10 13:33:35,260: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c85f6dc2-1686-4c1f-bb6d-d60302520a2e
[2017-05-10 13:33:35,260: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:33:35,263: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:33:35,263: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:33:35,264: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:33:35,264: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:33:35,264: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 00121ed7-f12f-4ae7-a209-8e2d2e020ee3
[2017-05-10 13:33:35,264: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 00121ed7-f12f-4ae7-a209-8e2d2e020ee3
[2017-05-10 13:33:35,300: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 00121ed7-f12f-4ae7-a209-8e2d2e020ee3
[2017-05-10 13:33:35,300: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 00121ed7-f12f-4ae7-a209-8e2d2e020ee3
[2017-05-10 13:33:35,301: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:33:35,304: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:33:35,304: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:33:35,304: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:33:35,304: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:33:35,304: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc addb3bd9-fc68-413c-9697-7ff38a2620eb
[2017-05-10 13:33:35,304: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc addb3bd9-fc68-413c-9697-7ff38a2620eb
[2017-05-10 13:33:35,336: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc addb3bd9-fc68-413c-9697-7ff38a2620eb
[2017-05-10 13:33:35,336: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc addb3bd9-fc68-413c-9697-7ff38a2620eb
[2017-05-10 13:33:35,337: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:33:35,340: INFO/job_scheduler] Job 71 finished 1 steps successfully
[2017-05-10 13:33:35,397: INFO/job_scheduler] Job 71: complete_job: Updating cache
[2017-05-10 13:33:35,397: INFO/job_scheduler] Job 71 completing (errored=False, cancelled=False)
[2017-05-10 13:33:35,479: INFO/job_scheduler] run_next: 0 runnable jobs of (13 pending, 3 tasked)
[2017-05-10 13:33:41,308: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:33:41,308: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 13:33:41,336: INFO/command_plan] add_jobs: created Job 72 (Trigger plugin poll for linux plugins)
[2017-05-10 13:33:41,378: INFO/job_scheduler] run_next: 1 runnable jobs of (14 pending, 3 tasked)
[2017-05-10 13:33:41,394: INFO/job_scheduler] Job 72: RunJobThread.run
[2017-05-10 13:33:41,400: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:33:41,436: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:33:41,436: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 93196ecb-ec8d-490e-8be9-084e03b70262
[2017-05-10 13:33:41,436: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 93196ecb-ec8d-490e-8be9-084e03b70262
[2017-05-10 13:33:41,448: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:33:41,473: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2dd51b03-3033-4e0c-b4ad-af1fbee2c576
[2017-05-10 13:33:41,473: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2dd51b03-3033-4e0c-b4ad-af1fbee2c576
[2017-05-10 13:33:41,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f7eaf451-872f-4471-bfa3-1a7265900c25
[2017-05-10 13:33:41,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f7eaf451-872f-4471-bfa3-1a7265900c25
[2017-05-10 13:33:41,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:33:41,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:33:41,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:33:41,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:33:41,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 93196ecb-ec8d-490e-8be9-084e03b70262
[2017-05-10 13:33:41,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 93196ecb-ec8d-490e-8be9-084e03b70262
[2017-05-10 13:33:41,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:33:41,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:33:41,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:33:41,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:33:41,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2dd51b03-3033-4e0c-b4ad-af1fbee2c576
[2017-05-10 13:33:41,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2dd51b03-3033-4e0c-b4ad-af1fbee2c576
[2017-05-10 13:33:41,569: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:33:41,569: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:33:41,569: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:33:41,569: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:33:41,569: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f7eaf451-872f-4471-bfa3-1a7265900c25
[2017-05-10 13:33:41,569: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f7eaf451-872f-4471-bfa3-1a7265900c25
[2017-05-10 13:33:41,570: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 93196ecb-ec8d-490e-8be9-084e03b70262
[2017-05-10 13:33:41,570: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 93196ecb-ec8d-490e-8be9-084e03b70262
[2017-05-10 13:33:41,570: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:33:41,575: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f7eaf451-872f-4471-bfa3-1a7265900c25
[2017-05-10 13:33:41,575: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f7eaf451-872f-4471-bfa3-1a7265900c25
[2017-05-10 13:33:41,575: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:33:41,603: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2dd51b03-3033-4e0c-b4ad-af1fbee2c576
[2017-05-10 13:33:41,603: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2dd51b03-3033-4e0c-b4ad-af1fbee2c576
[2017-05-10 13:33:41,604: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:33:41,604: INFO/job_scheduler] Job 72 finished 1 steps successfully
[2017-05-10 13:33:41,720: INFO/job_scheduler] Job 72: complete_job: Updating cache
[2017-05-10 13:33:41,720: INFO/job_scheduler] Job 72 completing (errored=False, cancelled=False)
[2017-05-10 13:33:41,761: INFO/job_scheduler] run_next: 0 runnable jobs of (13 pending, 3 tasked)
[2017-05-10 13:33:42,004: INFO/job_scheduler] Job 56 finished 5 steps successfully
[2017-05-10 13:33:42,045: INFO/job] StatefulObject.set_state MGS unformatted->formatted (intentional=True) 140430197213776
[2017-05-10 13:33:42,052: INFO/job] Job 56: StateChangeJob complete, setting state formatted on MGS
[2017-05-10 13:33:42,052: INFO/job] Job 56: not_deleted=True
[2017-05-10 13:33:42,065: INFO/job_scheduler] Job 56: complete_job: Updating cache
[2017-05-10 13:33:42,065: INFO/job_scheduler] Job 56: locked_item 140430197213776 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:33:42,075: INFO/job_scheduler] Job 56: locked_item 140430197674640 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-10 13:33:42,076: INFO/job_scheduler] Job 56: locked_item 140430197674832 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-10 13:33:42,077: INFO/job_scheduler] Job 56 completing (errored=False, cancelled=False)
[2017-05-10 13:33:42,154: INFO/job_scheduler] run_next: 1 runnable jobs of (13 pending, 2 tasked)
[2017-05-10 13:33:42,221: INFO/job_scheduler] Job 57: RunJobThread.run
[2017-05-10 13:33:42,222: INFO/job_scheduler] Job 57 finished 1 steps successfully
[2017-05-10 13:33:42,430: INFO/job] StatefulObject.set_state MGS formatted->registered (intentional=True) 140430197213776
[2017-05-10 13:33:42,435: INFO/job] Job 57: StateChangeJob complete, setting state registered on MGS
[2017-05-10 13:33:42,435: INFO/job] Job 57: not_deleted=True
[2017-05-10 13:33:42,436: INFO/job_scheduler] Job 57: complete_job: Updating cache
[2017-05-10 13:33:42,436: INFO/job_scheduler] Job 57: locked_item 140430197213776 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:33:42,449: INFO/job_scheduler] Job 57 completing (errored=False, cancelled=False)
[2017-05-10 13:33:42,524: INFO/job_scheduler] run_next: 1 runnable jobs of (12 pending, 2 tasked)
[2017-05-10 13:33:42,660: INFO/job_scheduler] Job 58: RunJobThread.run
[2017-05-10 13:33:42,660: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-05-10 13:33:42,690: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9cbf7acb-2410-4e39-b7bb-a8045bd6198f
[2017-05-10 13:33:42,690: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9cbf7acb-2410-4e39-b7bb-a8045bd6198f
[2017-05-10 13:33:43,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:33:43,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:33:43,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:33:43,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:33:43,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9cbf7acb-2410-4e39-b7bb-a8045bd6198f
[2017-05-10 13:33:43,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9cbf7acb-2410-4e39-b7bb-a8045bd6198f
[2017-05-10 13:33:43,810: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9cbf7acb-2410-4e39-b7bb-a8045bd6198f
[2017-05-10 13:33:43,810: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9cbf7acb-2410-4e39-b7bb-a8045bd6198f
[2017-05-10 13:33:43,810: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-05-10 13:33:43,833: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d87a856f-d6d5-4f9a-8888-28eb6d24053f
[2017-05-10 13:33:43,833: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d87a856f-d6d5-4f9a-8888-28eb6d24053f
[2017-05-10 13:33:44,376: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:33:44,376: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 13:33:44,400: INFO/command_plan] add_jobs: created Job 73 (Trigger plugin poll for linux plugins)
[2017-05-10 13:33:44,431: INFO/job_scheduler] run_next: 1 runnable jobs of (12 pending, 3 tasked)
[2017-05-10 13:33:44,454: INFO/job_scheduler] Job 73: RunJobThread.run
[2017-05-10 13:33:44,468: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:33:44,502: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:33:44,508: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d6bb2946-19e1-4ce1-b6cd-540d231a6f61
[2017-05-10 13:33:44,508: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d6bb2946-19e1-4ce1-b6cd-540d231a6f61
[2017-05-10 13:33:44,566: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:33:44,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 58e9ff21-690e-4911-a65a-70d361161c6e
[2017-05-10 13:33:44,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 58e9ff21-690e-4911-a65a-70d361161c6e
[2017-05-10 13:33:44,621: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e7fb1dfd-4192-4ab6-b22f-7131133215ca
[2017-05-10 13:33:44,621: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e7fb1dfd-4192-4ab6-b22f-7131133215ca
[2017-05-10 13:33:44,621: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:33:44,621: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:33:44,622: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:33:44,622: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:33:44,622: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d6bb2946-19e1-4ce1-b6cd-540d231a6f61
[2017-05-10 13:33:44,622: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d6bb2946-19e1-4ce1-b6cd-540d231a6f61
[2017-05-10 13:33:44,643: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d6bb2946-19e1-4ce1-b6cd-540d231a6f61
[2017-05-10 13:33:44,643: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d6bb2946-19e1-4ce1-b6cd-540d231a6f61
[2017-05-10 13:33:44,644: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:33:44,646: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:33:44,646: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:33:44,647: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:33:44,647: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:33:44,647: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 58e9ff21-690e-4911-a65a-70d361161c6e
[2017-05-10 13:33:44,647: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 58e9ff21-690e-4911-a65a-70d361161c6e
[2017-05-10 13:33:44,658: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 58e9ff21-690e-4911-a65a-70d361161c6e
[2017-05-10 13:33:44,658: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 58e9ff21-690e-4911-a65a-70d361161c6e
[2017-05-10 13:33:44,658: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:33:44,732: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:33:44,732: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:33:44,732: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:33:44,732: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:33:44,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e7fb1dfd-4192-4ab6-b22f-7131133215ca
[2017-05-10 13:33:44,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e7fb1dfd-4192-4ab6-b22f-7131133215ca
[2017-05-10 13:33:44,743: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e7fb1dfd-4192-4ab6-b22f-7131133215ca
[2017-05-10 13:33:44,743: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e7fb1dfd-4192-4ab6-b22f-7131133215ca
[2017-05-10 13:33:44,743: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:33:44,744: INFO/job_scheduler] Job 73 finished 1 steps successfully
[2017-05-10 13:33:44,800: INFO/job_scheduler] Job 73: complete_job: Updating cache
[2017-05-10 13:33:44,801: INFO/job_scheduler] Job 73 completing (errored=False, cancelled=False)
[2017-05-10 13:33:44,859: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 3 tasked)
[2017-05-10 13:33:45,020: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:33:45,020: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:33:45,023: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:33:45,023: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:33:45,023: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d87a856f-d6d5-4f9a-8888-28eb6d24053f
[2017-05-10 13:33:45,023: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d87a856f-d6d5-4f9a-8888-28eb6d24053f
[2017-05-10 13:33:45,071: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d87a856f-d6d5-4f9a-8888-28eb6d24053f
[2017-05-10 13:33:45,071: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d87a856f-d6d5-4f9a-8888-28eb6d24053f
[2017-05-10 13:33:45,073: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_target_store {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/MGS', 'device_type': 'zfs', 'mount_point': u'/mnt/MGS', 'uuid': u'8316997203038885076', 'backfstype': 'zfs'}
[2017-05-10 13:33:45,191: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b4792383-9556-496f-8b49-31000653b602
[2017-05-10 13:33:45,191: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b4792383-9556-496f-8b49-31000653b602
[2017-05-10 13:33:45,260: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:33:45,260: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:33:45,261: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:33:45,261: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:33:45,261: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b4792383-9556-496f-8b49-31000653b602
[2017-05-10 13:33:45,261: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b4792383-9556-496f-8b49-31000653b602
[2017-05-10 13:33:45,308: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b4792383-9556-496f-8b49-31000653b602
[2017-05-10 13:33:45,308: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b4792383-9556-496f-8b49-31000653b602
[2017-05-10 13:33:45,310: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_target_store {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/MGS', 'device_type': 'zfs', 'mount_point': u'/mnt/MGS', 'uuid': u'8316997203038885076', 'backfstype': 'zfs'}
[2017-05-10 13:33:45,345: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 97681a4d-26f1-4319-8040-948f090bbb7b
[2017-05-10 13:33:45,345: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 97681a4d-26f1-4319-8040-948f090bbb7b
[2017-05-10 13:33:45,504: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:33:45,504: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:33:45,505: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:33:45,505: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:33:45,505: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 97681a4d-26f1-4319-8040-948f090bbb7b
[2017-05-10 13:33:45,505: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 97681a4d-26f1-4319-8040-948f090bbb7b
[2017-05-10 13:33:45,524: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 97681a4d-26f1-4319-8040-948f090bbb7b
[2017-05-10 13:33:45,524: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 97681a4d-26f1-4319-8040-948f090bbb7b
[2017-05-10 13:33:45,526: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/MGS', 'mount_point': u'/mnt/MGS', 'uuid': u'8316997203038885076', 'ha_label': u'MGS_99a7c7', 'primary': True}
[2017-05-10 13:33:45,542: INFO/job_scheduler] Job 55 finished 5 steps successfully
[2017-05-10 13:33:45,546: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 877525e6-3547-4001-a155-5ed5777d788b
[2017-05-10 13:33:45,546: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 877525e6-3547-4001-a155-5ed5777d788b
[2017-05-10 13:33:45,567: INFO/job_scheduler] Job 53 finished 5 steps successfully
[2017-05-10 13:33:45,631: INFO/job] StatefulObject.set_state testfs-OST0001 unformatted->formatted (intentional=True) 140430465988624
[2017-05-10 13:33:45,642: INFO/job] Job 55: StateChangeJob complete, setting state formatted on testfs-OST0001
[2017-05-10 13:33:45,643: INFO/job] Job 55: not_deleted=True
[2017-05-10 13:33:45,651: INFO/job_scheduler] Job 55: complete_job: Updating cache
[2017-05-10 13:33:45,651: INFO/job_scheduler] Job 55: locked_item 140430465988624 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:33:45,658: INFO/job_scheduler] Job 55: locked_item 140430197619600 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-10 13:33:45,662: INFO/job_scheduler] Job 55: locked_item 140430197619792 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-10 13:33:45,667: INFO/job_scheduler] Job 55 completing (errored=False, cancelled=False)
[2017-05-10 13:33:45,767: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 2 tasked)
[2017-05-10 13:33:45,983: INFO/job] StatefulObject.set_state testfs-OST0000 unformatted->formatted (intentional=True) 140430600663248
[2017-05-10 13:33:46,001: INFO/job] Job 53: StateChangeJob complete, setting state formatted on testfs-OST0000
[2017-05-10 13:33:46,001: INFO/job] Job 53: not_deleted=True
[2017-05-10 13:33:46,035: INFO/job_scheduler] Job 53: complete_job: Updating cache
[2017-05-10 13:33:46,035: INFO/job_scheduler] Job 53: locked_item 140430600663248 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:33:46,044: INFO/job_scheduler] Job 53: locked_item 140430197714000 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-10 13:33:46,050: INFO/job_scheduler] Job 53: locked_item 140430197714192 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-10 13:33:46,054: INFO/job_scheduler] Job 53 completing (errored=False, cancelled=False)
[2017-05-10 13:33:46,168: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 1 tasked)
[2017-05-10 13:33:48,651: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:33:48,651: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:33:48,651: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:33:48,651: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:33:48,652: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 877525e6-3547-4001-a155-5ed5777d788b
[2017-05-10 13:33:48,652: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 877525e6-3547-4001-a155-5ed5777d788b
[2017-05-10 13:33:48,662: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 877525e6-3547-4001-a155-5ed5777d788b
[2017-05-10 13:33:48,662: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 877525e6-3547-4001-a155-5ed5777d788b
[2017-05-10 13:33:48,665: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/MGS', 'mount_point': u'/mnt/MGS', 'uuid': u'8316997203038885076', 'ha_label': u'MGS_99a7c7', 'primary': False}
[2017-05-10 13:33:48,679: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 13dd53c8-5be0-49da-bb57-7934ab878d5d
[2017-05-10 13:33:48,679: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 13dd53c8-5be0-49da-bb57-7934ab878d5d
[2017-05-10 13:33:49,795: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:33:49,795: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:33:49,796: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:33:49,796: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:33:49,796: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 13dd53c8-5be0-49da-bb57-7934ab878d5d
[2017-05-10 13:33:49,796: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 13dd53c8-5be0-49da-bb57-7934ab878d5d
[2017-05-10 13:33:49,797: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 13dd53c8-5be0-49da-bb57-7934ab878d5d
[2017-05-10 13:33:49,797: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 13dd53c8-5be0-49da-bb57-7934ab878d5d
[2017-05-10 13:33:49,798: INFO/job_scheduler] Job 58 finished 6 steps successfully
[2017-05-10 13:33:49,824: INFO/job] StatefulObject.set_state MGS registered->unmounted (intentional=True) 140430197213776
[2017-05-10 13:33:49,831: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on MGS at severity 30
[2017-05-10 13:33:49,838: INFO/job] Job 58: StateChangeJob complete, setting state unmounted on MGS
[2017-05-10 13:33:49,841: INFO/job] Job 58: not_deleted=True
[2017-05-10 13:33:49,843: INFO/job_scheduler] Job 58: complete_job: Updating cache
[2017-05-10 13:33:49,844: INFO/job_scheduler] Job 58: locked_item 140430197213776 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:33:49,862: INFO/job_scheduler] Job 58 completing (errored=False, cancelled=False)
[2017-05-10 13:33:49,908: INFO/job_scheduler] run_next: 1 runnable jobs of (11 pending, 0 tasked)
[2017-05-10 13:33:50,035: INFO/job_scheduler] Job 59: RunJobThread.run
[2017-05-10 13:33:50,037: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/MGS', 'device_type': 'zfs'}
[2017-05-10 13:33:50,046: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/MGS', 'device_type': 'zfs'}
[2017-05-10 13:33:50,048: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/MGS', 'device_type': 'zfs'}
[2017-05-10 13:33:50,063: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 410279eb-9588-4e81-9501-e43bbbc4802f
[2017-05-10 13:33:50,063: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 410279eb-9588-4e81-9501-e43bbbc4802f
[2017-05-10 13:33:50,076: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 938f23cb-f8f7-4cd8-90ce-a0e138db3d00
[2017-05-10 13:33:50,076: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 938f23cb-f8f7-4cd8-90ce-a0e138db3d00
[2017-05-10 13:33:50,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7de73136-e807-48de-8253-aa21964f42b2
[2017-05-10 13:33:50,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7de73136-e807-48de-8253-aa21964f42b2
[2017-05-10 13:33:52,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:33:52,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:33:52,199: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:33:52,199: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:33:52,199: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 410279eb-9588-4e81-9501-e43bbbc4802f
[2017-05-10 13:33:52,199: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 410279eb-9588-4e81-9501-e43bbbc4802f
[2017-05-10 13:33:52,201: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:33:52,201: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:33:52,202: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:33:52,202: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:33:52,202: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 938f23cb-f8f7-4cd8-90ce-a0e138db3d00
[2017-05-10 13:33:52,202: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 938f23cb-f8f7-4cd8-90ce-a0e138db3d00
[2017-05-10 13:33:52,232: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 410279eb-9588-4e81-9501-e43bbbc4802f
[2017-05-10 13:33:52,232: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 410279eb-9588-4e81-9501-e43bbbc4802f
[2017-05-10 13:33:52,244: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 938f23cb-f8f7-4cd8-90ce-a0e138db3d00
[2017-05-10 13:33:52,244: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 938f23cb-f8f7-4cd8-90ce-a0e138db3d00
[2017-05-10 13:33:54,216: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:33:54,216: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:33:54,217: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:33:54,217: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:33:54,217: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7de73136-e807-48de-8253-aa21964f42b2
[2017-05-10 13:33:54,217: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7de73136-e807-48de-8253-aa21964f42b2
[2017-05-10 13:33:54,263: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7de73136-e807-48de-8253-aa21964f42b2
[2017-05-10 13:33:54,263: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7de73136-e807-48de-8253-aa21964f42b2
[2017-05-10 13:33:54,264: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com import_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/MGS', 'pacemaker_ha_operation': False, 'device_type': 'zfs', 'validate_importable': True}
[2017-05-10 13:33:54,279: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2acdc029-4b8e-42e7-b607-1bbdb4fd63d9
[2017-05-10 13:33:54,279: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2acdc029-4b8e-42e7-b607-1bbdb4fd63d9
[2017-05-10 13:33:56,528: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 1 tasked)
[2017-05-10 13:34:00,513: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:34:00,513: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:34:00,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:34:00,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:34:00,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2acdc029-4b8e-42e7-b607-1bbdb4fd63d9
[2017-05-10 13:34:00,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2acdc029-4b8e-42e7-b607-1bbdb4fd63d9
[2017-05-10 13:34:00,557: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2acdc029-4b8e-42e7-b607-1bbdb4fd63d9
[2017-05-10 13:34:00,557: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2acdc029-4b8e-42e7-b607-1bbdb4fd63d9
[2017-05-10 13:34:00,557: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_target {'ha_label': u'MGS_99a7c7'}
[2017-05-10 13:34:00,570: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e8ba8f4c-d66d-4b6d-9fd4-5152ef750c22
[2017-05-10 13:34:00,570: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e8ba8f4c-d66d-4b6d-9fd4-5152ef750c22
[2017-05-10 13:34:14,745: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:34:14,745: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:34:14,745: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:34:14,745: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:34:14,745: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e8ba8f4c-d66d-4b6d-9fd4-5152ef750c22
[2017-05-10 13:34:14,745: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e8ba8f4c-d66d-4b6d-9fd4-5152ef750c22
[2017-05-10 13:34:14,763: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e8ba8f4c-d66d-4b6d-9fd4-5152ef750c22
[2017-05-10 13:34:14,763: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e8ba8f4c-d66d-4b6d-9fd4-5152ef750c22
[2017-05-10 13:34:14,764: INFO/job_scheduler] Job 59 finished 1 steps successfully
[2017-05-10 13:34:14,931: INFO/job] StatefulObject.set_state MGS unmounted->mounted (intentional=True) 140430197213776
[2017-05-10 13:34:14,962: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 20
[2017-05-10 13:34:14,969: INFO/job] Job 59: StateChangeJob complete, setting state mounted on MGS
[2017-05-10 13:34:14,969: INFO/job] Job 59: not_deleted=True
[2017-05-10 13:34:14,969: INFO/job_scheduler] Job 59: complete_job: Updating cache
[2017-05-10 13:34:14,969: INFO/job_scheduler] Job 59: locked_item 140430197213776 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:34:14,972: INFO/job_scheduler] Job 59 completing (errored=False, cancelled=False)
[2017-05-10 13:34:15,000: INFO/job_scheduler] run_next: 1 runnable jobs of (10 pending, 0 tasked)
[2017-05-10 13:34:15,033: WARNING/job] DependOn not satisfied: testfs-MDT0000 in state unformatted, not one of ['formatted'] (preferred formatted)
[2017-05-10 13:34:15,062: WARNING/job] DependOn not satisfied: testfs-MDT0000 in state unformatted, not one of ['formatted'] (preferred formatted)
[2017-05-10 13:34:15,062: WARNING/job_scheduler] Job 60: cancelling because of failed dependency
[2017-05-10 13:34:15,062: INFO/job_scheduler] Job 60 completing (errored=False, cancelled=True)
[2017-05-10 13:34:15,114: INFO/job_scheduler] run_next: 1 runnable jobs of (9 pending, 0 tasked)
[2017-05-10 13:34:15,149: WARNING/job] DependOn not satisfied: testfs-MDT0000 in state unformatted, not one of ['registered'] (preferred registered)
[2017-05-10 13:34:15,189: WARNING/job] DependOn not satisfied: testfs-MDT0000 in state unformatted, not one of ['registered'] (preferred registered)
[2017-05-10 13:34:15,189: WARNING/job_scheduler] Job 61: cancelling because of failed dependency
[2017-05-10 13:34:15,190: INFO/job_scheduler] Job 61 completing (errored=False, cancelled=True)
[2017-05-10 13:34:15,223: INFO/job_scheduler] run_next: 1 runnable jobs of (8 pending, 0 tasked)
[2017-05-10 13:34:15,237: WARNING/job] DependOn not satisfied: testfs-MDT0000 in state unformatted, not one of ['unmounted'] (preferred unmounted)
[2017-05-10 13:34:15,249: WARNING/job] DependOn not satisfied: testfs-MDT0000 in state unformatted, not one of ['unmounted'] (preferred unmounted)
[2017-05-10 13:34:15,249: WARNING/job_scheduler] Job 62: cancelling because of failed dependency
[2017-05-10 13:34:15,249: INFO/job_scheduler] Job 62 completing (errored=False, cancelled=True)
[2017-05-10 13:34:15,265: INFO/job_scheduler] run_next: 2 runnable jobs of (7 pending, 0 tasked)
[2017-05-10 13:34:15,271: WARNING/job] DependOn not satisfied: testfs-MDT0000 in state unformatted, not one of ['mounted'] (preferred mounted)
[2017-05-10 13:34:15,275: WARNING/job] DependOn not satisfied: testfs-MDT0000 in state unformatted, not one of ['mounted'] (preferred mounted)
[2017-05-10 13:34:15,296: WARNING/job_scheduler] Job 64: cancelling because of failed dependency
[2017-05-10 13:34:15,304: WARNING/job] DependOn not satisfied: testfs-MDT0000 in state unformatted, not one of ['mounted'] (preferred mounted)
[2017-05-10 13:34:15,310: WARNING/job] DependOn not satisfied: testfs-MDT0000 in state unformatted, not one of ['mounted'] (preferred mounted)
[2017-05-10 13:34:15,325: WARNING/job_scheduler] Job 63: cancelling because of failed dependency
[2017-05-10 13:34:15,325: INFO/job_scheduler] Job 64 completing (errored=False, cancelled=True)
[2017-05-10 13:34:15,370: INFO/job_scheduler] Job 63 completing (errored=False, cancelled=True)
[2017-05-10 13:34:15,426: INFO/job_scheduler] run_next: 2 runnable jobs of (5 pending, 0 tasked)
[2017-05-10 13:34:15,477: WARNING/job] DependOn not satisfied: testfs-OST0000 in state formatted, not one of ['registered'] (preferred registered)
[2017-05-10 13:34:15,509: WARNING/job] DependOn not satisfied: testfs-OST0000 in state formatted, not one of ['registered'] (preferred registered)
[2017-05-10 13:34:15,509: WARNING/job_scheduler] Job 65: cancelling because of failed dependency
[2017-05-10 13:34:15,541: WARNING/job] DependOn not satisfied: testfs-OST0001 in state formatted, not one of ['registered'] (preferred registered)
[2017-05-10 13:34:15,565: WARNING/job] DependOn not satisfied: testfs-OST0001 in state formatted, not one of ['registered'] (preferred registered)
[2017-05-10 13:34:15,565: WARNING/job_scheduler] Job 66: cancelling because of failed dependency
[2017-05-10 13:34:15,565: INFO/job_scheduler] Job 65 completing (errored=False, cancelled=True)
[2017-05-10 13:34:15,608: INFO/job_scheduler] Job 66 completing (errored=False, cancelled=True)
[2017-05-10 13:34:15,669: INFO/job_scheduler] run_next: 2 runnable jobs of (3 pending, 0 tasked)
[2017-05-10 13:34:15,698: WARNING/job] DependOn not satisfied: testfs-OST0000 in state formatted, not one of ['unmounted'] (preferred unmounted)
[2017-05-10 13:34:15,744: WARNING/job] DependOn not satisfied: testfs-OST0000 in state formatted, not one of ['unmounted'] (preferred unmounted)
[2017-05-10 13:34:15,744: WARNING/job_scheduler] Job 67: cancelling because of failed dependency
[2017-05-10 13:34:15,761: WARNING/job] DependOn not satisfied: testfs-OST0001 in state formatted, not one of ['unmounted'] (preferred unmounted)
[2017-05-10 13:34:15,786: WARNING/job] DependOn not satisfied: testfs-OST0001 in state formatted, not one of ['unmounted'] (preferred unmounted)
[2017-05-10 13:34:15,786: WARNING/job_scheduler] Job 68: cancelling because of failed dependency
[2017-05-10 13:34:15,786: INFO/job_scheduler] Job 67 completing (errored=False, cancelled=True)
[2017-05-10 13:34:15,820: INFO/job_scheduler] Job 68 completing (errored=False, cancelled=True)
[2017-05-10 13:34:15,854: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 13:34:15,904: WARNING/job] DependOn not satisfied: testfs-MDT0000 in state unformatted, not one of ['mounted'] (preferred mounted)
[2017-05-10 13:34:15,951: WARNING/job] DependOn not satisfied: testfs-MDT0000 in state unformatted, not one of ['mounted'] (preferred mounted)
[2017-05-10 13:34:15,963: WARNING/job] DependOn not satisfied: testfs-OST0000 in state formatted, not one of ['mounted'] (preferred mounted)
[2017-05-10 13:34:15,967: WARNING/job] DependOn not satisfied: testfs-OST0001 in state formatted, not one of ['mounted'] (preferred mounted)
[2017-05-10 13:34:15,979: WARNING/job_scheduler] Job 69: cancelling because of failed dependency
[2017-05-10 13:34:15,979: INFO/job_scheduler] Job 69 completing (errored=False, cancelled=True)
[2017-05-10 13:34:16,086: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:34:16,632: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:34:21,432: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:35:00,219: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:35:00,266: INFO/command_plan] set_state: <class 'chroma_core.models.filesystem.ManagedFilesystem'>-1 transitioning from unavailable to removed
[2017-05-10 13:35:00,347: INFO/command_plan] Created command 18 (Remove file system testfs from configuration) with 5 jobs
[2017-05-10 13:35:00,397: INFO/job_scheduler] run_next: 1 runnable jobs of (5 pending, 0 tasked)
[2017-05-10 13:35:00,509: INFO/job_scheduler] Job 74: open DB connection during completion
[2017-05-10 13:35:00,519: INFO/job] StatefulObject.set_state testfs unavailable->stopped (intentional=True) 140430197262992
[2017-05-10 13:35:00,523: INFO/job] Job 74: StateChangeJob complete, setting state stopped on testfs
[2017-05-10 13:35:00,523: INFO/job] Job 74: not_deleted=True
[2017-05-10 13:35:00,524: INFO/job_scheduler] Job 74: complete_job: Updating cache
[2017-05-10 13:35:00,524: INFO/job_scheduler] Job 74: locked_item 140430197262992 <class 'chroma_core.models.filesystem.ManagedFilesystem'> False True
[2017-05-10 13:35:00,530: INFO/job_scheduler] Job 74 completing (errored=False, cancelled=False)
[2017-05-10 13:35:00,544: INFO/job_scheduler] run_next: 3 runnable jobs of (4 pending, 0 tasked)
[2017-05-10 13:35:00,551: INFO/job_scheduler] Job 75: open DB connection during completion
[2017-05-10 13:35:00,560: INFO/job_scheduler] Job 76: open DB connection during completion
[2017-05-10 13:35:00,560: INFO/job_scheduler] Job 77: open DB connection during completion
[2017-05-10 13:35:00,570: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.target.ManagedTarget'> 3
[2017-05-10 13:35:00,570: INFO/job] StatefulObject.set_state testfs-OST0000 formatted->removed (intentional=True) 140430197807120
[2017-05-10 13:35:00,591: INFO/job] Job 75: StateChangeJob complete, setting state removed on testfs-OST0000
[2017-05-10 13:35:00,591: INFO/job] Job 75: not_deleted=None
[2017-05-10 13:35:00,591: INFO/job_scheduler] Job 75: complete_job: Updating cache
[2017-05-10 13:35:00,591: INFO/job_scheduler] Job 75: locked_item 140430197807120 <class 'chroma_core.models.target.ManagedTarget'> False None
[2017-05-10 13:35:00,591: INFO/job_scheduler] Job 75 completing (errored=False, cancelled=False)
[2017-05-10 13:35:00,636: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 13:35:00,655: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.target.ManagedTarget'> 2
[2017-05-10 13:35:00,655: INFO/job] StatefulObject.set_state testfs-MDT0000 unformatted->removed (intentional=True) 140430197773456
[2017-05-10 13:35:00,672: INFO/job] Job 76: StateChangeJob complete, setting state removed on testfs-MDT0000
[2017-05-10 13:35:00,672: INFO/job] Job 76: not_deleted=None
[2017-05-10 13:35:00,672: INFO/job_scheduler] Job 76: complete_job: Updating cache
[2017-05-10 13:35:00,672: INFO/job_scheduler] Job 76: locked_item 140430197773456 <class 'chroma_core.models.target.ManagedTarget'> False None
[2017-05-10 13:35:00,672: INFO/job_scheduler] Job 76 completing (errored=False, cancelled=False)
[2017-05-10 13:35:00,718: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 13:35:00,728: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.target.ManagedTarget'> 4
[2017-05-10 13:35:00,728: INFO/job] StatefulObject.set_state testfs-OST0001 formatted->removed (intentional=True) 140430197622544
[2017-05-10 13:35:00,734: INFO/job] Job 77: StateChangeJob complete, setting state removed on testfs-OST0001
[2017-05-10 13:35:00,734: INFO/job] Job 77: not_deleted=None
[2017-05-10 13:35:00,735: INFO/job_scheduler] Job 77: complete_job: Updating cache
[2017-05-10 13:35:00,735: INFO/job_scheduler] Job 77: locked_item 140430197622544 <class 'chroma_core.models.target.ManagedTarget'> False None
[2017-05-10 13:35:00,735: INFO/job_scheduler] Job 77 completing (errored=False, cancelled=False)
[2017-05-10 13:35:00,798: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 13:35:00,914: INFO/job_scheduler] Job 78: RunJobThread.run
[2017-05-10 13:35:00,915: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com purge_configuration {'mgs_device_path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/MGS', 'mgs_device_type': 'zfs', 'filesystem_name': u'testfs'}
[2017-05-10 13:35:00,950: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2a475178-32f4-4042-87a3-f4412bdd7628
[2017-05-10 13:35:00,950: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2a475178-32f4-4042-87a3-f4412bdd7628
[2017-05-10 13:35:01,032: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:35:01,032: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:35:01,032: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:35:01,032: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:35:01,032: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2a475178-32f4-4042-87a3-f4412bdd7628
[2017-05-10 13:35:01,032: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2a475178-32f4-4042-87a3-f4412bdd7628
[2017-05-10 13:35:01,076: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2a475178-32f4-4042-87a3-f4412bdd7628
[2017-05-10 13:35:01,076: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2a475178-32f4-4042-87a3-f4412bdd7628
[2017-05-10 13:35:01,078: INFO/job_scheduler] Job 78 finished 1 steps successfully
[2017-05-10 13:35:01,209: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.filesystem.ManagedFilesystem'> 1
[2017-05-10 13:35:01,209: INFO/job] StatefulObject.set_state testfs stopped->removed (intentional=True) 140430197262992
[2017-05-10 13:35:01,211: INFO/job] Job 78: StateChangeJob complete, setting state removed on testfs
[2017-05-10 13:35:01,211: INFO/job] Job 78: not_deleted=None
[2017-05-10 13:35:01,211: INFO/job_scheduler] Job 78: complete_job: Updating cache
[2017-05-10 13:35:01,211: INFO/job_scheduler] Job 78: locked_item 140430197262992 <class 'chroma_core.models.filesystem.ManagedFilesystem'> False None
[2017-05-10 13:35:01,212: INFO/job_scheduler] Job 78: locked_item 140430197841872 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:35:01,214: INFO/job_scheduler] Job 78 completing (errored=False, cancelled=False)
[2017-05-10 13:35:01,293: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:36:54,613: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:36:54,618: INFO/command_plan] set_state: <class 'chroma_core.models.target.ManagedTarget'>-1 transitioning from mounted to removed
[2017-05-10 13:36:54,671: INFO/command_plan] Created command 19 (Remove target MGS from the filesystem
*** WARNING *** this is destructive to your filesystem and irreversible) with 2 jobs
[2017-05-10 13:36:54,720: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 0 tasked)
[2017-05-10 13:36:54,781: INFO/job_scheduler] Job 79: RunJobThread.run
[2017-05-10 13:36:54,782: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_target {'ha_label': u'MGS_99a7c7'}
[2017-05-10 13:36:54,860: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a66f67b2-6045-41c2-9b3c-ed508c971b93
[2017-05-10 13:36:54,860: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a66f67b2-6045-41c2-9b3c-ed508c971b93
[2017-05-10 13:37:08,844: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 13:37:08,962: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:37:08,962: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:37:08,962: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:37:08,962: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:37:08,962: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a66f67b2-6045-41c2-9b3c-ed508c971b93
[2017-05-10 13:37:08,962: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a66f67b2-6045-41c2-9b3c-ed508c971b93
[2017-05-10 13:37:08,978: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a66f67b2-6045-41c2-9b3c-ed508c971b93
[2017-05-10 13:37:08,978: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a66f67b2-6045-41c2-9b3c-ed508c971b93
[2017-05-10 13:37:08,979: INFO/job_scheduler] Job 79 finished 1 steps successfully
[2017-05-10 13:37:09,077: INFO/job] StatefulObject.set_state MGS mounted->unmounted (intentional=True) 140430197615056
[2017-05-10 13:37:09,086: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on MGS at severity 30
[2017-05-10 13:37:09,090: INFO/job] Job 79: StateChangeJob complete, setting state unmounted on MGS
[2017-05-10 13:37:09,093: INFO/job] Job 79: not_deleted=True
[2017-05-10 13:37:09,094: INFO/job_scheduler] Job 79: complete_job: Updating cache
[2017-05-10 13:37:09,095: INFO/job_scheduler] Job 79: locked_item 140430197615056 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:37:09,101: INFO/job_scheduler] Job 79 completing (errored=False, cancelled=False)
[2017-05-10 13:37:09,152: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 13:37:09,211: INFO/job_scheduler] Job 80: RunJobThread.run
[2017-05-10 13:37:09,212: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com unconfigure_target_ha {'uuid': u'8316997203038885076', 'ha_label': u'MGS_99a7c7', 'primary': False}
[2017-05-10 13:37:09,301: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1533aef4-9a62-4950-bdb6-7a5591130c45
[2017-05-10 13:37:09,301: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1533aef4-9a62-4950-bdb6-7a5591130c45
[2017-05-10 13:37:17,113: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 13:37:27,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:37:27,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:37:27,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:37:27,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:37:27,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1533aef4-9a62-4950-bdb6-7a5591130c45
[2017-05-10 13:37:27,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1533aef4-9a62-4950-bdb6-7a5591130c45
[2017-05-10 13:37:27,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1533aef4-9a62-4950-bdb6-7a5591130c45
[2017-05-10 13:37:27,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1533aef4-9a62-4950-bdb6-7a5591130c45
[2017-05-10 13:37:27,525: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unconfigure_target_ha {'uuid': u'8316997203038885076', 'ha_label': u'MGS_99a7c7', 'primary': True}
[2017-05-10 13:37:27,540: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4effb6e3-1ed0-4424-9b32-c442951947e2
[2017-05-10 13:37:27,540: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4effb6e3-1ed0-4424-9b32-c442951947e2
[2017-05-10 13:37:46,736: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:37:46,736: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:37:46,736: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:37:46,736: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:37:46,736: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4effb6e3-1ed0-4424-9b32-c442951947e2
[2017-05-10 13:37:46,736: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4effb6e3-1ed0-4424-9b32-c442951947e2
[2017-05-10 13:37:46,758: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4effb6e3-1ed0-4424-9b32-c442951947e2
[2017-05-10 13:37:46,758: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4effb6e3-1ed0-4424-9b32-c442951947e2
[2017-05-10 13:37:46,760: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com unconfigure_target_store {'uuid': u'8316997203038885076'}
[2017-05-10 13:37:46,778: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5ce66be0-1d17-41e3-acb4-17deb9656c7f
[2017-05-10 13:37:46,778: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5ce66be0-1d17-41e3-acb4-17deb9656c7f
[2017-05-10 13:37:46,904: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:37:46,904: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:37:46,905: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:37:46,905: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:37:46,910: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5ce66be0-1d17-41e3-acb4-17deb9656c7f
[2017-05-10 13:37:46,910: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5ce66be0-1d17-41e3-acb4-17deb9656c7f
[2017-05-10 13:37:46,944: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5ce66be0-1d17-41e3-acb4-17deb9656c7f
[2017-05-10 13:37:46,944: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5ce66be0-1d17-41e3-acb4-17deb9656c7f
[2017-05-10 13:37:46,945: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unconfigure_target_store {'uuid': u'8316997203038885076'}
[2017-05-10 13:37:46,964: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0f7dbe60-5bc3-4181-9eb1-d0930f4396e1
[2017-05-10 13:37:46,964: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0f7dbe60-5bc3-4181-9eb1-d0930f4396e1
[2017-05-10 13:37:47,027: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:37:47,027: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:37:47,027: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:37:47,027: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:37:47,027: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0f7dbe60-5bc3-4181-9eb1-d0930f4396e1
[2017-05-10 13:37:47,027: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0f7dbe60-5bc3-4181-9eb1-d0930f4396e1
[2017-05-10 13:37:47,030: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0f7dbe60-5bc3-4181-9eb1-d0930f4396e1
[2017-05-10 13:37:47,030: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0f7dbe60-5bc3-4181-9eb1-d0930f4396e1
[2017-05-10 13:37:47,031: INFO/job_scheduler] Job 80 finished 4 steps successfully
[2017-05-10 13:37:47,071: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.target.ManagedTarget'> 1
[2017-05-10 13:37:47,071: INFO/job] StatefulObject.set_state MGS unmounted->removed (intentional=True) 140430197615056
[2017-05-10 13:37:47,094: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 20
[2017-05-10 13:37:47,105: INFO/job] Job 80: StateChangeJob complete, setting state removed on MGS
[2017-05-10 13:37:47,105: INFO/job] Job 80: not_deleted=None
[2017-05-10 13:37:47,105: INFO/job_scheduler] Job 80: complete_job: Updating cache
[2017-05-10 13:37:47,105: INFO/job_scheduler] Job 80: locked_item 140430197615056 <class 'chroma_core.models.target.ManagedTarget'> False None
[2017-05-10 13:37:47,106: INFO/job_scheduler] Job 80 completing (errored=False, cancelled=False)
[2017-05-10 13:37:47,184: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:38:08,822: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:38:08,834: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:38:10,154: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:38:10,170: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:38:10,335: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:38:10,354: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:38:12,651: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:38:12,668: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:38:21,412: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:38:21,413: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 13:38:21,444: INFO/command_plan] add_jobs: created Job 81 (Trigger plugin poll for linux plugins)
[2017-05-10 13:38:21,510: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 13:38:21,524: INFO/job_scheduler] Job 81: RunJobThread.run
[2017-05-10 13:38:21,525: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:38:21,541: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:38:21,544: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 15a557fa-ea7b-4ca3-918a-3675bba57bae
[2017-05-10 13:38:21,544: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 15a557fa-ea7b-4ca3-918a-3675bba57bae
[2017-05-10 13:38:21,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2c288f10-fc6c-4b33-ae09-195dc13f3551
[2017-05-10 13:38:21,559: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2c288f10-fc6c-4b33-ae09-195dc13f3551
[2017-05-10 13:38:21,556: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:38:21,576: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 587a762d-b7c5-4baf-8ab9-f5eb12a60dff
[2017-05-10 13:38:21,576: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 587a762d-b7c5-4baf-8ab9-f5eb12a60dff
[2017-05-10 13:38:21,601: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:38:21,601: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:38:21,601: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:38:21,601: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:38:21,601: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 15a557fa-ea7b-4ca3-918a-3675bba57bae
[2017-05-10 13:38:21,601: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 15a557fa-ea7b-4ca3-918a-3675bba57bae
[2017-05-10 13:38:21,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:38:21,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:38:21,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:38:21,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:38:21,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2c288f10-fc6c-4b33-ae09-195dc13f3551
[2017-05-10 13:38:21,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2c288f10-fc6c-4b33-ae09-195dc13f3551
[2017-05-10 13:38:21,624: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 15a557fa-ea7b-4ca3-918a-3675bba57bae
[2017-05-10 13:38:21,624: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 15a557fa-ea7b-4ca3-918a-3675bba57bae
[2017-05-10 13:38:21,624: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:38:21,636: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2c288f10-fc6c-4b33-ae09-195dc13f3551
[2017-05-10 13:38:21,636: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2c288f10-fc6c-4b33-ae09-195dc13f3551
[2017-05-10 13:38:21,637: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:38:21,783: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:38:21,783: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:38:21,783: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:38:21,783: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:38:21,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 587a762d-b7c5-4baf-8ab9-f5eb12a60dff
[2017-05-10 13:38:21,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 587a762d-b7c5-4baf-8ab9-f5eb12a60dff
[2017-05-10 13:38:21,802: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 587a762d-b7c5-4baf-8ab9-f5eb12a60dff
[2017-05-10 13:38:21,802: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 587a762d-b7c5-4baf-8ab9-f5eb12a60dff
[2017-05-10 13:38:21,803: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:38:21,803: INFO/job_scheduler] Job 81 finished 1 steps successfully
[2017-05-10 13:38:21,862: INFO/job_scheduler] Job 81: complete_job: Updating cache
[2017-05-10 13:38:21,862: INFO/job_scheduler] Job 81 completing (errored=False, cancelled=False)
[2017-05-10 13:38:21,910: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:39:00,834: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:39:00,835: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 13:39:00,862: INFO/command_plan] add_jobs: created Job 82 (Trigger plugin poll for linux plugins)
[2017-05-10 13:39:00,888: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 13:39:00,910: INFO/job_scheduler] Job 82: RunJobThread.run
[2017-05-10 13:39:00,945: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:39:00,966: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:39:00,975: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:39:00,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 81429d6f-4295-48d5-9ef9-874ffb6b7b4d
[2017-05-10 13:39:00,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 81429d6f-4295-48d5-9ef9-874ffb6b7b4d
[2017-05-10 13:39:00,990: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0299a31b-0994-493f-aaa0-b2d4f1114a0a
[2017-05-10 13:39:00,990: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0299a31b-0994-493f-aaa0-b2d4f1114a0a
[2017-05-10 13:39:01,002: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 293cc7cf-2ed9-4cc1-ba86-5f41c2d0d683
[2017-05-10 13:39:01,002: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 293cc7cf-2ed9-4cc1-ba86-5f41c2d0d683
[2017-05-10 13:39:01,077: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:39:01,077: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:39:01,078: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:39:01,078: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:39:01,078: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 81429d6f-4295-48d5-9ef9-874ffb6b7b4d
[2017-05-10 13:39:01,078: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 81429d6f-4295-48d5-9ef9-874ffb6b7b4d
[2017-05-10 13:39:01,081: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:39:01,081: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:39:01,082: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:39:01,082: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:39:01,082: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0299a31b-0994-493f-aaa0-b2d4f1114a0a
[2017-05-10 13:39:01,082: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0299a31b-0994-493f-aaa0-b2d4f1114a0a
[2017-05-10 13:39:01,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:39:01,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:39:01,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:39:01,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:39:01,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 293cc7cf-2ed9-4cc1-ba86-5f41c2d0d683
[2017-05-10 13:39:01,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 293cc7cf-2ed9-4cc1-ba86-5f41c2d0d683
[2017-05-10 13:39:01,106: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 81429d6f-4295-48d5-9ef9-874ffb6b7b4d
[2017-05-10 13:39:01,106: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 81429d6f-4295-48d5-9ef9-874ffb6b7b4d
[2017-05-10 13:39:01,107: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:39:01,118: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0299a31b-0994-493f-aaa0-b2d4f1114a0a
[2017-05-10 13:39:01,118: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0299a31b-0994-493f-aaa0-b2d4f1114a0a
[2017-05-10 13:39:01,119: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:39:01,127: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 293cc7cf-2ed9-4cc1-ba86-5f41c2d0d683
[2017-05-10 13:39:01,127: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 293cc7cf-2ed9-4cc1-ba86-5f41c2d0d683
[2017-05-10 13:39:01,127: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:39:01,128: INFO/job_scheduler] Job 82 finished 1 steps successfully
[2017-05-10 13:39:01,187: INFO/job_scheduler] Job 82: complete_job: Updating cache
[2017-05-10 13:39:01,187: INFO/job_scheduler] Job 82 completing (errored=False, cancelled=False)
[2017-05-10 13:39:01,230: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:39:02,164: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:39:02,164: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 13:39:02,212: INFO/command_plan] add_jobs: created Job 83 (Trigger plugin poll for linux plugins)
[2017-05-10 13:39:02,232: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 13:39:02,266: INFO/job_scheduler] Job 83: RunJobThread.run
[2017-05-10 13:39:02,280: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:39:02,299: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:39:02,316: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:39:02,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fd95d1b1-6fba-4525-b8de-193faf499ae6
[2017-05-10 13:39:02,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fd95d1b1-6fba-4525-b8de-193faf499ae6
[2017-05-10 13:39:02,337: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d11a4a09-a1f1-4aa1-947f-34d065b30588
[2017-05-10 13:39:02,337: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d11a4a09-a1f1-4aa1-947f-34d065b30588
[2017-05-10 13:39:02,352: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fe698b23-ba94-4f35-851e-54b5d3b78166
[2017-05-10 13:39:02,352: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fe698b23-ba94-4f35-851e-54b5d3b78166
[2017-05-10 13:39:02,405: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:39:02,405: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:39:02,405: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:39:02,405: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:39:02,406: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fd95d1b1-6fba-4525-b8de-193faf499ae6
[2017-05-10 13:39:02,406: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fd95d1b1-6fba-4525-b8de-193faf499ae6
[2017-05-10 13:39:02,406: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fd95d1b1-6fba-4525-b8de-193faf499ae6
[2017-05-10 13:39:02,406: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fd95d1b1-6fba-4525-b8de-193faf499ae6
[2017-05-10 13:39:02,406: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:39:02,408: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:39:02,408: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:39:02,408: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:39:02,408: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:39:02,408: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d11a4a09-a1f1-4aa1-947f-34d065b30588
[2017-05-10 13:39:02,408: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d11a4a09-a1f1-4aa1-947f-34d065b30588
[2017-05-10 13:39:02,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d11a4a09-a1f1-4aa1-947f-34d065b30588
[2017-05-10 13:39:02,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d11a4a09-a1f1-4aa1-947f-34d065b30588
[2017-05-10 13:39:02,412: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:39:02,436: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:39:02,436: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:39:02,436: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:39:02,436: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:39:02,436: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fe698b23-ba94-4f35-851e-54b5d3b78166
[2017-05-10 13:39:02,436: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fe698b23-ba94-4f35-851e-54b5d3b78166
[2017-05-10 13:39:02,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fe698b23-ba94-4f35-851e-54b5d3b78166
[2017-05-10 13:39:02,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fe698b23-ba94-4f35-851e-54b5d3b78166
[2017-05-10 13:39:02,472: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:39:02,473: INFO/job_scheduler] Job 83 finished 1 steps successfully
[2017-05-10 13:39:02,537: INFO/job_scheduler] Job 83: complete_job: Updating cache
[2017-05-10 13:39:02,537: INFO/job_scheduler] Job 83 completing (errored=False, cancelled=False)
[2017-05-10 13:39:02,577: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:39:07,081: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:39:07,082: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 13:39:07,098: INFO/command_plan] add_jobs: created Job 84 (Trigger plugin poll for linux plugins)
[2017-05-10 13:39:07,149: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 13:39:07,173: INFO/job_scheduler] Job 84: RunJobThread.run
[2017-05-10 13:39:07,203: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:39:07,230: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:39:07,235: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ccc12e00-e4c4-44c9-82d6-afc52c3e7370
[2017-05-10 13:39:07,235: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ccc12e00-e4c4-44c9-82d6-afc52c3e7370
[2017-05-10 13:39:07,248: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:39:07,253: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a86703ec-77af-4ea0-b38b-94f6f3bf7534
[2017-05-10 13:39:07,253: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a86703ec-77af-4ea0-b38b-94f6f3bf7534
[2017-05-10 13:39:07,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 86ce7ce8-2ad8-4893-a715-f5fcc64756d4
[2017-05-10 13:39:07,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 86ce7ce8-2ad8-4893-a715-f5fcc64756d4
[2017-05-10 13:39:07,377: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:39:07,377: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:39:07,377: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:39:07,377: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:39:07,377: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 86ce7ce8-2ad8-4893-a715-f5fcc64756d4
[2017-05-10 13:39:07,377: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 86ce7ce8-2ad8-4893-a715-f5fcc64756d4
[2017-05-10 13:39:07,392: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:39:07,392: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:39:07,392: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:39:07,392: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:39:07,392: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ccc12e00-e4c4-44c9-82d6-afc52c3e7370
[2017-05-10 13:39:07,392: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ccc12e00-e4c4-44c9-82d6-afc52c3e7370
[2017-05-10 13:39:07,394: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:39:07,394: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:39:07,394: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:39:07,394: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:39:07,395: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a86703ec-77af-4ea0-b38b-94f6f3bf7534
[2017-05-10 13:39:07,395: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a86703ec-77af-4ea0-b38b-94f6f3bf7534
[2017-05-10 13:39:07,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 86ce7ce8-2ad8-4893-a715-f5fcc64756d4
[2017-05-10 13:39:07,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 86ce7ce8-2ad8-4893-a715-f5fcc64756d4
[2017-05-10 13:39:07,399: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:39:07,425: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ccc12e00-e4c4-44c9-82d6-afc52c3e7370
[2017-05-10 13:39:07,425: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ccc12e00-e4c4-44c9-82d6-afc52c3e7370
[2017-05-10 13:39:07,425: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:39:07,433: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a86703ec-77af-4ea0-b38b-94f6f3bf7534
[2017-05-10 13:39:07,433: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a86703ec-77af-4ea0-b38b-94f6f3bf7534
[2017-05-10 13:39:07,433: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:39:07,434: INFO/job_scheduler] Job 84 finished 1 steps successfully
[2017-05-10 13:39:07,470: INFO/job_scheduler] Job 84: complete_job: Updating cache
[2017-05-10 13:39:07,470: INFO/job_scheduler] Job 84 completing (errored=False, cancelled=False)
[2017-05-10 13:39:07,517: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:39:15,209: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:39:15,211: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 13:39:15,233: INFO/command_plan] add_jobs: created Job 85 (Trigger plugin poll for linux plugins)
[2017-05-10 13:39:15,288: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 13:39:15,325: INFO/job_scheduler] Job 85: RunJobThread.run
[2017-05-10 13:39:15,362: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:39:15,374: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:39:15,376: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:39:15,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 62494603-f85f-45c7-9be6-02c468795fb1
[2017-05-10 13:39:15,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 62494603-f85f-45c7-9be6-02c468795fb1
[2017-05-10 13:39:15,396: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a0379268-37ab-4070-b3cf-699a2800d867
[2017-05-10 13:39:15,396: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a0379268-37ab-4070-b3cf-699a2800d867
[2017-05-10 13:39:15,415: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2fdb6767-74bd-4d85-bae0-da4f5dbc1f3d
[2017-05-10 13:39:15,415: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2fdb6767-74bd-4d85-bae0-da4f5dbc1f3d
[2017-05-10 13:39:15,471: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:39:15,471: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:39:15,471: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:39:15,471: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:39:15,471: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 62494603-f85f-45c7-9be6-02c468795fb1
[2017-05-10 13:39:15,471: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 62494603-f85f-45c7-9be6-02c468795fb1
[2017-05-10 13:39:15,475: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:39:15,475: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:39:15,475: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:39:15,475: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:39:15,475: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2fdb6767-74bd-4d85-bae0-da4f5dbc1f3d
[2017-05-10 13:39:15,475: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2fdb6767-74bd-4d85-bae0-da4f5dbc1f3d
[2017-05-10 13:39:15,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:39:15,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:39:15,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:39:15,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:39:15,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a0379268-37ab-4070-b3cf-699a2800d867
[2017-05-10 13:39:15,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a0379268-37ab-4070-b3cf-699a2800d867
[2017-05-10 13:39:15,481: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2fdb6767-74bd-4d85-bae0-da4f5dbc1f3d
[2017-05-10 13:39:15,481: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2fdb6767-74bd-4d85-bae0-da4f5dbc1f3d
[2017-05-10 13:39:15,481: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:39:15,506: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 62494603-f85f-45c7-9be6-02c468795fb1
[2017-05-10 13:39:15,506: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 62494603-f85f-45c7-9be6-02c468795fb1
[2017-05-10 13:39:15,507: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:39:15,524: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a0379268-37ab-4070-b3cf-699a2800d867
[2017-05-10 13:39:15,524: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a0379268-37ab-4070-b3cf-699a2800d867
[2017-05-10 13:39:15,556: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:39:15,556: INFO/job_scheduler] Job 85 finished 1 steps successfully
[2017-05-10 13:39:15,583: INFO/job_scheduler] Job 85: complete_job: Updating cache
[2017-05-10 13:39:15,583: INFO/job_scheduler] Job 85 completing (errored=False, cancelled=False)
[2017-05-10 13:39:15,637: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:40:38,565: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:40:38,567: INFO/command_plan] set_state: <class 'chroma_core.models.filesystem.ManagedFilesystem'>-2 transitioning from unavailable to available
[2017-05-10 13:40:38,919: INFO/command_plan] Created command 25 (Creating filesystem testfs) with 17 jobs
[2017-05-10 13:40:39,020: INFO/job_scheduler] run_next: 4 runnable jobs of (17 pending, 0 tasked)
[2017-05-10 13:40:39,595: INFO/job_scheduler] Job 86: RunJobThread.run
[2017-05-10 13:40:39,597: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1', 'device_type': 'zfs'}
[2017-05-10 13:40:39,605: INFO/job_scheduler] Job 87: RunJobThread.run
[2017-05-10 13:40:39,606: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1', 'device_type': 'zfs'}
[2017-05-10 13:40:39,606: INFO/job_scheduler] Job 88: RunJobThread.run
[2017-05-10 13:40:39,616: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e9dff79c-b510-49fd-98cf-9c17571aa5b2
[2017-05-10 13:40:39,616: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e9dff79c-b510-49fd-98cf-9c17571aa5b2
[2017-05-10 13:40:39,633: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1', 'device_type': 'zfs'}
[2017-05-10 13:40:39,607: INFO/job_scheduler] Job 89: RunJobThread.run
[2017-05-10 13:40:39,640: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', 'device_type': 'linux'}
[2017-05-10 13:40:39,663: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 85b028c9-7ec2-4e45-b8ec-e7650cb3653f
[2017-05-10 13:40:39,663: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 85b028c9-7ec2-4e45-b8ec-e7650cb3653f
[2017-05-10 13:40:39,668: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux'}
[2017-05-10 13:40:39,680: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5', 'device_type': 'zfs'}
[2017-05-10 13:40:39,705: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux'}
[2017-05-10 13:40:39,721: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5', 'device_type': 'zfs'}
[2017-05-10 13:40:39,725: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', 'device_type': 'linux'}
[2017-05-10 13:40:39,731: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux'}
[2017-05-10 13:40:39,742: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 09a98b43-5999-4b84-b767-27902793ae92
[2017-05-10 13:40:39,742: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 09a98b43-5999-4b84-b767-27902793ae92
[2017-05-10 13:40:39,753: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', 'device_type': 'linux'}
[2017-05-10 13:40:39,754: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5', 'device_type': 'zfs'}
[2017-05-10 13:40:39,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c910b511-6b66-4983-817f-d563a577c240
[2017-05-10 13:40:39,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c910b511-6b66-4983-817f-d563a577c240
[2017-05-10 13:40:39,812: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1bc1ccc0-cd87-4a44-809c-6a81d6a80228
[2017-05-10 13:40:39,812: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1bc1ccc0-cd87-4a44-809c-6a81d6a80228
[2017-05-10 13:40:39,860: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 146a8685-6f52-42d1-96c9-4141cb58fcea
[2017-05-10 13:40:39,860: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 146a8685-6f52-42d1-96c9-4141cb58fcea
[2017-05-10 13:40:39,886: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc af6a7192-a3d1-4eed-abfa-2c17f0d43c23
[2017-05-10 13:40:39,886: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc af6a7192-a3d1-4eed-abfa-2c17f0d43c23
[2017-05-10 13:40:39,912: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d7f92ba8-1025-469b-925c-97663efff215
[2017-05-10 13:40:39,912: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d7f92ba8-1025-469b-925c-97663efff215
[2017-05-10 13:40:39,949: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 507fbc79-4d21-4c17-ae84-5be6b3864300
[2017-05-10 13:40:39,949: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 507fbc79-4d21-4c17-ae84-5be6b3864300
[2017-05-10 13:40:39,969: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0d785ff6-664a-4c2e-8c72-d1b414197ebc
[2017-05-10 13:40:39,969: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0d785ff6-664a-4c2e-8c72-d1b414197ebc
[2017-05-10 13:40:40,013: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 59e1566b-92c2-4e5b-ade6-b390de2e5dd0
[2017-05-10 13:40:40,013: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 59e1566b-92c2-4e5b-ade6-b390de2e5dd0
[2017-05-10 13:40:40,042: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dffb2e86-22d5-4b13-b69b-ee89faf0b2b3
[2017-05-10 13:40:40,042: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dffb2e86-22d5-4b13-b69b-ee89faf0b2b3
[2017-05-10 13:40:40,042: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:40:40,042: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:40:40,042: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:40:40,042: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:40:40,042: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c910b511-6b66-4983-817f-d563a577c240
[2017-05-10 13:40:40,042: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c910b511-6b66-4983-817f-d563a577c240
[2017-05-10 13:40:40,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:40:40,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:40:40,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:40:40,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:40:40,044: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc af6a7192-a3d1-4eed-abfa-2c17f0d43c23
[2017-05-10 13:40:40,044: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc af6a7192-a3d1-4eed-abfa-2c17f0d43c23
[2017-05-10 13:40:40,044: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:40:40,044: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:40:40,044: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:40:40,044: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:40:40,044: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d7f92ba8-1025-469b-925c-97663efff215
[2017-05-10 13:40:40,044: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d7f92ba8-1025-469b-925c-97663efff215
[2017-05-10 13:40:40,045: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:40,045: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:40,045: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:40,045: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:40,045: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 507fbc79-4d21-4c17-ae84-5be6b3864300
[2017-05-10 13:40:40,045: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 507fbc79-4d21-4c17-ae84-5be6b3864300
[2017-05-10 13:40:40,046: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:40:40,046: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:40:40,046: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:40:40,046: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:40:40,046: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0d785ff6-664a-4c2e-8c72-d1b414197ebc
[2017-05-10 13:40:40,046: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0d785ff6-664a-4c2e-8c72-d1b414197ebc
[2017-05-10 13:40:40,056: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0d785ff6-664a-4c2e-8c72-d1b414197ebc
[2017-05-10 13:40:40,056: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0d785ff6-664a-4c2e-8c72-d1b414197ebc
[2017-05-10 13:40:40,076: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc af6a7192-a3d1-4eed-abfa-2c17f0d43c23
[2017-05-10 13:40:40,076: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc af6a7192-a3d1-4eed-abfa-2c17f0d43c23
[2017-05-10 13:40:40,077: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 507fbc79-4d21-4c17-ae84-5be6b3864300
[2017-05-10 13:40:40,077: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 507fbc79-4d21-4c17-ae84-5be6b3864300
[2017-05-10 13:40:40,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c910b511-6b66-4983-817f-d563a577c240
[2017-05-10 13:40:40,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c910b511-6b66-4983-817f-d563a577c240
[2017-05-10 13:40:40,088: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com import_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'pacemaker_ha_operation': False, 'device_type': 'linux', 'validate_importable': False}
[2017-05-10 13:40:40,096: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d7f92ba8-1025-469b-925c-97663efff215
[2017-05-10 13:40:40,096: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d7f92ba8-1025-469b-925c-97663efff215
[2017-05-10 13:40:40,106: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 46503369-1576-4b9f-9324-9ce2964ee595
[2017-05-10 13:40:40,106: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 46503369-1576-4b9f-9324-9ce2964ee595
[2017-05-10 13:40:40,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:40,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:40,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:40,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:40,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 59e1566b-92c2-4e5b-ade6-b390de2e5dd0
[2017-05-10 13:40:40,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 59e1566b-92c2-4e5b-ade6-b390de2e5dd0
[2017-05-10 13:40:40,153: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 59e1566b-92c2-4e5b-ade6-b390de2e5dd0
[2017-05-10 13:40:40,153: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 59e1566b-92c2-4e5b-ade6-b390de2e5dd0
[2017-05-10 13:40:40,155: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com import_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', 'pacemaker_ha_operation': False, 'device_type': 'linux', 'validate_importable': False}
[2017-05-10 13:40:40,170: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 23bc393a-ff99-4253-8c5d-051570cba0e4
[2017-05-10 13:40:40,170: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 23bc393a-ff99-4253-8c5d-051570cba0e4
[2017-05-10 13:40:40,173: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:40,173: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:40,173: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:40,173: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:40,174: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 46503369-1576-4b9f-9324-9ce2964ee595
[2017-05-10 13:40:40,174: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 46503369-1576-4b9f-9324-9ce2964ee595
[2017-05-10 13:40:40,221: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:40,221: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:40,233: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:40,233: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:40,233: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 23bc393a-ff99-4253-8c5d-051570cba0e4
[2017-05-10 13:40:40,233: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 23bc393a-ff99-4253-8c5d-051570cba0e4
[2017-05-10 13:40:40,223: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 46503369-1576-4b9f-9324-9ce2964ee595
[2017-05-10 13:40:40,223: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 46503369-1576-4b9f-9324-9ce2964ee595
[2017-05-10 13:40:40,236: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com check_block_device {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux'}
[2017-05-10 13:40:40,244: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 23bc393a-ff99-4253-8c5d-051570cba0e4
[2017-05-10 13:40:40,244: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 23bc393a-ff99-4253-8c5d-051570cba0e4
[2017-05-10 13:40:40,245: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com check_block_device {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', 'device_type': 'linux'}
[2017-05-10 13:40:40,266: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f8bb4ff6-f9ff-4449-a8bd-bb3f81f7b77c
[2017-05-10 13:40:40,266: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f8bb4ff6-f9ff-4449-a8bd-bb3f81f7b77c
[2017-05-10 13:40:40,303: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc eebe2c62-ac52-4505-8494-9c826f37e341
[2017-05-10 13:40:40,303: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc eebe2c62-ac52-4505-8494-9c826f37e341
[2017-05-10 13:40:40,851: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:40,851: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:40,851: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:40,851: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:40,851: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 09a98b43-5999-4b84-b767-27902793ae92
[2017-05-10 13:40:40,851: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 09a98b43-5999-4b84-b767-27902793ae92
[2017-05-10 13:40:40,857: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 09a98b43-5999-4b84-b767-27902793ae92
[2017-05-10 13:40:40,857: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 09a98b43-5999-4b84-b767-27902793ae92
[2017-05-10 13:40:41,020: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:40:41,020: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:40:41,020: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:40:41,020: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:40:41,020: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 146a8685-6f52-42d1-96c9-4141cb58fcea
[2017-05-10 13:40:41,020: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 146a8685-6f52-42d1-96c9-4141cb58fcea
[2017-05-10 13:40:41,041: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 146a8685-6f52-42d1-96c9-4141cb58fcea
[2017-05-10 13:40:41,041: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 146a8685-6f52-42d1-96c9-4141cb58fcea
[2017-05-10 13:40:41,148: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:41,148: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:41,148: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:41,148: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:41,149: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dffb2e86-22d5-4b13-b69b-ee89faf0b2b3
[2017-05-10 13:40:41,149: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dffb2e86-22d5-4b13-b69b-ee89faf0b2b3
[2017-05-10 13:40:41,160: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dffb2e86-22d5-4b13-b69b-ee89faf0b2b3
[2017-05-10 13:40:41,160: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dffb2e86-22d5-4b13-b69b-ee89faf0b2b3
[2017-05-10 13:40:41,378: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:41,378: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:41,378: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:41,378: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:41,378: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f8bb4ff6-f9ff-4449-a8bd-bb3f81f7b77c
[2017-05-10 13:40:41,378: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f8bb4ff6-f9ff-4449-a8bd-bb3f81f7b77c
[2017-05-10 13:40:41,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:41,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:41,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:41,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:41,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc eebe2c62-ac52-4505-8494-9c826f37e341
[2017-05-10 13:40:41,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc eebe2c62-ac52-4505-8494-9c826f37e341
[2017-05-10 13:40:41,392: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f8bb4ff6-f9ff-4449-a8bd-bb3f81f7b77c
[2017-05-10 13:40:41,392: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f8bb4ff6-f9ff-4449-a8bd-bb3f81f7b77c
[2017-05-10 13:40:41,401: INFO/job] MGS passed pre-format check, allowing subsequent reformats
[2017-05-10 13:40:41,426: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc eebe2c62-ac52-4505-8494-9c826f37e341
[2017-05-10 13:40:41,426: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc eebe2c62-ac52-4505-8494-9c826f37e341
[2017-05-10 13:40:41,428: INFO/job] testfs-MDT0000 passed pre-format check, allowing subsequent reformats
[2017-05-10 13:40:41,466: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com format_target {'backfstype': 'ldiskfs', 'target_name': u'MGS', 'target_types': 'mgs', 'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux', 'failnode': [u'10.14.81.225@tcp0']}
[2017-05-10 13:40:41,485: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6f8437d6-9a43-4809-b8b4-9c8015a34f41
[2017-05-10 13:40:41,485: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6f8437d6-9a43-4809-b8b4-9c8015a34f41
[2017-05-10 13:40:41,492: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com format_target {'index': 0, 'mkfsoptions': '-J size=2048', 'backfstype': 'ldiskfs', 'target_name': u'testfs-MDT0000', 'mgsnode': ((u'10.14.81.224@tcp0',), (u'10.14.81.225@tcp0',)), 'target_types': 'mdt', 'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', 'device_type': 'linux', 'failnode': [u'10.14.81.224@tcp0'], 'fsname': u'testfs'}
[2017-05-10 13:40:41,511: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6d3b36ba-6abd-4c2e-96c9-c0ff3880318f
[2017-05-10 13:40:41,511: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6d3b36ba-6abd-4c2e-96c9-c0ff3880318f
[2017-05-10 13:40:41,709: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:41,709: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:41,709: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:41,709: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:41,709: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e9dff79c-b510-49fd-98cf-9c17571aa5b2
[2017-05-10 13:40:41,709: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e9dff79c-b510-49fd-98cf-9c17571aa5b2
[2017-05-10 13:40:41,752: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e9dff79c-b510-49fd-98cf-9c17571aa5b2
[2017-05-10 13:40:41,752: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e9dff79c-b510-49fd-98cf-9c17571aa5b2
[2017-05-10 13:40:42,732: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:40:42,732: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:40:42,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:40:42,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:40:42,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 85b028c9-7ec2-4e45-b8ec-e7650cb3653f
[2017-05-10 13:40:42,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 85b028c9-7ec2-4e45-b8ec-e7650cb3653f
[2017-05-10 13:40:42,743: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 85b028c9-7ec2-4e45-b8ec-e7650cb3653f
[2017-05-10 13:40:42,743: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 85b028c9-7ec2-4e45-b8ec-e7650cb3653f
[2017-05-10 13:40:42,743: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com import_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1', 'pacemaker_ha_operation': False, 'device_type': 'zfs', 'validate_importable': False}
[2017-05-10 13:40:42,775: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 32840a3c-01d2-4a99-b7e8-cd05cabf8b1b
[2017-05-10 13:40:42,775: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 32840a3c-01d2-4a99-b7e8-cd05cabf8b1b
[2017-05-10 13:40:44,114: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:44,114: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:44,114: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:44,114: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:44,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1bc1ccc0-cd87-4a44-809c-6a81d6a80228
[2017-05-10 13:40:44,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1bc1ccc0-cd87-4a44-809c-6a81d6a80228
[2017-05-10 13:40:44,134: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1bc1ccc0-cd87-4a44-809c-6a81d6a80228
[2017-05-10 13:40:44,134: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1bc1ccc0-cd87-4a44-809c-6a81d6a80228
[2017-05-10 13:40:44,135: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com import_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5', 'pacemaker_ha_operation': False, 'device_type': 'zfs', 'validate_importable': False}
[2017-05-10 13:40:44,161: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ccd81a09-e679-4c70-b072-c22d2a5f17b6
[2017-05-10 13:40:44,161: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ccd81a09-e679-4c70-b072-c22d2a5f17b6
[2017-05-10 13:40:45,888: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:40:45,888: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:40:45,888: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:40:45,888: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:40:45,889: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 32840a3c-01d2-4a99-b7e8-cd05cabf8b1b
[2017-05-10 13:40:45,889: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 32840a3c-01d2-4a99-b7e8-cd05cabf8b1b
[2017-05-10 13:40:45,891: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 32840a3c-01d2-4a99-b7e8-cd05cabf8b1b
[2017-05-10 13:40:45,891: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 32840a3c-01d2-4a99-b7e8-cd05cabf8b1b
[2017-05-10 13:40:45,892: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com check_block_device {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1', 'device_type': 'zfs'}
[2017-05-10 13:40:45,906: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9d5013fe-1595-4cf4-b078-fc5b973eb8e1
[2017-05-10 13:40:45,906: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9d5013fe-1595-4cf4-b078-fc5b973eb8e1
[2017-05-10 13:40:46,998: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:40:46,998: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:40:46,998: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:40:46,998: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:40:46,999: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9d5013fe-1595-4cf4-b078-fc5b973eb8e1
[2017-05-10 13:40:46,999: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9d5013fe-1595-4cf4-b078-fc5b973eb8e1
[2017-05-10 13:40:47,024: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9d5013fe-1595-4cf4-b078-fc5b973eb8e1
[2017-05-10 13:40:47,024: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9d5013fe-1595-4cf4-b078-fc5b973eb8e1
[2017-05-10 13:40:47,026: INFO/job] testfs-OST0000 passed pre-format check, allowing subsequent reformats
[2017-05-10 13:40:47,089: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com format_target {'index': 0, 'backfstype': 'zfs', 'target_name': u'testfs-OST0000', 'mgsnode': ((u'10.14.81.224@tcp0',), (u'10.14.81.225@tcp0',)), 'target_types': 'ost', 'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1', 'device_type': 'zfs', 'failnode': [u'10.14.81.227@tcp0'], 'fsname': u'testfs'}
[2017-05-10 13:40:47,107: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc af1735b8-a566-492e-9608-8d38f46b7bd8
[2017-05-10 13:40:47,107: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc af1735b8-a566-492e-9608-8d38f46b7bd8
[2017-05-10 13:40:48,280: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:40:48,280: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:40:48,280: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:40:48,280: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:40:48,280: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ccd81a09-e679-4c70-b072-c22d2a5f17b6
[2017-05-10 13:40:48,280: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ccd81a09-e679-4c70-b072-c22d2a5f17b6
[2017-05-10 13:40:48,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ccd81a09-e679-4c70-b072-c22d2a5f17b6
[2017-05-10 13:40:48,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ccd81a09-e679-4c70-b072-c22d2a5f17b6
[2017-05-10 13:40:48,283: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com check_block_device {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5', 'device_type': 'zfs'}
[2017-05-10 13:40:48,300: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d9720059-d5d6-4b59-a2c9-0b72713844e4
[2017-05-10 13:40:48,300: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d9720059-d5d6-4b59-a2c9-0b72713844e4
[2017-05-10 13:40:48,627: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:48,627: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:48,627: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:48,627: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:48,627: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6f8437d6-9a43-4809-b8b4-9c8015a34f41
[2017-05-10 13:40:48,627: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6f8437d6-9a43-4809-b8b4-9c8015a34f41
[2017-05-10 13:40:48,666: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6f8437d6-9a43-4809-b8b4-9c8015a34f41
[2017-05-10 13:40:48,666: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6f8437d6-9a43-4809-b8b4-9c8015a34f41
[2017-05-10 13:40:48,722: INFO/job] Updating mtm volume_nodes for target MGS
[2017-05-10 13:40:48,839: INFO/job_scheduler] Job 88 finished 5 steps successfully
[2017-05-10 13:40:48,879: INFO/job] StatefulObject.set_state MGS unformatted->formatted (intentional=True) 140430197713232
[2017-05-10 13:40:48,885: INFO/job] Job 88: StateChangeJob complete, setting state formatted on MGS
[2017-05-10 13:40:48,885: INFO/job] Job 88: not_deleted=True
[2017-05-10 13:40:48,892: INFO/job_scheduler] Job 88: complete_job: Updating cache
[2017-05-10 13:40:48,892: INFO/job_scheduler] Job 88: locked_item 140430197713232 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:40:48,895: INFO/job_scheduler] Job 88: locked_item 140430198048208 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-10 13:40:48,896: INFO/job_scheduler] Job 88: locked_item 140430198047952 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-10 13:40:48,898: INFO/job_scheduler] Job 88 completing (errored=False, cancelled=False)
[2017-05-10 13:40:48,932: INFO/job_scheduler] run_next: 1 runnable jobs of (13 pending, 3 tasked)
[2017-05-10 13:40:48,988: INFO/job_scheduler] Job 90: RunJobThread.run
[2017-05-10 13:40:48,989: INFO/job_scheduler] Job 90 finished 1 steps successfully
[2017-05-10 13:40:49,021: INFO/job] StatefulObject.set_state MGS formatted->registered (intentional=True) 140430197713232
[2017-05-10 13:40:49,026: INFO/job] Job 90: StateChangeJob complete, setting state registered on MGS
[2017-05-10 13:40:49,026: INFO/job] Job 90: not_deleted=True
[2017-05-10 13:40:49,026: INFO/job_scheduler] Job 90: complete_job: Updating cache
[2017-05-10 13:40:49,026: INFO/job_scheduler] Job 90: locked_item 140430197713232 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:40:49,029: INFO/job_scheduler] Job 90 completing (errored=False, cancelled=False)
[2017-05-10 13:40:49,063: INFO/job_scheduler] run_next: 1 runnable jobs of (12 pending, 3 tasked)
[2017-05-10 13:40:49,159: INFO/job_scheduler] Job 91: RunJobThread.run
[2017-05-10 13:40:49,159: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-05-10 13:40:49,184: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e73fc139-0015-4818-92d3-8acd93686243
[2017-05-10 13:40:49,184: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e73fc139-0015-4818-92d3-8acd93686243
[2017-05-10 13:40:49,391: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:40:49,391: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:40:49,391: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:40:49,391: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:40:49,391: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d9720059-d5d6-4b59-a2c9-0b72713844e4
[2017-05-10 13:40:49,391: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d9720059-d5d6-4b59-a2c9-0b72713844e4
[2017-05-10 13:40:49,429: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d9720059-d5d6-4b59-a2c9-0b72713844e4
[2017-05-10 13:40:49,429: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d9720059-d5d6-4b59-a2c9-0b72713844e4
[2017-05-10 13:40:49,431: INFO/job] testfs-OST0001 passed pre-format check, allowing subsequent reformats
[2017-05-10 13:40:49,559: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com format_target {'index': 1, 'backfstype': 'zfs', 'target_name': u'testfs-OST0001', 'mgsnode': ((u'10.14.81.224@tcp0',), (u'10.14.81.225@tcp0',)), 'target_types': 'ost', 'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5', 'device_type': 'zfs', 'failnode': [u'10.14.81.226@tcp0'], 'fsname': u'testfs'}
[2017-05-10 13:40:49,608: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bda04e3b-a3be-4396-8555-068ec6576500
[2017-05-10 13:40:49,608: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bda04e3b-a3be-4396-8555-068ec6576500
[2017-05-10 13:40:49,649: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:49,649: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:49,649: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:49,649: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:49,649: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6d3b36ba-6abd-4c2e-96c9-c0ff3880318f
[2017-05-10 13:40:49,649: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6d3b36ba-6abd-4c2e-96c9-c0ff3880318f
[2017-05-10 13:40:49,689: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6d3b36ba-6abd-4c2e-96c9-c0ff3880318f
[2017-05-10 13:40:49,689: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6d3b36ba-6abd-4c2e-96c9-c0ff3880318f
[2017-05-10 13:40:49,992: INFO/job] Updating mtm volume_nodes for target testfs-MDT0000
[2017-05-10 13:40:50,204: INFO/job_scheduler] Job 87 finished 5 steps successfully
[2017-05-10 13:40:50,275: INFO/job] StatefulObject.set_state testfs-MDT0000 unformatted->formatted (intentional=True) 140430197698704
[2017-05-10 13:40:50,282: INFO/job] Job 87: StateChangeJob complete, setting state formatted on testfs-MDT0000
[2017-05-10 13:40:50,282: INFO/job] Job 87: not_deleted=True
[2017-05-10 13:40:50,291: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:50,291: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:50,291: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:50,291: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:50,293: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e73fc139-0015-4818-92d3-8acd93686243
[2017-05-10 13:40:50,293: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e73fc139-0015-4818-92d3-8acd93686243
[2017-05-10 13:40:50,297: INFO/job_scheduler] Job 87: complete_job: Updating cache
[2017-05-10 13:40:50,299: INFO/job_scheduler] Job 87: locked_item 140430197698704 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:40:50,301: INFO/job_scheduler] Job 87: locked_item 140430197702096 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-10 13:40:50,303: INFO/job_scheduler] Job 87: locked_item 140430197702224 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-10 13:40:50,305: INFO/job_scheduler] Job 87 completing (errored=False, cancelled=False)
[2017-05-10 13:40:50,309: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e73fc139-0015-4818-92d3-8acd93686243
[2017-05-10 13:40:50,309: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e73fc139-0015-4818-92d3-8acd93686243
[2017-05-10 13:40:50,310: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-05-10 13:40:50,331: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 322b79b3-0c61-4684-a039-8c2dfd91631e
[2017-05-10 13:40:50,331: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 322b79b3-0c61-4684-a039-8c2dfd91631e
[2017-05-10 13:40:50,388: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 3 tasked)
[2017-05-10 13:40:51,445: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:51,445: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:51,446: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:51,446: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:51,446: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 322b79b3-0c61-4684-a039-8c2dfd91631e
[2017-05-10 13:40:51,446: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 322b79b3-0c61-4684-a039-8c2dfd91631e
[2017-05-10 13:40:51,449: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 322b79b3-0c61-4684-a039-8c2dfd91631e
[2017-05-10 13:40:51,449: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 322b79b3-0c61-4684-a039-8c2dfd91631e
[2017-05-10 13:40:51,451: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_target_store {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux', 'mount_point': u'/mnt/MGS', 'uuid': u'e903f0d3-6ff8-4196-9b86-1ce2f7cd2cc4', 'backfstype': 'ldiskfs'}
[2017-05-10 13:40:51,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8e25144d-37e1-4457-b262-4e2ac23eae6e
[2017-05-10 13:40:51,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8e25144d-37e1-4457-b262-4e2ac23eae6e
[2017-05-10 13:40:51,574: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:51,574: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:51,574: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:51,574: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:51,574: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8e25144d-37e1-4457-b262-4e2ac23eae6e
[2017-05-10 13:40:51,574: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8e25144d-37e1-4457-b262-4e2ac23eae6e
[2017-05-10 13:40:51,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8e25144d-37e1-4457-b262-4e2ac23eae6e
[2017-05-10 13:40:51,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8e25144d-37e1-4457-b262-4e2ac23eae6e
[2017-05-10 13:40:51,600: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_target_store {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux', 'mount_point': u'/mnt/MGS', 'uuid': u'e903f0d3-6ff8-4196-9b86-1ce2f7cd2cc4', 'backfstype': 'ldiskfs'}
[2017-05-10 13:40:51,614: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 15a475c3-0b6d-4979-90f3-fa9dff2ac551
[2017-05-10 13:40:51,614: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 15a475c3-0b6d-4979-90f3-fa9dff2ac551
[2017-05-10 13:40:51,697: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:51,697: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:51,697: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:51,697: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:51,697: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 15a475c3-0b6d-4979-90f3-fa9dff2ac551
[2017-05-10 13:40:51,697: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 15a475c3-0b6d-4979-90f3-fa9dff2ac551
[2017-05-10 13:40:51,730: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 15a475c3-0b6d-4979-90f3-fa9dff2ac551
[2017-05-10 13:40:51,730: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 15a475c3-0b6d-4979-90f3-fa9dff2ac551
[2017-05-10 13:40:51,737: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'mount_point': u'/mnt/MGS', 'uuid': u'e903f0d3-6ff8-4196-9b86-1ce2f7cd2cc4', 'ha_label': u'MGS_8e7c4d', 'primary': True}
[2017-05-10 13:40:51,757: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9250a730-6713-430b-9671-7c65a8a7d00d
[2017-05-10 13:40:51,757: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9250a730-6713-430b-9671-7c65a8a7d00d
[2017-05-10 13:40:54,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:54,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:54,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:54,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:54,940: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9250a730-6713-430b-9671-7c65a8a7d00d
[2017-05-10 13:40:54,940: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9250a730-6713-430b-9671-7c65a8a7d00d
[2017-05-10 13:40:54,981: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9250a730-6713-430b-9671-7c65a8a7d00d
[2017-05-10 13:40:54,981: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9250a730-6713-430b-9671-7c65a8a7d00d
[2017-05-10 13:40:54,983: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'mount_point': u'/mnt/MGS', 'uuid': u'e903f0d3-6ff8-4196-9b86-1ce2f7cd2cc4', 'ha_label': u'MGS_8e7c4d', 'primary': False}
[2017-05-10 13:40:55,007: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0d200b66-eae5-4816-963a-39d4fc212287
[2017-05-10 13:40:55,007: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0d200b66-eae5-4816-963a-39d4fc212287
[2017-05-10 13:40:56,091: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:56,091: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:56,092: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:56,092: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:56,092: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0d200b66-eae5-4816-963a-39d4fc212287
[2017-05-10 13:40:56,092: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0d200b66-eae5-4816-963a-39d4fc212287
[2017-05-10 13:40:56,122: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0d200b66-eae5-4816-963a-39d4fc212287
[2017-05-10 13:40:56,122: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0d200b66-eae5-4816-963a-39d4fc212287
[2017-05-10 13:40:56,122: INFO/job_scheduler] Job 91 finished 6 steps successfully
[2017-05-10 13:40:56,166: INFO/job] StatefulObject.set_state MGS registered->unmounted (intentional=True) 140430197713232
[2017-05-10 13:40:56,173: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on MGS at severity 30
[2017-05-10 13:40:56,193: INFO/job] Job 91: StateChangeJob complete, setting state unmounted on MGS
[2017-05-10 13:40:56,194: INFO/job] Job 91: not_deleted=True
[2017-05-10 13:40:56,194: INFO/job_scheduler] Job 91: complete_job: Updating cache
[2017-05-10 13:40:56,194: INFO/job_scheduler] Job 91: locked_item 140430197713232 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:40:56,197: INFO/job_scheduler] Job 91 completing (errored=False, cancelled=False)
[2017-05-10 13:40:56,233: INFO/job_scheduler] run_next: 1 runnable jobs of (11 pending, 2 tasked)
[2017-05-10 13:40:56,363: INFO/job_scheduler] Job 92: RunJobThread.run
[2017-05-10 13:40:56,369: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux'}
[2017-05-10 13:40:56,389: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ae9de100-e1b2-4d20-947b-cc5c59fbb22a
[2017-05-10 13:40:56,389: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ae9de100-e1b2-4d20-947b-cc5c59fbb22a
[2017-05-10 13:40:56,399: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux'}
[2017-05-10 13:40:56,429: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5406bb6d-83e5-4a47-b6ba-a25497ebd124
[2017-05-10 13:40:56,429: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5406bb6d-83e5-4a47-b6ba-a25497ebd124
[2017-05-10 13:40:56,432: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux'}
[2017-05-10 13:40:56,488: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 28b9137d-f342-4e69-a91a-f0d2d8f89780
[2017-05-10 13:40:56,488: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 28b9137d-f342-4e69-a91a-f0d2d8f89780
[2017-05-10 13:40:56,492: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:40:56,492: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:40:56,492: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:40:56,492: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:40:56,492: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ae9de100-e1b2-4d20-947b-cc5c59fbb22a
[2017-05-10 13:40:56,492: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ae9de100-e1b2-4d20-947b-cc5c59fbb22a
[2017-05-10 13:40:56,517: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:56,517: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:56,517: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:56,517: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:40:56,517: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5406bb6d-83e5-4a47-b6ba-a25497ebd124
[2017-05-10 13:40:56,517: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5406bb6d-83e5-4a47-b6ba-a25497ebd124
[2017-05-10 13:40:56,530: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ae9de100-e1b2-4d20-947b-cc5c59fbb22a
[2017-05-10 13:40:56,530: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ae9de100-e1b2-4d20-947b-cc5c59fbb22a
[2017-05-10 13:40:56,533: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:40:56,533: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:40:56,534: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:40:56,534: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:40:56,534: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 28b9137d-f342-4e69-a91a-f0d2d8f89780
[2017-05-10 13:40:56,534: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 28b9137d-f342-4e69-a91a-f0d2d8f89780
[2017-05-10 13:40:56,560: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 28b9137d-f342-4e69-a91a-f0d2d8f89780
[2017-05-10 13:40:56,560: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 28b9137d-f342-4e69-a91a-f0d2d8f89780
[2017-05-10 13:40:56,564: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5406bb6d-83e5-4a47-b6ba-a25497ebd124
[2017-05-10 13:40:56,564: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5406bb6d-83e5-4a47-b6ba-a25497ebd124
[2017-05-10 13:40:56,564: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com import_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'pacemaker_ha_operation': False, 'device_type': 'linux', 'validate_importable': True}
[2017-05-10 13:40:56,586: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 560e773d-da0a-4d8c-aa4b-9def7fac52a5
[2017-05-10 13:40:56,586: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 560e773d-da0a-4d8c-aa4b-9def7fac52a5
[2017-05-10 13:40:56,668: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:56,668: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:56,669: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:56,669: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:40:56,669: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 560e773d-da0a-4d8c-aa4b-9def7fac52a5
[2017-05-10 13:40:56,669: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 560e773d-da0a-4d8c-aa4b-9def7fac52a5
[2017-05-10 13:40:56,713: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 560e773d-da0a-4d8c-aa4b-9def7fac52a5
[2017-05-10 13:40:56,713: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 560e773d-da0a-4d8c-aa4b-9def7fac52a5
[2017-05-10 13:40:56,713: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_target {'ha_label': u'MGS_8e7c4d'}
[2017-05-10 13:40:56,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1abc3942-55bf-46f0-bc1a-8c7725f0c6d3
[2017-05-10 13:40:56,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1abc3942-55bf-46f0-bc1a-8c7725f0c6d3
[2017-05-10 13:40:57,285: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:40:57,285: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:40:57,286: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:40:57,286: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:40:57,286: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc af1735b8-a566-492e-9608-8d38f46b7bd8
[2017-05-10 13:40:57,286: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc af1735b8-a566-492e-9608-8d38f46b7bd8
[2017-05-10 13:40:57,297: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc af1735b8-a566-492e-9608-8d38f46b7bd8
[2017-05-10 13:40:57,297: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc af1735b8-a566-492e-9608-8d38f46b7bd8
[2017-05-10 13:40:57,342: INFO/job] Updating mtm volume_nodes for target testfs-OST0000
[2017-05-10 13:40:58,708: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 3 tasked)
[2017-05-10 13:40:58,748: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:40:58,748: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:40:58,748: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:40:58,748: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:40:58,749: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bda04e3b-a3be-4396-8555-068ec6576500
[2017-05-10 13:40:58,749: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bda04e3b-a3be-4396-8555-068ec6576500
[2017-05-10 13:40:58,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bda04e3b-a3be-4396-8555-068ec6576500
[2017-05-10 13:40:58,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bda04e3b-a3be-4396-8555-068ec6576500
[2017-05-10 13:40:58,818: INFO/job] Updating mtm volume_nodes for target testfs-OST0001
[2017-05-10 13:41:00,528: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 3 tasked)
[2017-05-10 13:41:02,875: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:41:02,875: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:41:02,875: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:41:02,875: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:41:02,876: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1abc3942-55bf-46f0-bc1a-8c7725f0c6d3
[2017-05-10 13:41:02,876: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1abc3942-55bf-46f0-bc1a-8c7725f0c6d3
[2017-05-10 13:41:02,921: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1abc3942-55bf-46f0-bc1a-8c7725f0c6d3
[2017-05-10 13:41:02,921: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1abc3942-55bf-46f0-bc1a-8c7725f0c6d3
[2017-05-10 13:41:02,921: INFO/job_scheduler] Job 92 finished 1 steps successfully
[2017-05-10 13:41:03,008: INFO/job] StatefulObject.set_state MGS unmounted->mounted (intentional=True) 140430197713232
[2017-05-10 13:41:03,031: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 20
[2017-05-10 13:41:03,040: INFO/job] Job 92: StateChangeJob complete, setting state mounted on MGS
[2017-05-10 13:41:03,040: INFO/job] Job 92: not_deleted=True
[2017-05-10 13:41:03,040: INFO/job_scheduler] Job 92: complete_job: Updating cache
[2017-05-10 13:41:03,040: INFO/job_scheduler] Job 92: locked_item 140430197713232 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:41:03,044: INFO/job_scheduler] Job 92 completing (errored=False, cancelled=False)
[2017-05-10 13:41:03,193: INFO/job_scheduler] run_next: 1 runnable jobs of (10 pending, 2 tasked)
[2017-05-10 13:41:03,319: INFO/job_scheduler] Job 93: RunJobThread.run
[2017-05-10 13:41:03,321: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com register_target {'device_path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', 'mount_point': u'/mnt/testfs-MDT0000', 'backfstype': 'ldiskfs'}
[2017-05-10 13:41:03,356: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1486c64a-3007-4551-b9e9-22f1d858ba45
[2017-05-10 13:41:03,356: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1486c64a-3007-4551-b9e9-22f1d858ba45
[2017-05-10 13:41:07,631: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:07,631: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:07,632: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:07,632: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:07,632: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1486c64a-3007-4551-b9e9-22f1d858ba45
[2017-05-10 13:41:07,632: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1486c64a-3007-4551-b9e9-22f1d858ba45
[2017-05-10 13:41:07,672: INFO/job_scheduler] run_next: 0 runnable jobs of (9 pending, 3 tasked)
[2017-05-10 13:41:07,685: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1486c64a-3007-4551-b9e9-22f1d858ba45
[2017-05-10 13:41:07,685: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1486c64a-3007-4551-b9e9-22f1d858ba45
[2017-05-10 13:41:07,687: INFO/job_scheduler] Job 93 finished 2 steps successfully
[2017-05-10 13:41:07,793: INFO/job] StatefulObject.set_state testfs-MDT0000 formatted->registered (intentional=True) 140430197698704
[2017-05-10 13:41:07,798: INFO/job] Job 93: StateChangeJob complete, setting state registered on testfs-MDT0000
[2017-05-10 13:41:07,798: INFO/job] Job 93: not_deleted=True
[2017-05-10 13:41:07,798: INFO/job_scheduler] Job 93: complete_job: Updating cache
[2017-05-10 13:41:07,798: INFO/job_scheduler] Job 93: locked_item 140430197698704 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:41:07,801: INFO/job_scheduler] Job 93 completing (errored=False, cancelled=False)
[2017-05-10 13:41:07,845: INFO/job_scheduler] run_next: 1 runnable jobs of (9 pending, 2 tasked)
[2017-05-10 13:41:07,924: INFO/job_scheduler] Job 94: RunJobThread.run
[2017-05-10 13:41:07,925: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-05-10 13:41:07,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7996098e-9216-466a-a39e-bcd7e2dc4655
[2017-05-10 13:41:07,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7996098e-9216-466a-a39e-bcd7e2dc4655
[2017-05-10 13:41:09,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:09,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:09,090: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:09,090: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:09,091: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7996098e-9216-466a-a39e-bcd7e2dc4655
[2017-05-10 13:41:09,091: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7996098e-9216-466a-a39e-bcd7e2dc4655
[2017-05-10 13:41:09,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7996098e-9216-466a-a39e-bcd7e2dc4655
[2017-05-10 13:41:09,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7996098e-9216-466a-a39e-bcd7e2dc4655
[2017-05-10 13:41:09,116: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-05-10 13:41:09,144: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f3f1928c-05c9-40fc-995e-7f1a22916fa3
[2017-05-10 13:41:09,144: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f3f1928c-05c9-40fc-995e-7f1a22916fa3
[2017-05-10 13:41:09,540: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 3 tasked)
[2017-05-10 13:41:10,263: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:41:10,263: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:41:10,264: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:41:10,264: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:41:10,264: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f3f1928c-05c9-40fc-995e-7f1a22916fa3
[2017-05-10 13:41:10,264: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f3f1928c-05c9-40fc-995e-7f1a22916fa3
[2017-05-10 13:41:10,308: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f3f1928c-05c9-40fc-995e-7f1a22916fa3
[2017-05-10 13:41:10,308: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f3f1928c-05c9-40fc-995e-7f1a22916fa3
[2017-05-10 13:41:10,310: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_target_store {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', 'device_type': 'linux', 'mount_point': u'/mnt/testfs-MDT0000', 'uuid': u'e4158786-3ffd-4c1c-9a7d-99e762123ce1', 'backfstype': 'ldiskfs'}
[2017-05-10 13:41:10,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7a652418-f06d-4012-a1a5-64fee735d205
[2017-05-10 13:41:10,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7a652418-f06d-4012-a1a5-64fee735d205
[2017-05-10 13:41:10,438: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:10,438: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:10,438: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:10,438: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:10,439: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7a652418-f06d-4012-a1a5-64fee735d205
[2017-05-10 13:41:10,439: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7a652418-f06d-4012-a1a5-64fee735d205
[2017-05-10 13:41:10,486: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7a652418-f06d-4012-a1a5-64fee735d205
[2017-05-10 13:41:10,486: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7a652418-f06d-4012-a1a5-64fee735d205
[2017-05-10 13:41:10,487: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_target_store {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', 'device_type': 'linux', 'mount_point': u'/mnt/testfs-MDT0000', 'uuid': u'e4158786-3ffd-4c1c-9a7d-99e762123ce1', 'backfstype': 'ldiskfs'}
[2017-05-10 13:41:10,504: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f5c4cd7f-e4cf-427b-a026-dcd70550a8de
[2017-05-10 13:41:10,504: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f5c4cd7f-e4cf-427b-a026-dcd70550a8de
[2017-05-10 13:41:10,557: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:41:10,557: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:41:10,557: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:41:10,557: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:41:10,558: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f5c4cd7f-e4cf-427b-a026-dcd70550a8de
[2017-05-10 13:41:10,558: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f5c4cd7f-e4cf-427b-a026-dcd70550a8de
[2017-05-10 13:41:10,571: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f5c4cd7f-e4cf-427b-a026-dcd70550a8de
[2017-05-10 13:41:10,571: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f5c4cd7f-e4cf-427b-a026-dcd70550a8de
[2017-05-10 13:41:10,573: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', 'mount_point': u'/mnt/testfs-MDT0000', 'uuid': u'e4158786-3ffd-4c1c-9a7d-99e762123ce1', 'ha_label': u'testfs-MDT0000_0770c2', 'primary': True}
[2017-05-10 13:41:10,592: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 667f4873-9180-409d-b35a-4fbf00a70116
[2017-05-10 13:41:10,592: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 667f4873-9180-409d-b35a-4fbf00a70116
[2017-05-10 13:41:13,727: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:13,727: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:13,727: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:13,727: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:13,727: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 667f4873-9180-409d-b35a-4fbf00a70116
[2017-05-10 13:41:13,727: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 667f4873-9180-409d-b35a-4fbf00a70116
[2017-05-10 13:41:13,758: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 667f4873-9180-409d-b35a-4fbf00a70116
[2017-05-10 13:41:13,758: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 667f4873-9180-409d-b35a-4fbf00a70116
[2017-05-10 13:41:13,760: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', 'mount_point': u'/mnt/testfs-MDT0000', 'uuid': u'e4158786-3ffd-4c1c-9a7d-99e762123ce1', 'ha_label': u'testfs-MDT0000_0770c2', 'primary': False}
[2017-05-10 13:41:13,775: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0e97bf8d-4895-4051-9bc6-8e2ff8f8170b
[2017-05-10 13:41:13,775: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0e97bf8d-4895-4051-9bc6-8e2ff8f8170b
[2017-05-10 13:41:14,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:41:14,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:41:14,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:41:14,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:41:14,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0e97bf8d-4895-4051-9bc6-8e2ff8f8170b
[2017-05-10 13:41:14,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0e97bf8d-4895-4051-9bc6-8e2ff8f8170b
[2017-05-10 13:41:14,892: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0e97bf8d-4895-4051-9bc6-8e2ff8f8170b
[2017-05-10 13:41:14,892: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0e97bf8d-4895-4051-9bc6-8e2ff8f8170b
[2017-05-10 13:41:14,892: INFO/job_scheduler] Job 94 finished 6 steps successfully
[2017-05-10 13:41:14,927: INFO/job] StatefulObject.set_state testfs-MDT0000 registered->unmounted (intentional=True) 140430197698704
[2017-05-10 13:41:14,933: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on testfs-MDT0000 at severity 30
[2017-05-10 13:41:14,936: INFO/job] Job 94: StateChangeJob complete, setting state unmounted on testfs-MDT0000
[2017-05-10 13:41:14,937: INFO/job] Job 94: not_deleted=True
[2017-05-10 13:41:14,937: INFO/job_scheduler] Job 94: complete_job: Updating cache
[2017-05-10 13:41:14,937: INFO/job_scheduler] Job 94: locked_item 140430197698704 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:41:14,952: INFO/job_scheduler] Job 94 completing (errored=False, cancelled=False)
[2017-05-10 13:41:14,984: INFO/job_scheduler] run_next: 1 runnable jobs of (8 pending, 2 tasked)
[2017-05-10 13:41:15,075: INFO/job_scheduler] Job 95: RunJobThread.run
[2017-05-10 13:41:15,123: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', 'device_type': 'linux'}
[2017-05-10 13:41:15,167: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', 'device_type': 'linux'}
[2017-05-10 13:41:15,167: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', 'device_type': 'linux'}
[2017-05-10 13:41:15,172: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 948d76e9-9fdb-4d4a-9399-efb7f87159a7
[2017-05-10 13:41:15,172: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 948d76e9-9fdb-4d4a-9399-efb7f87159a7
[2017-05-10 13:41:15,186: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c7ffc82e-26ce-4682-92c3-6d917d86daae
[2017-05-10 13:41:15,186: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c7ffc82e-26ce-4682-92c3-6d917d86daae
[2017-05-10 13:41:15,217: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7dea211f-c6fb-430a-a5c2-85c6d30622eb
[2017-05-10 13:41:15,217: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7dea211f-c6fb-430a-a5c2-85c6d30622eb
[2017-05-10 13:41:15,285: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:41:15,285: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:41:15,285: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:41:15,285: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:41:15,285: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 948d76e9-9fdb-4d4a-9399-efb7f87159a7
[2017-05-10 13:41:15,285: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 948d76e9-9fdb-4d4a-9399-efb7f87159a7
[2017-05-10 13:41:15,307: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 948d76e9-9fdb-4d4a-9399-efb7f87159a7
[2017-05-10 13:41:15,307: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 948d76e9-9fdb-4d4a-9399-efb7f87159a7
[2017-05-10 13:41:15,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:41:15,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:41:15,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:41:15,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:41:15,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c7ffc82e-26ce-4682-92c3-6d917d86daae
[2017-05-10 13:41:15,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c7ffc82e-26ce-4682-92c3-6d917d86daae
[2017-05-10 13:41:15,318: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:41:15,318: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:41:15,318: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:41:15,318: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:41:15,318: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7dea211f-c6fb-430a-a5c2-85c6d30622eb
[2017-05-10 13:41:15,318: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7dea211f-c6fb-430a-a5c2-85c6d30622eb
[2017-05-10 13:41:15,319: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c7ffc82e-26ce-4682-92c3-6d917d86daae
[2017-05-10 13:41:15,319: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c7ffc82e-26ce-4682-92c3-6d917d86daae
[2017-05-10 13:41:15,343: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7dea211f-c6fb-430a-a5c2-85c6d30622eb
[2017-05-10 13:41:15,343: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7dea211f-c6fb-430a-a5c2-85c6d30622eb
[2017-05-10 13:41:15,343: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com import_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', 'pacemaker_ha_operation': False, 'device_type': 'linux', 'validate_importable': True}
[2017-05-10 13:41:15,392: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 889db712-ee8a-41af-ad6e-5443260ae65f
[2017-05-10 13:41:15,392: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 889db712-ee8a-41af-ad6e-5443260ae65f
[2017-05-10 13:41:15,456: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:15,456: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:15,457: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:15,457: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:15,457: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 889db712-ee8a-41af-ad6e-5443260ae65f
[2017-05-10 13:41:15,457: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 889db712-ee8a-41af-ad6e-5443260ae65f
[2017-05-10 13:41:15,465: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 889db712-ee8a-41af-ad6e-5443260ae65f
[2017-05-10 13:41:15,465: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 889db712-ee8a-41af-ad6e-5443260ae65f
[2017-05-10 13:41:15,465: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_target {'ha_label': u'testfs-MDT0000_0770c2'}
[2017-05-10 13:41:15,505: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 66fc4940-ef73-4249-922b-76002a626a42
[2017-05-10 13:41:15,505: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 66fc4940-ef73-4249-922b-76002a626a42
[2017-05-10 13:41:18,720: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 3 tasked)
[2017-05-10 13:41:18,738: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 3 tasked)
[2017-05-10 13:41:20,581: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 3 tasked)
[2017-05-10 13:41:20,605: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 3 tasked)
[2017-05-10 13:41:21,607: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:21,607: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:21,607: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:21,607: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:21,608: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 66fc4940-ef73-4249-922b-76002a626a42
[2017-05-10 13:41:21,608: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 66fc4940-ef73-4249-922b-76002a626a42
[2017-05-10 13:41:21,629: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 66fc4940-ef73-4249-922b-76002a626a42
[2017-05-10 13:41:21,629: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 66fc4940-ef73-4249-922b-76002a626a42
[2017-05-10 13:41:21,632: INFO/job_scheduler] Job 95 finished 1 steps successfully
[2017-05-10 13:41:21,914: INFO/job] StatefulObject.set_state testfs-MDT0000 unmounted->mounted (intentional=True) 140430197698704
[2017-05-10 13:41:21,951: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm6.lotus.hpdd.lab.intel.com at severity 20
[2017-05-10 13:41:21,958: INFO/job] Job 95: StateChangeJob complete, setting state mounted on testfs-MDT0000
[2017-05-10 13:41:21,958: INFO/job] Job 95: not_deleted=True
[2017-05-10 13:41:21,959: INFO/job_scheduler] Job 95: complete_job: Updating cache
[2017-05-10 13:41:21,959: INFO/job_scheduler] Job 95: locked_item 140430197698704 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:41:21,962: INFO/job_scheduler] Job 95 completing (errored=False, cancelled=False)
[2017-05-10 13:41:21,997: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 2 tasked)
[2017-05-10 13:41:28,346: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:41:28,346: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 13:41:28,364: INFO/command_plan] add_jobs: created Job 103 (Trigger plugin poll for linux plugins)
[2017-05-10 13:41:28,381: INFO/job_scheduler] run_next: 1 runnable jobs of (8 pending, 2 tasked)
[2017-05-10 13:41:28,408: INFO/job_scheduler] Job 103: RunJobThread.run
[2017-05-10 13:41:28,432: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:41:28,459: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:41:28,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ae352db1-b611-4e9e-82a7-58f1149d9ac6
[2017-05-10 13:41:28,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ae352db1-b611-4e9e-82a7-58f1149d9ac6
[2017-05-10 13:41:28,487: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:41:28,499: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5171572d-347e-4dc7-8226-26475ab08d3f
[2017-05-10 13:41:28,499: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5171572d-347e-4dc7-8226-26475ab08d3f
[2017-05-10 13:41:28,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c8f83ff0-e086-47da-b075-7a0cdef46b61
[2017-05-10 13:41:28,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c8f83ff0-e086-47da-b075-7a0cdef46b61
[2017-05-10 13:41:28,637: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:28,637: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:28,638: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:28,638: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:28,638: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5171572d-347e-4dc7-8226-26475ab08d3f
[2017-05-10 13:41:28,638: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5171572d-347e-4dc7-8226-26475ab08d3f
[2017-05-10 13:41:28,643: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:41:28,643: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:41:28,643: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:41:28,643: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:41:28,643: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ae352db1-b611-4e9e-82a7-58f1149d9ac6
[2017-05-10 13:41:28,643: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ae352db1-b611-4e9e-82a7-58f1149d9ac6
[2017-05-10 13:41:28,644: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ae352db1-b611-4e9e-82a7-58f1149d9ac6
[2017-05-10 13:41:28,644: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ae352db1-b611-4e9e-82a7-58f1149d9ac6
[2017-05-10 13:41:28,645: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:41:28,647: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:41:28,647: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:41:28,647: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:41:28,647: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:41:28,647: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c8f83ff0-e086-47da-b075-7a0cdef46b61
[2017-05-10 13:41:28,647: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c8f83ff0-e086-47da-b075-7a0cdef46b61
[2017-05-10 13:41:28,651: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c8f83ff0-e086-47da-b075-7a0cdef46b61
[2017-05-10 13:41:28,651: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c8f83ff0-e086-47da-b075-7a0cdef46b61
[2017-05-10 13:41:28,651: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:41:28,679: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5171572d-347e-4dc7-8226-26475ab08d3f
[2017-05-10 13:41:28,679: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5171572d-347e-4dc7-8226-26475ab08d3f
[2017-05-10 13:41:28,679: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:41:28,679: INFO/job_scheduler] Job 103 finished 1 steps successfully
[2017-05-10 13:41:28,737: INFO/job_scheduler] Job 103: complete_job: Updating cache
[2017-05-10 13:41:28,737: INFO/job_scheduler] Job 103 completing (errored=False, cancelled=False)
[2017-05-10 13:41:28,822: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 2 tasked)
[2017-05-10 13:41:28,994: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 2 tasked)
[2017-05-10 13:41:29,033: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 2 tasked)
[2017-05-10 13:41:47,682: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:41:47,683: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 13:41:47,729: INFO/command_plan] add_jobs: created Job 104 (Trigger plugin poll for linux plugins)
[2017-05-10 13:41:47,774: INFO/job_scheduler] run_next: 1 runnable jobs of (8 pending, 2 tasked)
[2017-05-10 13:41:47,814: INFO/job_scheduler] Job 104: RunJobThread.run
[2017-05-10 13:41:47,851: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:41:47,875: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:41:47,887: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:41:47,941: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f9166a19-656c-48bd-b75b-3962d995e3f8
[2017-05-10 13:41:47,941: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f9166a19-656c-48bd-b75b-3962d995e3f8
[2017-05-10 13:41:47,968: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 72ee3556-7de3-4484-8db2-f6bcdc3f7662
[2017-05-10 13:41:47,968: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 72ee3556-7de3-4484-8db2-f6bcdc3f7662
[2017-05-10 13:41:47,993: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e019d6fc-0277-40d0-920b-24ddf1a0c2ca
[2017-05-10 13:41:47,993: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e019d6fc-0277-40d0-920b-24ddf1a0c2ca
[2017-05-10 13:41:48,044: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:41:48,044: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:41:48,044: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:41:48,044: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:41:48,044: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f9166a19-656c-48bd-b75b-3962d995e3f8
[2017-05-10 13:41:48,044: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f9166a19-656c-48bd-b75b-3962d995e3f8
[2017-05-10 13:41:48,047: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:48,047: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:48,047: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:48,047: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:41:48,047: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 72ee3556-7de3-4484-8db2-f6bcdc3f7662
[2017-05-10 13:41:48,047: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 72ee3556-7de3-4484-8db2-f6bcdc3f7662
[2017-05-10 13:41:48,050: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 72ee3556-7de3-4484-8db2-f6bcdc3f7662
[2017-05-10 13:41:48,050: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 72ee3556-7de3-4484-8db2-f6bcdc3f7662
[2017-05-10 13:41:48,050: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:41:48,066: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:41:48,066: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:41:48,066: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:41:48,066: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:41:48,067: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e019d6fc-0277-40d0-920b-24ddf1a0c2ca
[2017-05-10 13:41:48,067: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e019d6fc-0277-40d0-920b-24ddf1a0c2ca
[2017-05-10 13:41:48,068: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f9166a19-656c-48bd-b75b-3962d995e3f8
[2017-05-10 13:41:48,068: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f9166a19-656c-48bd-b75b-3962d995e3f8
[2017-05-10 13:41:48,068: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:41:48,114: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e019d6fc-0277-40d0-920b-24ddf1a0c2ca
[2017-05-10 13:41:48,114: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e019d6fc-0277-40d0-920b-24ddf1a0c2ca
[2017-05-10 13:41:48,114: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:41:48,114: INFO/job_scheduler] Job 104 finished 1 steps successfully
[2017-05-10 13:41:48,132: INFO/job_scheduler] Job 104: complete_job: Updating cache
[2017-05-10 13:41:48,132: INFO/job_scheduler] Job 104 completing (errored=False, cancelled=False)
[2017-05-10 13:41:48,165: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 2 tasked)
[2017-05-10 13:41:48,867: INFO/job_scheduler] Job 86 finished 5 steps successfully
[2017-05-10 13:41:48,925: INFO/job] StatefulObject.set_state testfs-OST0000 unformatted->formatted (intentional=True) 140430197702160
[2017-05-10 13:41:48,933: INFO/job] Job 86: StateChangeJob complete, setting state formatted on testfs-OST0000
[2017-05-10 13:41:48,933: INFO/job] Job 86: not_deleted=True
[2017-05-10 13:41:48,951: INFO/job_scheduler] Job 86: complete_job: Updating cache
[2017-05-10 13:41:48,951: INFO/job_scheduler] Job 86: locked_item 140430197702160 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:41:48,957: INFO/job_scheduler] Job 86: locked_item 140429660219792 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-10 13:41:48,960: INFO/job_scheduler] Job 86: locked_item 140430197056336 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-10 13:41:48,964: INFO/job_scheduler] Job 86 completing (errored=False, cancelled=False)
[2017-05-10 13:41:49,041: INFO/job_scheduler] run_next: 1 runnable jobs of (7 pending, 1 tasked)
[2017-05-10 13:41:49,128: INFO/job_scheduler] Job 96: RunJobThread.run
[2017-05-10 13:41:49,129: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com register_target {'device_path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/testfs-OST0000', 'mount_point': u'/mnt/testfs-OST0000', 'backfstype': 'zfs'}
[2017-05-10 13:41:49,157: INFO/job_scheduler] Job 89 finished 5 steps successfully
[2017-05-10 13:41:49,182: INFO/job] StatefulObject.set_state testfs-OST0001 unformatted->formatted (intentional=True) 140430197700880
[2017-05-10 13:41:49,186: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 71f45899-1ddb-4a80-a87e-c0d8d48e6dd8
[2017-05-10 13:41:49,186: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 71f45899-1ddb-4a80-a87e-c0d8d48e6dd8
[2017-05-10 13:41:49,194: INFO/job] Job 89: StateChangeJob complete, setting state formatted on testfs-OST0001
[2017-05-10 13:41:49,194: INFO/job] Job 89: not_deleted=True
[2017-05-10 13:41:49,214: INFO/job_scheduler] Job 89: complete_job: Updating cache
[2017-05-10 13:41:49,218: INFO/job_scheduler] Job 89: locked_item 140430197700880 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:41:49,222: INFO/job_scheduler] Job 89: locked_item 140430198049488 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-10 13:41:49,225: INFO/job_scheduler] Job 89: locked_item 140430198048592 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-05-10 13:41:49,230: INFO/job_scheduler] Job 89 completing (errored=False, cancelled=False)
[2017-05-10 13:41:49,341: INFO/job_scheduler] run_next: 1 runnable jobs of (6 pending, 1 tasked)
[2017-05-10 13:41:49,422: INFO/job_scheduler] Job 97: RunJobThread.run
[2017-05-10 13:41:49,423: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com register_target {'device_path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/testfs-OST0001', 'mount_point': u'/mnt/testfs-OST0001', 'backfstype': 'zfs'}
[2017-05-10 13:41:49,463: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 76489a61-0e46-4343-8e2d-b3bae4f56fb7
[2017-05-10 13:41:49,463: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 76489a61-0e46-4343-8e2d-b3bae4f56fb7
[2017-05-10 13:41:59,334: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:41:59,334: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:41:59,335: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:41:59,335: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:41:59,335: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 71f45899-1ddb-4a80-a87e-c0d8d48e6dd8
[2017-05-10 13:41:59,335: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 71f45899-1ddb-4a80-a87e-c0d8d48e6dd8
[2017-05-10 13:41:59,362: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 71f45899-1ddb-4a80-a87e-c0d8d48e6dd8
[2017-05-10 13:41:59,362: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 71f45899-1ddb-4a80-a87e-c0d8d48e6dd8
[2017-05-10 13:41:59,367: INFO/job_scheduler] Job 96 finished 2 steps successfully
[2017-05-10 13:41:59,591: INFO/job] StatefulObject.set_state testfs-OST0000 formatted->registered (intentional=True) 140430197702160
[2017-05-10 13:41:59,603: INFO/job] Job 96: StateChangeJob complete, setting state registered on testfs-OST0000
[2017-05-10 13:41:59,603: INFO/job] Job 96: not_deleted=True
[2017-05-10 13:41:59,603: INFO/job_scheduler] Job 96: complete_job: Updating cache
[2017-05-10 13:41:59,604: INFO/job_scheduler] Job 96: locked_item 140430197702160 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:41:59,610: INFO/job_scheduler] Job 96 completing (errored=False, cancelled=False)
[2017-05-10 13:41:59,658: INFO/job_scheduler] run_next: 1 runnable jobs of (5 pending, 1 tasked)
[2017-05-10 13:41:59,926: INFO/job_scheduler] Job 98: RunJobThread.run
[2017-05-10 13:41:59,926: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-05-10 13:41:59,969: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3b2a9263-fa9f-4303-9e2d-32ff368d3e5a
[2017-05-10 13:41:59,969: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3b2a9263-fa9f-4303-9e2d-32ff368d3e5a
[2017-05-10 13:42:00,581: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:00,581: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:00,581: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:00,581: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:00,581: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 76489a61-0e46-4343-8e2d-b3bae4f56fb7
[2017-05-10 13:42:00,581: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 76489a61-0e46-4343-8e2d-b3bae4f56fb7
[2017-05-10 13:42:00,604: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 76489a61-0e46-4343-8e2d-b3bae4f56fb7
[2017-05-10 13:42:00,604: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 76489a61-0e46-4343-8e2d-b3bae4f56fb7
[2017-05-10 13:42:00,606: INFO/job_scheduler] Job 97 finished 2 steps successfully
[2017-05-10 13:42:00,845: INFO/job] StatefulObject.set_state testfs-OST0001 formatted->registered (intentional=True) 140430197700880
[2017-05-10 13:42:00,852: INFO/job] Job 97: StateChangeJob complete, setting state registered on testfs-OST0001
[2017-05-10 13:42:00,852: INFO/job] Job 97: not_deleted=True
[2017-05-10 13:42:00,852: INFO/job_scheduler] Job 97: complete_job: Updating cache
[2017-05-10 13:42:00,852: INFO/job_scheduler] Job 97: locked_item 140430197700880 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:42:00,856: INFO/job_scheduler] Job 97 completing (errored=False, cancelled=False)
[2017-05-10 13:42:00,923: INFO/job_scheduler] run_next: 1 runnable jobs of (4 pending, 1 tasked)
[2017-05-10 13:42:01,081: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:01,081: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:01,081: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:01,081: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:01,081: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3b2a9263-fa9f-4303-9e2d-32ff368d3e5a
[2017-05-10 13:42:01,081: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3b2a9263-fa9f-4303-9e2d-32ff368d3e5a
[2017-05-10 13:42:01,093: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3b2a9263-fa9f-4303-9e2d-32ff368d3e5a
[2017-05-10 13:42:01,093: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3b2a9263-fa9f-4303-9e2d-32ff368d3e5a
[2017-05-10 13:42:01,094: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-05-10 13:42:01,104: INFO/job_scheduler] Job 99: RunJobThread.run
[2017-05-10 13:42:01,104: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-05-10 13:42:01,137: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b6a4f03f-9fca-4846-8019-64c39c8d3012
[2017-05-10 13:42:01,137: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b6a4f03f-9fca-4846-8019-64c39c8d3012
[2017-05-10 13:42:01,197: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a35cce3d-24ee-4db8-9dc0-e61be1b8063f
[2017-05-10 13:42:01,197: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a35cce3d-24ee-4db8-9dc0-e61be1b8063f
[2017-05-10 13:42:02,232: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:02,232: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:02,232: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:02,232: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:02,232: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b6a4f03f-9fca-4846-8019-64c39c8d3012
[2017-05-10 13:42:02,232: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b6a4f03f-9fca-4846-8019-64c39c8d3012
[2017-05-10 13:42:02,251: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:02,251: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:02,251: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:02,251: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:02,251: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a35cce3d-24ee-4db8-9dc0-e61be1b8063f
[2017-05-10 13:42:02,251: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a35cce3d-24ee-4db8-9dc0-e61be1b8063f
[2017-05-10 13:42:02,252: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b6a4f03f-9fca-4846-8019-64c39c8d3012
[2017-05-10 13:42:02,252: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b6a4f03f-9fca-4846-8019-64c39c8d3012
[2017-05-10 13:42:02,254: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_target_store {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/testfs-OST0000', 'device_type': 'zfs', 'mount_point': u'/mnt/testfs-OST0000', 'uuid': u'13921496978265162699', 'backfstype': 'zfs'}
[2017-05-10 13:42:02,263: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a35cce3d-24ee-4db8-9dc0-e61be1b8063f
[2017-05-10 13:42:02,263: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a35cce3d-24ee-4db8-9dc0-e61be1b8063f
[2017-05-10 13:42:02,264: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-05-10 13:42:02,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ab702742-4999-49a6-bd7b-cc597511bd4a
[2017-05-10 13:42:02,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ab702742-4999-49a6-bd7b-cc597511bd4a
[2017-05-10 13:42:02,291: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c5415898-307c-4974-898d-ac2a6d4736ff
[2017-05-10 13:42:02,291: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c5415898-307c-4974-898d-ac2a6d4736ff
[2017-05-10 13:42:02,376: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:02,376: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:02,376: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:02,376: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:02,376: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ab702742-4999-49a6-bd7b-cc597511bd4a
[2017-05-10 13:42:02,376: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ab702742-4999-49a6-bd7b-cc597511bd4a
[2017-05-10 13:42:02,398: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ab702742-4999-49a6-bd7b-cc597511bd4a
[2017-05-10 13:42:02,398: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ab702742-4999-49a6-bd7b-cc597511bd4a
[2017-05-10 13:42:02,400: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_target_store {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/testfs-OST0000', 'device_type': 'zfs', 'mount_point': u'/mnt/testfs-OST0000', 'uuid': u'13921496978265162699', 'backfstype': 'zfs'}
[2017-05-10 13:42:02,419: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 65b848ff-4466-4242-9ac1-6b824f5c596a
[2017-05-10 13:42:02,419: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 65b848ff-4466-4242-9ac1-6b824f5c596a
[2017-05-10 13:42:02,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:02,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:02,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:02,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:02,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 65b848ff-4466-4242-9ac1-6b824f5c596a
[2017-05-10 13:42:02,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 65b848ff-4466-4242-9ac1-6b824f5c596a
[2017-05-10 13:42:02,534: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 65b848ff-4466-4242-9ac1-6b824f5c596a
[2017-05-10 13:42:02,534: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 65b848ff-4466-4242-9ac1-6b824f5c596a
[2017-05-10 13:42:02,535: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/testfs-OST0000', 'mount_point': u'/mnt/testfs-OST0000', 'uuid': u'13921496978265162699', 'ha_label': u'testfs-OST0000_b8032c', 'primary': True}
[2017-05-10 13:42:02,554: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d168f9be-4a60-499a-a4ea-3e68e4b7049f
[2017-05-10 13:42:02,554: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d168f9be-4a60-499a-a4ea-3e68e4b7049f
[2017-05-10 13:42:03,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:03,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:03,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:03,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:03,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c5415898-307c-4974-898d-ac2a6d4736ff
[2017-05-10 13:42:03,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c5415898-307c-4974-898d-ac2a6d4736ff
[2017-05-10 13:42:03,383: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c5415898-307c-4974-898d-ac2a6d4736ff
[2017-05-10 13:42:03,383: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c5415898-307c-4974-898d-ac2a6d4736ff
[2017-05-10 13:42:03,385: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_target_store {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/testfs-OST0001', 'device_type': 'zfs', 'mount_point': u'/mnt/testfs-OST0001', 'uuid': u'1961503762134256363', 'backfstype': 'zfs'}
[2017-05-10 13:42:03,432: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2de6fe95-9649-4e39-8012-b7b6df48c08b
[2017-05-10 13:42:03,432: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2de6fe95-9649-4e39-8012-b7b6df48c08b
[2017-05-10 13:42:03,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:03,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:03,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:03,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:03,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2de6fe95-9649-4e39-8012-b7b6df48c08b
[2017-05-10 13:42:03,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2de6fe95-9649-4e39-8012-b7b6df48c08b
[2017-05-10 13:42:03,550: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2de6fe95-9649-4e39-8012-b7b6df48c08b
[2017-05-10 13:42:03,550: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2de6fe95-9649-4e39-8012-b7b6df48c08b
[2017-05-10 13:42:03,552: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_target_store {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/testfs-OST0001', 'device_type': 'zfs', 'mount_point': u'/mnt/testfs-OST0001', 'uuid': u'1961503762134256363', 'backfstype': 'zfs'}
[2017-05-10 13:42:03,565: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4667add6-f204-45ff-9d53-d7fbc30f6d0e
[2017-05-10 13:42:03,565: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4667add6-f204-45ff-9d53-d7fbc30f6d0e
[2017-05-10 13:42:03,683: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:03,683: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:03,683: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:03,683: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:03,683: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4667add6-f204-45ff-9d53-d7fbc30f6d0e
[2017-05-10 13:42:03,683: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4667add6-f204-45ff-9d53-d7fbc30f6d0e
[2017-05-10 13:42:03,729: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4667add6-f204-45ff-9d53-d7fbc30f6d0e
[2017-05-10 13:42:03,729: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4667add6-f204-45ff-9d53-d7fbc30f6d0e
[2017-05-10 13:42:03,732: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/testfs-OST0001', 'mount_point': u'/mnt/testfs-OST0001', 'uuid': u'1961503762134256363', 'ha_label': u'testfs-OST0001_865fe0', 'primary': True}
[2017-05-10 13:42:03,748: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f9383091-7519-46ec-a7ab-17bb6105f1f7
[2017-05-10 13:42:03,748: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f9383091-7519-46ec-a7ab-17bb6105f1f7
[2017-05-10 13:42:05,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:05,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:05,692: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:05,692: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:05,692: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d168f9be-4a60-499a-a4ea-3e68e4b7049f
[2017-05-10 13:42:05,692: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d168f9be-4a60-499a-a4ea-3e68e4b7049f
[2017-05-10 13:42:05,730: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d168f9be-4a60-499a-a4ea-3e68e4b7049f
[2017-05-10 13:42:05,730: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d168f9be-4a60-499a-a4ea-3e68e4b7049f
[2017-05-10 13:42:05,732: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/testfs-OST0000', 'mount_point': u'/mnt/testfs-OST0000', 'uuid': u'13921496978265162699', 'ha_label': u'testfs-OST0000_b8032c', 'primary': False}
[2017-05-10 13:42:05,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 52a1147f-dfb3-40d6-a135-069a0bbc9132
[2017-05-10 13:42:05,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 52a1147f-dfb3-40d6-a135-069a0bbc9132
[2017-05-10 13:42:06,825: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:06,825: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:06,825: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:06,825: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:06,825: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 52a1147f-dfb3-40d6-a135-069a0bbc9132
[2017-05-10 13:42:06,825: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 52a1147f-dfb3-40d6-a135-069a0bbc9132
[2017-05-10 13:42:06,871: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 52a1147f-dfb3-40d6-a135-069a0bbc9132
[2017-05-10 13:42:06,871: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 52a1147f-dfb3-40d6-a135-069a0bbc9132
[2017-05-10 13:42:06,871: INFO/job_scheduler] Job 98 finished 6 steps successfully
[2017-05-10 13:42:06,899: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:06,899: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:06,899: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:06,899: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:06,899: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f9383091-7519-46ec-a7ab-17bb6105f1f7
[2017-05-10 13:42:06,899: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f9383091-7519-46ec-a7ab-17bb6105f1f7
[2017-05-10 13:42:06,921: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f9383091-7519-46ec-a7ab-17bb6105f1f7
[2017-05-10 13:42:06,921: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f9383091-7519-46ec-a7ab-17bb6105f1f7
[2017-05-10 13:42:06,923: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/testfs-OST0001', 'mount_point': u'/mnt/testfs-OST0001', 'uuid': u'1961503762134256363', 'ha_label': u'testfs-OST0001_865fe0', 'primary': False}
[2017-05-10 13:42:06,937: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 838e0640-0f02-4229-b0c5-d488f9fad503
[2017-05-10 13:42:06,937: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 838e0640-0f02-4229-b0c5-d488f9fad503
[2017-05-10 13:42:06,961: INFO/job] StatefulObject.set_state testfs-OST0000 registered->unmounted (intentional=True) 140430197702160
[2017-05-10 13:42:06,967: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on testfs-OST0000 at severity 30
[2017-05-10 13:42:06,975: INFO/job] Job 98: StateChangeJob complete, setting state unmounted on testfs-OST0000
[2017-05-10 13:42:06,976: INFO/job] Job 98: not_deleted=True
[2017-05-10 13:42:06,977: INFO/job_scheduler] Job 98: complete_job: Updating cache
[2017-05-10 13:42:06,977: INFO/job_scheduler] Job 98: locked_item 140430197702160 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:42:06,987: INFO/job_scheduler] Job 98 completing (errored=False, cancelled=False)
[2017-05-10 13:42:07,115: INFO/job_scheduler] run_next: 1 runnable jobs of (3 pending, 1 tasked)
[2017-05-10 13:42:07,212: INFO/job_scheduler] Job 101: RunJobThread.run
[2017-05-10 13:42:07,229: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/testfs-OST0000', 'device_type': 'zfs'}
[2017-05-10 13:42:07,254: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 33e5edf9-8ef1-4a5d-9132-d921e5c1645c
[2017-05-10 13:42:07,254: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 33e5edf9-8ef1-4a5d-9132-d921e5c1645c
[2017-05-10 13:42:08,025: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:08,025: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:08,025: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:08,025: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:08,025: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 838e0640-0f02-4229-b0c5-d488f9fad503
[2017-05-10 13:42:08,025: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 838e0640-0f02-4229-b0c5-d488f9fad503
[2017-05-10 13:42:08,055: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 838e0640-0f02-4229-b0c5-d488f9fad503
[2017-05-10 13:42:08,055: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 838e0640-0f02-4229-b0c5-d488f9fad503
[2017-05-10 13:42:08,055: INFO/job_scheduler] Job 99 finished 6 steps successfully
[2017-05-10 13:42:08,140: INFO/job] StatefulObject.set_state testfs-OST0001 registered->unmounted (intentional=True) 140430197700880
[2017-05-10 13:42:08,144: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on testfs-OST0001 at severity 30
[2017-05-10 13:42:08,153: INFO/job] Job 99: StateChangeJob complete, setting state unmounted on testfs-OST0001
[2017-05-10 13:42:08,154: INFO/job] Job 99: not_deleted=True
[2017-05-10 13:42:08,155: INFO/job_scheduler] Job 99: complete_job: Updating cache
[2017-05-10 13:42:08,156: INFO/job_scheduler] Job 99: locked_item 140430197700880 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:42:08,177: INFO/job_scheduler] Job 99 completing (errored=False, cancelled=False)
[2017-05-10 13:42:08,240: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 1 tasked)
[2017-05-10 13:42:08,371: INFO/job_scheduler] Job 100: RunJobThread.run
[2017-05-10 13:42:08,373: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/testfs-OST0001', 'device_type': 'zfs'}
[2017-05-10 13:42:08,388: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5d64bf64-ea3e-4ebf-8b5e-7b992af03540
[2017-05-10 13:42:08,388: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5d64bf64-ea3e-4ebf-8b5e-7b992af03540
[2017-05-10 13:42:08,394: INFO/job_scheduler] notify: Dropping update of 8 (testfs-OST0001) because it has been updated since
[2017-05-10 13:42:08,394: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 13:42:08,396: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 13:42:09,393: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:09,393: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:09,393: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:09,393: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:09,393: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 33e5edf9-8ef1-4a5d-9132-d921e5c1645c
[2017-05-10 13:42:09,393: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 33e5edf9-8ef1-4a5d-9132-d921e5c1645c
[2017-05-10 13:42:09,424: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 33e5edf9-8ef1-4a5d-9132-d921e5c1645c
[2017-05-10 13:42:09,424: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 33e5edf9-8ef1-4a5d-9132-d921e5c1645c
[2017-05-10 13:42:09,425: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com import_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/testfs-OST0000', 'pacemaker_ha_operation': False, 'device_type': 'zfs', 'validate_importable': True}
[2017-05-10 13:42:09,437: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 50bb8c03-6c44-4621-b11f-04009da6d089
[2017-05-10 13:42:09,437: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 50bb8c03-6c44-4621-b11f-04009da6d089
[2017-05-10 13:42:11,483: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 13:42:11,499: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 13:42:14,914: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:14,914: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:14,915: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:14,915: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:14,915: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5d64bf64-ea3e-4ebf-8b5e-7b992af03540
[2017-05-10 13:42:14,915: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5d64bf64-ea3e-4ebf-8b5e-7b992af03540
[2017-05-10 13:42:14,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5d64bf64-ea3e-4ebf-8b5e-7b992af03540
[2017-05-10 13:42:14,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5d64bf64-ea3e-4ebf-8b5e-7b992af03540
[2017-05-10 13:42:14,923: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com import_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/testfs-OST0001', 'pacemaker_ha_operation': False, 'device_type': 'zfs', 'validate_importable': True}
[2017-05-10 13:42:14,938: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6c85a39f-fc3f-4ade-b525-02724c63acb9
[2017-05-10 13:42:14,938: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6c85a39f-fc3f-4ade-b525-02724c63acb9
[2017-05-10 13:42:17,438: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:42:17,438: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 13:42:17,473: INFO/command_plan] add_jobs: created Job 105 (Trigger plugin poll for linux plugins)
[2017-05-10 13:42:17,518: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 2 tasked)
[2017-05-10 13:42:17,545: INFO/job_scheduler] Job 105: RunJobThread.run
[2017-05-10 13:42:17,568: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:42:17,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc df48f5a6-53d8-4e8d-901e-045db44cc86a
[2017-05-10 13:42:17,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc df48f5a6-53d8-4e8d-901e-045db44cc86a
[2017-05-10 13:42:17,616: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:42:17,640: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:42:17,648: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a15f047e-fa4f-404b-9d9e-a8ebca68eb61
[2017-05-10 13:42:17,648: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a15f047e-fa4f-404b-9d9e-a8ebca68eb61
[2017-05-10 13:42:17,650: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:17,650: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:17,650: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:17,650: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:17,651: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 50bb8c03-6c44-4621-b11f-04009da6d089
[2017-05-10 13:42:17,651: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 50bb8c03-6c44-4621-b11f-04009da6d089
[2017-05-10 13:42:17,679: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 50bb8c03-6c44-4621-b11f-04009da6d089
[2017-05-10 13:42:17,679: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 50bb8c03-6c44-4621-b11f-04009da6d089
[2017-05-10 13:42:17,680: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_target {'ha_label': u'testfs-OST0000_b8032c'}
[2017-05-10 13:42:17,687: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 779a10da-bf11-44ac-927c-09ba68dd1144
[2017-05-10 13:42:17,687: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 779a10da-bf11-44ac-927c-09ba68dd1144
[2017-05-10 13:42:17,706: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9df8af85-fe30-49a9-92f1-9eb39f95cab8
[2017-05-10 13:42:17,706: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9df8af85-fe30-49a9-92f1-9eb39f95cab8
[2017-05-10 13:42:17,721: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:42:17,721: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:42:17,721: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:42:17,721: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:42:17,721: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc df48f5a6-53d8-4e8d-901e-045db44cc86a
[2017-05-10 13:42:17,721: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc df48f5a6-53d8-4e8d-901e-045db44cc86a
[2017-05-10 13:42:17,722: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:17,722: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:17,722: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:17,722: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:17,722: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a15f047e-fa4f-404b-9d9e-a8ebca68eb61
[2017-05-10 13:42:17,722: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a15f047e-fa4f-404b-9d9e-a8ebca68eb61
[2017-05-10 13:42:17,724: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc df48f5a6-53d8-4e8d-901e-045db44cc86a
[2017-05-10 13:42:17,724: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc df48f5a6-53d8-4e8d-901e-045db44cc86a
[2017-05-10 13:42:17,724: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:42:17,744: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a15f047e-fa4f-404b-9d9e-a8ebca68eb61
[2017-05-10 13:42:17,744: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a15f047e-fa4f-404b-9d9e-a8ebca68eb61
[2017-05-10 13:42:17,745: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:42:17,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:17,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:17,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:17,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:17,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 779a10da-bf11-44ac-927c-09ba68dd1144
[2017-05-10 13:42:17,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 779a10da-bf11-44ac-927c-09ba68dd1144
[2017-05-10 13:42:17,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 779a10da-bf11-44ac-927c-09ba68dd1144
[2017-05-10 13:42:17,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 779a10da-bf11-44ac-927c-09ba68dd1144
[2017-05-10 13:42:17,784: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:42:17,784: INFO/job_scheduler] Job 105 finished 1 steps successfully
[2017-05-10 13:42:18,149: INFO/job_scheduler] Job 105: complete_job: Updating cache
[2017-05-10 13:42:18,149: INFO/job_scheduler] Job 105 completing (errored=False, cancelled=False)
[2017-05-10 13:42:18,289: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 13:42:22,067: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:22,067: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:22,067: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:22,067: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:22,068: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6c85a39f-fc3f-4ade-b525-02724c63acb9
[2017-05-10 13:42:22,068: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6c85a39f-fc3f-4ade-b525-02724c63acb9
[2017-05-10 13:42:22,108: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6c85a39f-fc3f-4ade-b525-02724c63acb9
[2017-05-10 13:42:22,108: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6c85a39f-fc3f-4ade-b525-02724c63acb9
[2017-05-10 13:42:22,109: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_target {'ha_label': u'testfs-OST0001_865fe0'}
[2017-05-10 13:42:22,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8352af3f-a81a-4681-a2f6-e357982d3415
[2017-05-10 13:42:22,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8352af3f-a81a-4681-a2f6-e357982d3415
[2017-05-10 13:42:23,964: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:42:23,965: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 13:42:24,002: INFO/command_plan] add_jobs: created Job 106 (Trigger plugin poll for linux plugins)
[2017-05-10 13:42:24,081: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 2 tasked)
[2017-05-10 13:42:24,101: INFO/job_scheduler] Job 106: RunJobThread.run
[2017-05-10 13:42:24,115: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:42:24,152: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4bedd352-37f6-4d8a-8455-740399190710
[2017-05-10 13:42:24,152: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4bedd352-37f6-4d8a-8455-740399190710
[2017-05-10 13:42:24,182: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:42:24,231: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ba3dcf9e-098b-42a6-8e39-132eeaf1551d
[2017-05-10 13:42:24,231: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ba3dcf9e-098b-42a6-8e39-132eeaf1551d
[2017-05-10 13:42:24,231: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:42:24,231: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:42:24,232: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:42:24,232: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:42:24,232: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4bedd352-37f6-4d8a-8455-740399190710
[2017-05-10 13:42:24,232: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4bedd352-37f6-4d8a-8455-740399190710
[2017-05-10 13:42:24,233: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-05-10 13:42:24,242: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4bedd352-37f6-4d8a-8455-740399190710
[2017-05-10 13:42:24,242: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4bedd352-37f6-4d8a-8455-740399190710
[2017-05-10 13:42:24,243: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:42:24,310: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 59765f05-d428-42c0-90e5-73d7a2ee36d7
[2017-05-10 13:42:24,310: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 59765f05-d428-42c0-90e5-73d7a2ee36d7
[2017-05-10 13:42:24,311: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:24,311: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:24,312: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:24,312: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:24,312: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ba3dcf9e-098b-42a6-8e39-132eeaf1551d
[2017-05-10 13:42:24,312: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ba3dcf9e-098b-42a6-8e39-132eeaf1551d
[2017-05-10 13:42:24,363: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ba3dcf9e-098b-42a6-8e39-132eeaf1551d
[2017-05-10 13:42:24,363: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ba3dcf9e-098b-42a6-8e39-132eeaf1551d
[2017-05-10 13:42:24,363: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:42:24,415: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:24,415: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:24,416: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:24,416: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:24,416: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 59765f05-d428-42c0-90e5-73d7a2ee36d7
[2017-05-10 13:42:24,416: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 59765f05-d428-42c0-90e5-73d7a2ee36d7
[2017-05-10 13:42:24,430: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 59765f05-d428-42c0-90e5-73d7a2ee36d7
[2017-05-10 13:42:24,430: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 59765f05-d428-42c0-90e5-73d7a2ee36d7
[2017-05-10 13:42:24,430: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-05-10 13:42:24,430: INFO/job_scheduler] Job 106 finished 1 steps successfully
[2017-05-10 13:42:24,558: INFO/job_scheduler] Job 106: complete_job: Updating cache
[2017-05-10 13:42:24,559: INFO/job_scheduler] Job 106 completing (errored=False, cancelled=False)
[2017-05-10 13:42:24,659: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 13:42:29,850: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:29,850: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:29,851: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:29,851: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:42:29,851: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9df8af85-fe30-49a9-92f1-9eb39f95cab8
[2017-05-10 13:42:29,851: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9df8af85-fe30-49a9-92f1-9eb39f95cab8
[2017-05-10 13:42:29,854: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9df8af85-fe30-49a9-92f1-9eb39f95cab8
[2017-05-10 13:42:29,854: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9df8af85-fe30-49a9-92f1-9eb39f95cab8
[2017-05-10 13:42:29,855: INFO/job_scheduler] Job 101 finished 1 steps successfully
[2017-05-10 13:42:30,085: INFO/job] StatefulObject.set_state testfs-OST0000 unmounted->mounted (intentional=True) 140430197702160
[2017-05-10 13:42:30,136: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 20
[2017-05-10 13:42:30,145: INFO/job] Job 101: StateChangeJob complete, setting state mounted on testfs-OST0000
[2017-05-10 13:42:30,145: INFO/job] Job 101: not_deleted=True
[2017-05-10 13:42:30,145: INFO/job_scheduler] Job 101: complete_job: Updating cache
[2017-05-10 13:42:30,145: INFO/job_scheduler] Job 101: locked_item 140430197702160 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:42:30,151: INFO/job_scheduler] Job 101 completing (errored=False, cancelled=False)
[2017-05-10 13:42:30,218: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 13:42:30,527: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 13:42:30,569: INFO/job_scheduler] notify: Dropping update of 7 (testfs-OST0000) because it has been updated since
[2017-05-10 13:42:30,570: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 13:42:33,344: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 13:42:33,368: INFO/job_scheduler] notify: Dropping update of 7 (testfs-OST0000) because it has been updated since
[2017-05-10 13:42:33,369: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 13:42:38,365: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:38,365: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:38,365: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:38,365: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:42:38,365: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8352af3f-a81a-4681-a2f6-e357982d3415
[2017-05-10 13:42:38,365: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8352af3f-a81a-4681-a2f6-e357982d3415
[2017-05-10 13:42:38,390: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 13:42:38,412: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 13:42:38,413: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8352af3f-a81a-4681-a2f6-e357982d3415
[2017-05-10 13:42:38,413: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8352af3f-a81a-4681-a2f6-e357982d3415
[2017-05-10 13:42:38,416: INFO/job_scheduler] Job 100 finished 1 steps successfully
[2017-05-10 13:42:38,576: INFO/job] StatefulObject.set_state testfs-OST0001 unmounted->mounted (intentional=True) 140430197700880
[2017-05-10 13:42:38,604: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm8.lotus.hpdd.lab.intel.com at severity 20
[2017-05-10 13:42:38,612: INFO/job] Job 100: StateChangeJob complete, setting state mounted on testfs-OST0001
[2017-05-10 13:42:38,620: INFO/job] Job 100: not_deleted=True
[2017-05-10 13:42:38,624: INFO/job_scheduler] Job 100: complete_job: Updating cache
[2017-05-10 13:42:38,625: INFO/job_scheduler] Job 100: locked_item 140430197700880 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:42:38,635: INFO/job_scheduler] Job 100 completing (errored=False, cancelled=False)
[2017-05-10 13:42:38,676: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 13:42:38,758: INFO/job_scheduler] Job 102: open DB connection during completion
[2017-05-10 13:42:38,768: INFO/job] StatefulObject.set_state testfs unavailable->available (intentional=True) 140429660218704
[2017-05-10 13:42:38,771: INFO/job] Job 102: StateChangeJob complete, setting state available on testfs
[2017-05-10 13:42:38,772: INFO/job] Job 102: not_deleted=True
[2017-05-10 13:42:38,772: INFO/job_scheduler] Job 102: complete_job: Updating cache
[2017-05-10 13:42:38,772: INFO/job_scheduler] Job 102: locked_item 140429660218704 <class 'chroma_core.models.filesystem.ManagedFilesystem'> False True
[2017-05-10 13:42:38,778: INFO/job_scheduler] Job 102 completing (errored=False, cancelled=False)
[2017-05-10 13:42:39,052: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:42:41,615: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:42:41,649: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:43:28,347: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:43:28,380: INFO/job_scheduler] _notify: Updating .state of item 5 (MGS) from mounted to unmounted
[2017-05-10 13:43:28,380: INFO/job] StatefulObject.set_state MGS mounted->unmounted (intentional=False) 140430198318672
[2017-05-10 13:43:28,393: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on MGS at severity 40
[2017-05-10 13:43:28,393: INFO/job_scheduler] _notify: Updating .active_mount_id of item 5 (MGS) from 9 to None
[2017-05-10 13:43:28,416: INFO/job_scheduler] _notify: Set active_mount_id=None on MGS (ManagedTarget-5) and saved
[2017-05-10 13:43:28,428: INFO/job_scheduler] _notify: Updating .state of item 2 (testfs) from available to unavailable
[2017-05-10 13:43:28,428: INFO/job] StatefulObject.set_state testfs available->unavailable (intentional=False) 140430198364944
[2017-05-10 13:43:28,440: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:43:30,031: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:43:30,063: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:43:38,377: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:43:38,544: INFO/job_scheduler] _notify: Updating .state of item 5 (MGS) from unmounted to mounted
[2017-05-10 13:43:38,544: INFO/job] StatefulObject.set_state MGS unmounted->mounted (intentional=False) 140430198328528
[2017-05-10 13:43:38,593: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 20
[2017-05-10 13:43:38,598: INFO/job_scheduler] _notify: Updating .active_mount_id of item 5 (MGS) from None to 9
[2017-05-10 13:43:38,601: INFO/job_scheduler] _notify: Set active_mount_id=9 on MGS (ManagedTarget-5) and saved
[2017-05-10 13:43:38,612: INFO/job_scheduler] _notify: Updating .state of item 2 (testfs) from unavailable to available
[2017-05-10 13:43:38,612: INFO/job] StatefulObject.set_state testfs unavailable->available (intentional=False) 140430198435600
[2017-05-10 13:43:38,627: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:43:40,336: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:43:40,345: INFO/command_plan] set_state: <class 'chroma_core.models.target.ManagedTarget'>-5 transitioning from mounted to unmounted
[2017-05-10 13:43:40,391: INFO/command_plan] Created command 30 (Stop target MGS) with 1 jobs
[2017-05-10 13:43:40,471: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 13:43:40,553: INFO/job_scheduler] Job 107: RunJobThread.run
[2017-05-10 13:43:40,555: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_target {'ha_label': u'MGS_8e7c4d'}
[2017-05-10 13:43:40,604: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4fe7d674-176b-47ff-9124-c15eb8f458d5
[2017-05-10 13:43:40,604: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4fe7d674-176b-47ff-9124-c15eb8f458d5
[2017-05-10 13:43:41,153: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 13:43:41,174: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 13:43:50,162: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 13:43:50,240: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 13:43:50,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:43:50,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:43:50,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:43:50,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:43:50,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4fe7d674-176b-47ff-9124-c15eb8f458d5
[2017-05-10 13:43:50,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4fe7d674-176b-47ff-9124-c15eb8f458d5
[2017-05-10 13:43:50,831: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4fe7d674-176b-47ff-9124-c15eb8f458d5
[2017-05-10 13:43:50,831: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4fe7d674-176b-47ff-9124-c15eb8f458d5
[2017-05-10 13:43:50,832: INFO/job_scheduler] Job 107 finished 1 steps successfully
[2017-05-10 13:43:50,956: INFO/job] StatefulObject.set_state MGS mounted->unmounted (intentional=True) 140430198366160
[2017-05-10 13:43:50,969: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on MGS at severity 30
[2017-05-10 13:43:50,992: INFO/job] Job 107: StateChangeJob complete, setting state unmounted on MGS
[2017-05-10 13:43:50,992: INFO/job] Job 107: not_deleted=True
[2017-05-10 13:43:50,993: INFO/job_scheduler] Job 107: complete_job: Updating cache
[2017-05-10 13:43:50,993: INFO/job_scheduler] Job 107: locked_item 140430198366160 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:43:50,997: INFO/job_scheduler] Job 107 completing (errored=False, cancelled=False)
[2017-05-10 13:43:51,012: INFO/job_scheduler] _notify: Updating .state of item 2 (testfs) from available to unavailable
[2017-05-10 13:43:51,012: INFO/job] StatefulObject.set_state testfs available->unavailable (intentional=False) 140430198381200
[2017-05-10 13:43:51,174: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:43:53,443: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:43:53,446: INFO/command_plan] set_state: <class 'chroma_core.models.filesystem.ManagedFilesystem'>-2 transitioning from unavailable to stopped
[2017-05-10 13:43:53,528: INFO/command_plan] Created command 31 (Stop file system testfs) with 4 jobs
[2017-05-10 13:43:53,551: INFO/job_scheduler] run_next: 3 runnable jobs of (4 pending, 0 tasked)
[2017-05-10 13:43:53,661: INFO/job_scheduler] Job 108: RunJobThread.run
[2017-05-10 13:43:53,663: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com stop_target {'ha_label': u'testfs-MDT0000_0770c2'}
[2017-05-10 13:43:53,692: INFO/job_scheduler] Job 109: RunJobThread.run
[2017-05-10 13:43:53,694: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com stop_target {'ha_label': u'testfs-OST0001_865fe0'}
[2017-05-10 13:43:53,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 30ee56be-cfc1-45f7-bcc5-4dec8079bd7d
[2017-05-10 13:43:53,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 30ee56be-cfc1-45f7-bcc5-4dec8079bd7d
[2017-05-10 13:43:53,740: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e5dbb72f-348e-4fe7-a411-c4f373a3d234
[2017-05-10 13:43:53,740: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e5dbb72f-348e-4fe7-a411-c4f373a3d234
[2017-05-10 13:43:53,742: INFO/job_scheduler] Job 110: RunJobThread.run
[2017-05-10 13:43:53,744: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com stop_target {'ha_label': u'testfs-OST0000_b8032c'}
[2017-05-10 13:43:53,809: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 503bcb2d-76bd-4b44-a856-6d66e59e1da9
[2017-05-10 13:43:53,809: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 503bcb2d-76bd-4b44-a856-6d66e59e1da9
[2017-05-10 13:43:57,804: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:43:57,804: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:43:57,804: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:43:57,804: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:43:57,804: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 30ee56be-cfc1-45f7-bcc5-4dec8079bd7d
[2017-05-10 13:43:57,804: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 30ee56be-cfc1-45f7-bcc5-4dec8079bd7d
[2017-05-10 13:43:57,819: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 30ee56be-cfc1-45f7-bcc5-4dec8079bd7d
[2017-05-10 13:43:57,819: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 30ee56be-cfc1-45f7-bcc5-4dec8079bd7d
[2017-05-10 13:43:57,819: INFO/job_scheduler] Job 108 finished 1 steps successfully
[2017-05-10 13:43:57,909: INFO/job] StatefulObject.set_state testfs-MDT0000 mounted->unmounted (intentional=True) 140430198425424
[2017-05-10 13:43:57,918: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on testfs-MDT0000 at severity 30
[2017-05-10 13:43:57,941: INFO/job] Job 108: StateChangeJob complete, setting state unmounted on testfs-MDT0000
[2017-05-10 13:43:57,941: INFO/job] Job 108: not_deleted=True
[2017-05-10 13:43:57,941: INFO/job_scheduler] Job 108: complete_job: Updating cache
[2017-05-10 13:43:57,941: INFO/job_scheduler] Job 108: locked_item 140430198425424 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:43:57,944: INFO/job_scheduler] Job 108 completing (errored=False, cancelled=False)
[2017-05-10 13:43:57,986: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 13:43:58,385: INFO/job_scheduler] notify: Dropping update of 6 (testfs-MDT0000) because it has been updated since
[2017-05-10 13:43:58,386: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 13:43:58,465: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 13:44:00,168: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 13:44:00,182: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 13:44:05,899: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:44:05,899: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:44:05,899: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:44:05,899: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:44:05,899: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e5dbb72f-348e-4fe7-a411-c4f373a3d234
[2017-05-10 13:44:05,899: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e5dbb72f-348e-4fe7-a411-c4f373a3d234
[2017-05-10 13:44:05,917: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e5dbb72f-348e-4fe7-a411-c4f373a3d234
[2017-05-10 13:44:05,917: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e5dbb72f-348e-4fe7-a411-c4f373a3d234
[2017-05-10 13:44:05,918: INFO/job_scheduler] Job 109 finished 1 steps successfully
[2017-05-10 13:44:06,007: INFO/job] StatefulObject.set_state testfs-OST0001 mounted->unmounted (intentional=True) 140430198284176
[2017-05-10 13:44:06,014: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on testfs-OST0001 at severity 30
[2017-05-10 13:44:06,022: INFO/job] Job 109: StateChangeJob complete, setting state unmounted on testfs-OST0001
[2017-05-10 13:44:06,023: INFO/job] Job 109: not_deleted=True
[2017-05-10 13:44:06,024: INFO/job_scheduler] Job 109: complete_job: Updating cache
[2017-05-10 13:44:06,024: INFO/job_scheduler] Job 109: locked_item 140430198284176 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:44:06,038: INFO/job_scheduler] Job 109 completing (errored=False, cancelled=False)
[2017-05-10 13:44:06,082: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 13:44:08,669: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 13:44:08,688: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 13:44:10,810: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 13:44:10,833: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 13:44:18,705: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 13:44:18,721: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 13:44:20,028: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:44:20,028: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:44:20,028: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:44:20,028: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:44:20,029: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 503bcb2d-76bd-4b44-a856-6d66e59e1da9
[2017-05-10 13:44:20,029: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 503bcb2d-76bd-4b44-a856-6d66e59e1da9
[2017-05-10 13:44:20,044: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 503bcb2d-76bd-4b44-a856-6d66e59e1da9
[2017-05-10 13:44:20,044: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 503bcb2d-76bd-4b44-a856-6d66e59e1da9
[2017-05-10 13:44:20,045: INFO/job_scheduler] Job 110 finished 1 steps successfully
[2017-05-10 13:44:20,426: INFO/job] StatefulObject.set_state testfs-OST0000 mounted->unmounted (intentional=True) 140430198283344
[2017-05-10 13:44:20,431: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on testfs-OST0000 at severity 30
[2017-05-10 13:44:20,440: INFO/job] Job 110: StateChangeJob complete, setting state unmounted on testfs-OST0000
[2017-05-10 13:44:20,441: INFO/job] Job 110: not_deleted=True
[2017-05-10 13:44:20,442: INFO/job_scheduler] Job 110: complete_job: Updating cache
[2017-05-10 13:44:20,442: INFO/job_scheduler] Job 110: locked_item 140430198283344 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:44:20,456: INFO/job_scheduler] Job 110 completing (errored=False, cancelled=False)
[2017-05-10 13:44:20,502: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 13:44:20,614: INFO/job_scheduler] Job 111: open DB connection during completion
[2017-05-10 13:44:20,642: INFO/job] StatefulObject.set_state testfs unavailable->stopped (intentional=True) 140430197979088
[2017-05-10 13:44:20,646: INFO/job] Job 111: StateChangeJob complete, setting state stopped on testfs
[2017-05-10 13:44:20,647: INFO/job] Job 111: not_deleted=True
[2017-05-10 13:44:20,647: INFO/job_scheduler] Job 111: complete_job: Updating cache
[2017-05-10 13:44:20,647: INFO/job_scheduler] Job 111: locked_item 140430197979088 <class 'chroma_core.models.filesystem.ManagedFilesystem'> False True
[2017-05-10 13:44:20,652: INFO/job_scheduler] Job 111 completing (errored=False, cancelled=False)
[2017-05-10 13:44:20,743: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:44:20,892: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:44:20,922: INFO/job_scheduler] notify: Dropping update of 7 (testfs-OST0000) because it has been updated since
[2017-05-10 13:44:20,922: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:44:49,148: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:44:49,154: INFO/command_plan] set_state: <class 'chroma_core.models.lnet_configuration.LNetConfiguration'>-1 transitioning from lnet_down to lnet_up
[2017-05-10 13:44:49,196: INFO/command_plan] Created command 32 (Start the LNet networking layer.) with 1 jobs
[2017-05-10 13:44:49,234: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 13:44:49,260: INFO/job_scheduler] Job 112: RunJobThread.run
[2017-05-10 13:44:49,261: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_lnet {}
[2017-05-10 13:44:49,292: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 995dcccb-a788-41f6-a930-85f3018bd4e8
[2017-05-10 13:44:49,292: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 995dcccb-a788-41f6-a930-85f3018bd4e8
[2017-05-10 13:44:51,372: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:44:51,372: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:44:51,372: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:44:51,372: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:44:51,373: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 995dcccb-a788-41f6-a930-85f3018bd4e8
[2017-05-10 13:44:51,373: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 995dcccb-a788-41f6-a930-85f3018bd4e8
[2017-05-10 13:44:51,407: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 995dcccb-a788-41f6-a930-85f3018bd4e8
[2017-05-10 13:44:51,407: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 995dcccb-a788-41f6-a930-85f3018bd4e8
[2017-05-10 13:44:51,408: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 13:44:51,432: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6665ce02-db37-4888-a009-ad67d311b1c0
[2017-05-10 13:44:51,432: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6665ce02-db37-4888-a009-ad67d311b1c0
[2017-05-10 13:44:52,519: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:44:52,519: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:44:52,520: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:44:52,520: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:44:52,520: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6665ce02-db37-4888-a009-ad67d311b1c0
[2017-05-10 13:44:52,520: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6665ce02-db37-4888-a009-ad67d311b1c0
[2017-05-10 13:44:52,552: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6665ce02-db37-4888-a009-ad67d311b1c0
[2017-05-10 13:44:52,552: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6665ce02-db37-4888-a009-ad67d311b1c0
[2017-05-10 13:44:52,700: INFO/job_scheduler] Job 112 finished 2 steps successfully
[2017-05-10 13:44:52,753: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 140430198049552
[2017-05-10 13:44:52,778: INFO/job] Job 112: StateChangeJob complete, setting state lnet_up on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:44:52,778: INFO/job] Job 112: not_deleted=True
[2017-05-10 13:44:52,778: INFO/job_scheduler] Job 112: complete_job: Updating cache
[2017-05-10 13:44:52,779: INFO/job_scheduler] Job 112: locked_item 140430198049552 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:44:52,800: INFO/job_scheduler] Job 112 completing (errored=False, cancelled=False)
[2017-05-10 13:44:52,942: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:44:55,167: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:44:55,180: INFO/command_plan] set_state: <class 'chroma_core.models.lnet_configuration.LNetConfiguration'>-1 transitioning from lnet_up to lnet_down
[2017-05-10 13:44:55,207: INFO/command_plan] Created command 33 (Shut down the LNet networking layer and stop any targets running on this server.) with 1 jobs
[2017-05-10 13:44:55,233: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 13:44:55,240: INFO/job_scheduler] Job 113: RunJobThread.run
[2017-05-10 13:44:55,241: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_lnet {}
[2017-05-10 13:44:55,278: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 60288c68-b1e1-464e-87a6-e4627f27050e
[2017-05-10 13:44:55,278: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 60288c68-b1e1-464e-87a6-e4627f27050e
[2017-05-10 13:45:08,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:08,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:08,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:08,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:08,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 60288c68-b1e1-464e-87a6-e4627f27050e
[2017-05-10 13:45:08,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 60288c68-b1e1-464e-87a6-e4627f27050e
[2017-05-10 13:45:08,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 60288c68-b1e1-464e-87a6-e4627f27050e
[2017-05-10 13:45:08,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 60288c68-b1e1-464e-87a6-e4627f27050e
[2017-05-10 13:45:08,452: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 13:45:08,469: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b90a9b64-42c2-42c4-ac9f-a651f36d5aa3
[2017-05-10 13:45:08,469: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b90a9b64-42c2-42c4-ac9f-a651f36d5aa3
[2017-05-10 13:45:09,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:09,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:09,551: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:09,551: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:09,551: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b90a9b64-42c2-42c4-ac9f-a651f36d5aa3
[2017-05-10 13:45:09,551: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b90a9b64-42c2-42c4-ac9f-a651f36d5aa3
[2017-05-10 13:45:09,600: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b90a9b64-42c2-42c4-ac9f-a651f36d5aa3
[2017-05-10 13:45:09,600: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b90a9b64-42c2-42c4-ac9f-a651f36d5aa3
[2017-05-10 13:45:09,641: INFO/job_scheduler] Job 113 finished 2 steps successfully
[2017-05-10 13:45:09,684: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_up->lnet_down (intentional=True) 140430198012112
[2017-05-10 13:45:09,691: INFO/job] Job 113: StateChangeJob complete, setting state lnet_down on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:45:09,691: INFO/job] Job 113: not_deleted=True
[2017-05-10 13:45:09,691: INFO/job_scheduler] Job 113: complete_job: Updating cache
[2017-05-10 13:45:09,691: INFO/job_scheduler] Job 113: locked_item 140430198012112 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:45:09,694: INFO/job_scheduler] Job 113 completing (errored=False, cancelled=False)
[2017-05-10 13:45:09,751: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:45:11,916: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:45:11,928: INFO/command_plan] set_state: <class 'chroma_core.models.filesystem.ManagedFilesystem'>-2 transitioning from stopped to available
[2017-05-10 13:45:12,090: INFO/command_plan] Created command 34 (Start file system testfs) with 6 jobs
[2017-05-10 13:45:12,288: INFO/job_scheduler] run_next: 3 runnable jobs of (6 pending, 0 tasked)
[2017-05-10 13:45:12,557: INFO/job_scheduler] Job 114: RunJobThread.run
[2017-05-10 13:45:12,558: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_lnet {}
[2017-05-10 13:45:12,577: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 12f0fa1d-a2f8-47ef-a8e4-dc1246c5f10b
[2017-05-10 13:45:12,577: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 12f0fa1d-a2f8-47ef-a8e4-dc1246c5f10b
[2017-05-10 13:45:12,583: INFO/job_scheduler] Job 115: RunJobThread.run
[2017-05-10 13:45:12,585: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/testfs-OST0001', 'device_type': 'zfs'}
[2017-05-10 13:45:12,594: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/testfs-OST0001', 'device_type': 'zfs'}
[2017-05-10 13:45:12,595: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/testfs-OST0001', 'device_type': 'zfs'}
[2017-05-10 13:45:12,596: INFO/job_scheduler] Job 116: RunJobThread.run
[2017-05-10 13:45:12,598: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/testfs-OST0000', 'device_type': 'zfs'}
[2017-05-10 13:45:12,609: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 333992c0-f807-4df0-a010-670fd9307bf6
[2017-05-10 13:45:12,609: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 333992c0-f807-4df0-a010-670fd9307bf6
[2017-05-10 13:45:12,617: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/testfs-OST0000', 'device_type': 'zfs'}
[2017-05-10 13:45:12,634: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 782f47a8-92bc-4e6f-a52a-23902f1f3f72
[2017-05-10 13:45:12,634: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 782f47a8-92bc-4e6f-a52a-23902f1f3f72
[2017-05-10 13:45:12,663: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 141e91f0-7445-4ebb-ac97-21d54c8b00ff
[2017-05-10 13:45:12,663: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 141e91f0-7445-4ebb-ac97-21d54c8b00ff
[2017-05-10 13:45:12,675: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/testfs-OST0000', 'device_type': 'zfs'}
[2017-05-10 13:45:12,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1a419fb1-88f5-4c07-8303-aaa2922a1599
[2017-05-10 13:45:12,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1a419fb1-88f5-4c07-8303-aaa2922a1599
[2017-05-10 13:45:12,756: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e14bb502-abfc-4dc4-8797-fcfb7a1770d9
[2017-05-10 13:45:12,756: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e14bb502-abfc-4dc4-8797-fcfb7a1770d9
[2017-05-10 13:45:12,793: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b477e865-43da-4957-93dc-9708c8c41aff
[2017-05-10 13:45:12,793: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b477e865-43da-4957-93dc-9708c8c41aff
[2017-05-10 13:45:13,772: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:45:13,772: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:45:13,772: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:45:13,772: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:45:13,772: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 782f47a8-92bc-4e6f-a52a-23902f1f3f72
[2017-05-10 13:45:13,772: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 782f47a8-92bc-4e6f-a52a-23902f1f3f72
[2017-05-10 13:45:13,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 782f47a8-92bc-4e6f-a52a-23902f1f3f72
[2017-05-10 13:45:13,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 782f47a8-92bc-4e6f-a52a-23902f1f3f72
[2017-05-10 13:45:13,839: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:45:13,839: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:45:13,839: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:45:13,839: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:45:13,839: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e14bb502-abfc-4dc4-8797-fcfb7a1770d9
[2017-05-10 13:45:13,839: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e14bb502-abfc-4dc4-8797-fcfb7a1770d9
[2017-05-10 13:45:13,871: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e14bb502-abfc-4dc4-8797-fcfb7a1770d9
[2017-05-10 13:45:13,871: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e14bb502-abfc-4dc4-8797-fcfb7a1770d9
[2017-05-10 13:45:14,705: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:14,705: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:14,705: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:14,705: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:14,705: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 12f0fa1d-a2f8-47ef-a8e4-dc1246c5f10b
[2017-05-10 13:45:14,705: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 12f0fa1d-a2f8-47ef-a8e4-dc1246c5f10b
[2017-05-10 13:45:14,741: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 12f0fa1d-a2f8-47ef-a8e4-dc1246c5f10b
[2017-05-10 13:45:14,741: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 12f0fa1d-a2f8-47ef-a8e4-dc1246c5f10b
[2017-05-10 13:45:14,742: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 13:45:14,758: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e26651a3-5505-423b-a0a6-7c2077585669
[2017-05-10 13:45:14,758: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e26651a3-5505-423b-a0a6-7c2077585669
[2017-05-10 13:45:15,814: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:15,814: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:15,814: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:15,814: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:15,814: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1a419fb1-88f5-4c07-8303-aaa2922a1599
[2017-05-10 13:45:15,814: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1a419fb1-88f5-4c07-8303-aaa2922a1599
[2017-05-10 13:45:15,815: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:15,815: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:15,815: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:15,815: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:15,815: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e26651a3-5505-423b-a0a6-7c2077585669
[2017-05-10 13:45:15,815: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e26651a3-5505-423b-a0a6-7c2077585669
[2017-05-10 13:45:15,824: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1a419fb1-88f5-4c07-8303-aaa2922a1599
[2017-05-10 13:45:15,824: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1a419fb1-88f5-4c07-8303-aaa2922a1599
[2017-05-10 13:45:15,829: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e26651a3-5505-423b-a0a6-7c2077585669
[2017-05-10 13:45:15,829: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e26651a3-5505-423b-a0a6-7c2077585669
[2017-05-10 13:45:15,983: INFO/job_scheduler] Job 114 finished 2 steps successfully
[2017-05-10 13:45:16,003: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 140430198014160
[2017-05-10 13:45:16,011: INFO/job] Job 114: StateChangeJob complete, setting state lnet_up on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:45:16,011: INFO/job] Job 114: not_deleted=True
[2017-05-10 13:45:16,011: INFO/job_scheduler] Job 114: complete_job: Updating cache
[2017-05-10 13:45:16,011: INFO/job_scheduler] Job 114: locked_item 140430198014160 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:45:16,015: INFO/job_scheduler] Job 114 completing (errored=False, cancelled=False)
[2017-05-10 13:45:16,038: INFO/job_scheduler] run_next: 2 runnable jobs of (3 pending, 2 tasked)
[2017-05-10 13:45:16,145: INFO/job_scheduler] Job 117: RunJobThread.run
[2017-05-10 13:45:16,147: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', 'device_type': 'linux'}
[2017-05-10 13:45:16,148: INFO/job_scheduler] Job 118: RunJobThread.run
[2017-05-10 13:45:16,164: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', 'device_type': 'linux'}
[2017-05-10 13:45:16,164: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux'}
[2017-05-10 13:45:16,165: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux'}
[2017-05-10 13:45:16,165: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', 'device_type': 'linux'}
[2017-05-10 13:45:16,166: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux'}
[2017-05-10 13:45:16,169: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 010e9449-72bf-434d-bb2c-39c8b7f9ba62
[2017-05-10 13:45:16,169: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 010e9449-72bf-434d-bb2c-39c8b7f9ba62
[2017-05-10 13:45:16,194: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2ae6592b-b482-4472-9cb8-a8f0c0421425
[2017-05-10 13:45:16,194: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2ae6592b-b482-4472-9cb8-a8f0c0421425
[2017-05-10 13:45:16,208: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f92da374-8ebf-4feb-b2cd-08700695020a
[2017-05-10 13:45:16,208: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f92da374-8ebf-4feb-b2cd-08700695020a
[2017-05-10 13:45:16,227: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 44af272a-4901-4bc4-989b-a704b911a802
[2017-05-10 13:45:16,227: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 44af272a-4901-4bc4-989b-a704b911a802
[2017-05-10 13:45:16,251: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0ab18cf6-595f-4bd0-b68a-ff1bfc7b8570
[2017-05-10 13:45:16,251: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0ab18cf6-595f-4bd0-b68a-ff1bfc7b8570
[2017-05-10 13:45:16,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 88356ca5-f777-48dd-9b5e-8502551baa6d
[2017-05-10 13:45:16,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 88356ca5-f777-48dd-9b5e-8502551baa6d
[2017-05-10 13:45:16,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:45:16,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:45:16,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:45:16,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:45:16,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 010e9449-72bf-434d-bb2c-39c8b7f9ba62
[2017-05-10 13:45:16,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 010e9449-72bf-434d-bb2c-39c8b7f9ba62
[2017-05-10 13:45:16,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:45:16,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:45:16,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:45:16,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:45:16,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f92da374-8ebf-4feb-b2cd-08700695020a
[2017-05-10 13:45:16,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f92da374-8ebf-4feb-b2cd-08700695020a
[2017-05-10 13:45:16,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:16,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:16,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:16,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:16,283: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2ae6592b-b482-4472-9cb8-a8f0c0421425
[2017-05-10 13:45:16,283: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2ae6592b-b482-4472-9cb8-a8f0c0421425
[2017-05-10 13:45:16,293: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2ae6592b-b482-4472-9cb8-a8f0c0421425
[2017-05-10 13:45:16,293: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2ae6592b-b482-4472-9cb8-a8f0c0421425
[2017-05-10 13:45:16,293: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f92da374-8ebf-4feb-b2cd-08700695020a
[2017-05-10 13:45:16,293: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f92da374-8ebf-4feb-b2cd-08700695020a
[2017-05-10 13:45:16,305: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:45:16,305: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:45:16,305: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:45:16,305: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:45:16,305: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 44af272a-4901-4bc4-989b-a704b911a802
[2017-05-10 13:45:16,305: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 44af272a-4901-4bc4-989b-a704b911a802
[2017-05-10 13:45:16,316: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 44af272a-4901-4bc4-989b-a704b911a802
[2017-05-10 13:45:16,316: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 44af272a-4901-4bc4-989b-a704b911a802
[2017-05-10 13:45:16,321: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 010e9449-72bf-434d-bb2c-39c8b7f9ba62
[2017-05-10 13:45:16,321: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 010e9449-72bf-434d-bb2c-39c8b7f9ba62
[2017-05-10 13:45:16,334: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:45:16,334: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:45:16,334: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:45:16,334: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:45:16,334: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 88356ca5-f777-48dd-9b5e-8502551baa6d
[2017-05-10 13:45:16,334: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 88356ca5-f777-48dd-9b5e-8502551baa6d
[2017-05-10 13:45:16,336: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:45:16,336: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:45:16,337: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:45:16,337: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:45:16,337: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0ab18cf6-595f-4bd0-b68a-ff1bfc7b8570
[2017-05-10 13:45:16,337: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0ab18cf6-595f-4bd0-b68a-ff1bfc7b8570
[2017-05-10 13:45:16,349: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 88356ca5-f777-48dd-9b5e-8502551baa6d
[2017-05-10 13:45:16,349: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 88356ca5-f777-48dd-9b5e-8502551baa6d
[2017-05-10 13:45:16,349: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com import_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'pacemaker_ha_operation': False, 'device_type': 'linux', 'validate_importable': True}
[2017-05-10 13:45:16,364: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 67bafdce-834d-4d19-bf7d-878d5e3f2299
[2017-05-10 13:45:16,364: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 67bafdce-834d-4d19-bf7d-878d5e3f2299
[2017-05-10 13:45:16,378: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0ab18cf6-595f-4bd0-b68a-ff1bfc7b8570
[2017-05-10 13:45:16,378: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0ab18cf6-595f-4bd0-b68a-ff1bfc7b8570
[2017-05-10 13:45:16,378: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com import_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', 'pacemaker_ha_operation': False, 'device_type': 'linux', 'validate_importable': True}
[2017-05-10 13:45:16,394: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f5f33c21-1546-4770-93f8-e1948a4c5d5a
[2017-05-10 13:45:16,394: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f5f33c21-1546-4770-93f8-e1948a4c5d5a
[2017-05-10 13:45:16,468: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:16,468: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:16,469: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:16,469: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:16,469: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 67bafdce-834d-4d19-bf7d-878d5e3f2299
[2017-05-10 13:45:16,469: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 67bafdce-834d-4d19-bf7d-878d5e3f2299
[2017-05-10 13:45:16,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:45:16,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:45:16,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:45:16,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:45:16,473: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f5f33c21-1546-4770-93f8-e1948a4c5d5a
[2017-05-10 13:45:16,473: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f5f33c21-1546-4770-93f8-e1948a4c5d5a
[2017-05-10 13:45:16,487: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 67bafdce-834d-4d19-bf7d-878d5e3f2299
[2017-05-10 13:45:16,487: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 67bafdce-834d-4d19-bf7d-878d5e3f2299
[2017-05-10 13:45:16,487: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_target {'ha_label': u'MGS_8e7c4d'}
[2017-05-10 13:45:16,510: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 130029a2-5960-427b-b98b-c332f9ed6b21
[2017-05-10 13:45:16,510: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 130029a2-5960-427b-b98b-c332f9ed6b21
[2017-05-10 13:45:16,519: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f5f33c21-1546-4770-93f8-e1948a4c5d5a
[2017-05-10 13:45:16,519: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f5f33c21-1546-4770-93f8-e1948a4c5d5a
[2017-05-10 13:45:16,519: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_target {'ha_label': u'testfs-MDT0000_0770c2'}
[2017-05-10 13:45:16,561: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 376311b1-aa9b-49f1-93bf-9d4dd6057222
[2017-05-10 13:45:16,561: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 376311b1-aa9b-49f1-93bf-9d4dd6057222
[2017-05-10 13:45:16,724: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:45:16,724: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:45:16,724: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:45:16,724: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:45:16,725: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 333992c0-f807-4df0-a010-670fd9307bf6
[2017-05-10 13:45:16,725: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 333992c0-f807-4df0-a010-670fd9307bf6
[2017-05-10 13:45:16,734: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 333992c0-f807-4df0-a010-670fd9307bf6
[2017-05-10 13:45:16,734: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 333992c0-f807-4df0-a010-670fd9307bf6
[2017-05-10 13:45:16,734: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com import_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/testfs-OST0001', 'pacemaker_ha_operation': False, 'device_type': 'zfs', 'validate_importable': True}
[2017-05-10 13:45:16,749: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3b24f535-7f67-4270-8ff0-64ad0082996e
[2017-05-10 13:45:16,749: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3b24f535-7f67-4270-8ff0-64ad0082996e
[2017-05-10 13:45:16,915: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:16,915: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:16,915: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:16,915: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:16,915: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b477e865-43da-4957-93dc-9708c8c41aff
[2017-05-10 13:45:16,915: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b477e865-43da-4957-93dc-9708c8c41aff
[2017-05-10 13:45:16,917: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b477e865-43da-4957-93dc-9708c8c41aff
[2017-05-10 13:45:16,917: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b477e865-43da-4957-93dc-9708c8c41aff
[2017-05-10 13:45:18,952: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:45:18,952: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:45:18,952: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:45:18,952: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:45:18,952: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 141e91f0-7445-4ebb-ac97-21d54c8b00ff
[2017-05-10 13:45:18,952: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 141e91f0-7445-4ebb-ac97-21d54c8b00ff
[2017-05-10 13:45:18,995: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 141e91f0-7445-4ebb-ac97-21d54c8b00ff
[2017-05-10 13:45:18,995: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 141e91f0-7445-4ebb-ac97-21d54c8b00ff
[2017-05-10 13:45:18,996: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com import_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/testfs-OST0000', 'pacemaker_ha_operation': False, 'device_type': 'zfs', 'validate_importable': True}
[2017-05-10 13:45:19,016: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5bd5415a-f42f-437a-91aa-061fe9c41594
[2017-05-10 13:45:19,016: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5bd5415a-f42f-437a-91aa-061fe9c41594
[2017-05-10 13:45:23,928: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:45:23,928: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:45:23,929: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:45:23,929: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:45:23,929: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3b24f535-7f67-4270-8ff0-64ad0082996e
[2017-05-10 13:45:23,929: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3b24f535-7f67-4270-8ff0-64ad0082996e
[2017-05-10 13:45:23,935: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3b24f535-7f67-4270-8ff0-64ad0082996e
[2017-05-10 13:45:23,935: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3b24f535-7f67-4270-8ff0-64ad0082996e
[2017-05-10 13:45:23,936: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_target {'ha_label': u'testfs-OST0001_865fe0'}
[2017-05-10 13:45:23,951: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 84974700-dda8-432c-ad58-a6949e11a444
[2017-05-10 13:45:23,951: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 84974700-dda8-432c-ad58-a6949e11a444
[2017-05-10 13:45:24,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:24,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:24,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:24,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:45:24,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 130029a2-5960-427b-b98b-c332f9ed6b21
[2017-05-10 13:45:24,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 130029a2-5960-427b-b98b-c332f9ed6b21
[2017-05-10 13:45:24,678: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 130029a2-5960-427b-b98b-c332f9ed6b21
[2017-05-10 13:45:24,678: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 130029a2-5960-427b-b98b-c332f9ed6b21
[2017-05-10 13:45:24,679: INFO/job_scheduler] Job 118 finished 1 steps successfully
[2017-05-10 13:45:24,779: INFO/job] StatefulObject.set_state MGS unmounted->mounted (intentional=True) 140430198125328
[2017-05-10 13:45:24,814: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 20
[2017-05-10 13:45:24,841: INFO/job] Job 118: StateChangeJob complete, setting state mounted on MGS
[2017-05-10 13:45:24,841: INFO/job] Job 118: not_deleted=True
[2017-05-10 13:45:24,841: INFO/job_scheduler] Job 118: complete_job: Updating cache
[2017-05-10 13:45:24,841: INFO/job_scheduler] Job 118: locked_item 140430198125328 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:45:24,856: INFO/job_scheduler] Job 118 completing (errored=False, cancelled=False)
[2017-05-10 13:45:24,879: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 13:45:28,101: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:45:28,101: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:45:28,102: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:45:28,102: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:45:28,102: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5bd5415a-f42f-437a-91aa-061fe9c41594
[2017-05-10 13:45:28,102: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5bd5415a-f42f-437a-91aa-061fe9c41594
[2017-05-10 13:45:28,143: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5bd5415a-f42f-437a-91aa-061fe9c41594
[2017-05-10 13:45:28,143: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5bd5415a-f42f-437a-91aa-061fe9c41594
[2017-05-10 13:45:28,144: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_target {'ha_label': u'testfs-OST0000_b8032c'}
[2017-05-10 13:45:28,183: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 729e7afe-2927-4798-aba8-c6444b60d627
[2017-05-10 13:45:28,183: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 729e7afe-2927-4798-aba8-c6444b60d627
[2017-05-10 13:45:28,745: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:45:28,745: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:45:28,745: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:45:28,745: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:45:28,745: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 376311b1-aa9b-49f1-93bf-9d4dd6057222
[2017-05-10 13:45:28,745: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 376311b1-aa9b-49f1-93bf-9d4dd6057222
[2017-05-10 13:45:28,757: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 376311b1-aa9b-49f1-93bf-9d4dd6057222
[2017-05-10 13:45:28,757: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 376311b1-aa9b-49f1-93bf-9d4dd6057222
[2017-05-10 13:45:28,758: INFO/job_scheduler] Job 117 finished 1 steps successfully
[2017-05-10 13:45:28,912: INFO/job] StatefulObject.set_state testfs-MDT0000 unmounted->mounted (intentional=True) 140430198285456
[2017-05-10 13:45:28,951: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm6.lotus.hpdd.lab.intel.com at severity 20
[2017-05-10 13:45:28,972: INFO/job] Job 117: StateChangeJob complete, setting state mounted on testfs-MDT0000
[2017-05-10 13:45:28,972: INFO/job] Job 117: not_deleted=True
[2017-05-10 13:45:28,974: INFO/job_scheduler] Job 117: complete_job: Updating cache
[2017-05-10 13:45:28,976: INFO/job_scheduler] Job 117: locked_item 140430198285456 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:45:28,983: INFO/job_scheduler] Job 117 completing (errored=False, cancelled=False)
[2017-05-10 13:45:29,104: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 13:45:29,680: INFO/job_scheduler] notify: Dropping update of 6 (testfs-MDT0000) because it has been updated since
[2017-05-10 13:45:29,681: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 13:45:29,723: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 13:45:30,465: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 13:45:30,483: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 13:45:36,125: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:45:36,125: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:45:36,126: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:45:36,126: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:45:36,126: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 84974700-dda8-432c-ad58-a6949e11a444
[2017-05-10 13:45:36,126: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 84974700-dda8-432c-ad58-a6949e11a444
[2017-05-10 13:45:36,149: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 84974700-dda8-432c-ad58-a6949e11a444
[2017-05-10 13:45:36,149: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 84974700-dda8-432c-ad58-a6949e11a444
[2017-05-10 13:45:36,150: INFO/job_scheduler] Job 115 finished 1 steps successfully
[2017-05-10 13:45:36,346: INFO/job] StatefulObject.set_state testfs-OST0001 unmounted->mounted (intentional=True) 140430197797264
[2017-05-10 13:45:36,386: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm8.lotus.hpdd.lab.intel.com at severity 20
[2017-05-10 13:45:36,430: INFO/job] Job 115: StateChangeJob complete, setting state mounted on testfs-OST0001
[2017-05-10 13:45:36,431: INFO/job] Job 115: not_deleted=True
[2017-05-10 13:45:36,431: INFO/job_scheduler] Job 115: complete_job: Updating cache
[2017-05-10 13:45:36,431: INFO/job_scheduler] Job 115: locked_item 140430197797264 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:45:36,435: INFO/job_scheduler] Job 115 completing (errored=False, cancelled=False)
[2017-05-10 13:45:36,489: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 13:45:38,955: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 13:45:39,029: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 13:45:44,547: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:45:44,547: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:45:44,548: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:45:44,548: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:45:44,548: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 729e7afe-2927-4798-aba8-c6444b60d627
[2017-05-10 13:45:44,548: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 729e7afe-2927-4798-aba8-c6444b60d627
[2017-05-10 13:45:44,579: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 729e7afe-2927-4798-aba8-c6444b60d627
[2017-05-10 13:45:44,579: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 729e7afe-2927-4798-aba8-c6444b60d627
[2017-05-10 13:45:44,580: INFO/job_scheduler] Job 116 finished 1 steps successfully
[2017-05-10 13:45:44,753: INFO/job] StatefulObject.set_state testfs-OST0000 unmounted->mounted (intentional=True) 140430198099536
[2017-05-10 13:45:44,787: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 20
[2017-05-10 13:45:44,807: INFO/job] Job 116: StateChangeJob complete, setting state mounted on testfs-OST0000
[2017-05-10 13:45:44,807: INFO/job] Job 116: not_deleted=True
[2017-05-10 13:45:44,807: INFO/job_scheduler] Job 116: complete_job: Updating cache
[2017-05-10 13:45:44,807: INFO/job_scheduler] Job 116: locked_item 140430198099536 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:45:44,812: INFO/job_scheduler] Job 116 completing (errored=False, cancelled=False)
[2017-05-10 13:45:44,861: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 13:45:45,033: INFO/job_scheduler] Job 119: open DB connection during completion
[2017-05-10 13:45:45,037: INFO/job] StatefulObject.set_state testfs stopped->available (intentional=True) 140430198511696
[2017-05-10 13:45:45,043: INFO/job] Job 119: StateChangeJob complete, setting state available on testfs
[2017-05-10 13:45:45,043: INFO/job] Job 119: not_deleted=True
[2017-05-10 13:45:45,043: INFO/job_scheduler] Job 119: complete_job: Updating cache
[2017-05-10 13:45:45,043: INFO/job_scheduler] Job 119: locked_item 140430198511696 <class 'chroma_core.models.filesystem.ManagedFilesystem'> False True
[2017-05-10 13:45:45,053: INFO/job_scheduler] Job 119 completing (errored=False, cancelled=False)
[2017-05-10 13:45:45,174: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:45:45,220: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:45:45,297: INFO/job_scheduler] notify: Dropping update of 7 (testfs-OST0000) because it has been updated since
[2017-05-10 13:45:45,297: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:45:51,987: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:45:52,041: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:45:52,165: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:45:52,205: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:45:58,798: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:45:58,809: INFO/job_scheduler] _notify: Updating .state of item 5 (MGS) from mounted to unmounted
[2017-05-10 13:45:58,810: INFO/job] StatefulObject.set_state MGS mounted->unmounted (intentional=False) 140430198431504
[2017-05-10 13:45:58,820: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on MGS at severity 40
[2017-05-10 13:45:58,820: INFO/job_scheduler] _notify: Updating .active_mount_id of item 5 (MGS) from 9 to None
[2017-05-10 13:45:58,824: INFO/job_scheduler] _notify: Set active_mount_id=None on MGS (ManagedTarget-5) and saved
[2017-05-10 13:45:58,850: INFO/job_scheduler] _notify: Updating .state of item 2 (testfs) from available to unavailable
[2017-05-10 13:45:58,850: INFO/job] StatefulObject.set_state testfs available->unavailable (intentional=False) 140430197199696
[2017-05-10 13:45:58,860: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:46:00,581: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:46:00,628: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:46:18,801: INFO/job_scheduler] _notify: Updating .state of item 6 (testfs-MDT0000) from mounted to unmounted
[2017-05-10 13:46:18,801: INFO/job] StatefulObject.set_state testfs-MDT0000 mounted->unmounted (intentional=False) 140430197251792
[2017-05-10 13:46:18,808: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on testfs-MDT0000 at severity 40
[2017-05-10 13:46:18,808: INFO/job_scheduler] _notify: Updating .active_mount_id of item 6 (testfs-MDT0000) from 11 to None
[2017-05-10 13:46:18,826: INFO/job_scheduler] _notify: Set active_mount_id=None on testfs-MDT0000 (ManagedTarget-6) and saved
[2017-05-10 13:46:18,852: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:46:18,893: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:46:20,626: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:46:20,664: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:46:29,080: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:46:29,092: INFO/job_scheduler] _notify: Updating .state of item 7 (testfs-OST0000) from mounted to unmounted
[2017-05-10 13:46:29,092: INFO/job] StatefulObject.set_state testfs-OST0000 mounted->unmounted (intentional=False) 140430198135248
[2017-05-10 13:46:29,097: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on testfs-OST0000 at severity 40
[2017-05-10 13:46:29,098: INFO/job_scheduler] _notify: Updating .active_mount_id of item 7 (testfs-OST0000) from 13 to None
[2017-05-10 13:46:29,110: INFO/job_scheduler] _notify: Set active_mount_id=None on testfs-OST0000 (ManagedTarget-7) and saved
[2017-05-10 13:46:29,141: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:46:31,304: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:46:31,344: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:46:41,287: INFO/job_scheduler] _notify: Updating .state of item 8 (testfs-OST0001) from mounted to unmounted
[2017-05-10 13:46:41,287: INFO/job] StatefulObject.set_state testfs-OST0001 mounted->unmounted (intentional=False) 140430197261648
[2017-05-10 13:46:41,300: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on testfs-OST0001 at severity 40
[2017-05-10 13:46:41,300: INFO/job_scheduler] _notify: Updating .active_mount_id of item 8 (testfs-OST0001) from 15 to None
[2017-05-10 13:46:41,316: INFO/job_scheduler] _notify: Set active_mount_id=None on testfs-OST0001 (ManagedTarget-8) and saved
[2017-05-10 13:46:41,324: INFO/job_scheduler] _notify: Updating .state of item 2 (testfs) from unavailable to stopped
[2017-05-10 13:46:41,324: INFO/job] StatefulObject.set_state testfs unavailable->stopped (intentional=False) 140430198031568
[2017-05-10 13:46:41,354: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:46:41,376: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:46:50,105: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:46:50,161: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:47:12,604: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:47:12,613: INFO/job_scheduler] _notify: Updating .state of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration) from started to stopped
[2017-05-10 13:47:12,613: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration started->stopped (intentional=False) 140430197186640
[2017-05-10 13:47:12,621: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-10 13:47:12,634: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:47:20,765: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from True to False
[2017-05-10 13:47:20,772: INFO/job_scheduler] _notify: Set corosync_reported_up=False on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-05-10 13:47:20,789: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:47:22,744: INFO/job_scheduler] _notify: Updating .state of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from started to stopped
[2017-05-10 13:47:22,744: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration started->stopped (intentional=False) 140430198425424
[2017-05-10 13:47:22,752: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-10 13:47:22,780: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:47:22,799: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:47:40,869: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-10 13:47:40,875: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-05-10 13:47:40,883: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:47:42,780: INFO/job_scheduler] _notify: Updating .state of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from stopped to started
[2017-05-10 13:47:42,780: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=False) 140430198402704
[2017-05-10 13:47:42,814: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 13:47:42,839: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:47:42,854: INFO/job_scheduler] _notify: Updating .state of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration) from stopped to started
[2017-05-10 13:47:42,854: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=False) 140430197271824
[2017-05-10 13:47:42,909: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 13:47:42,918: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:47:50,853: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:47:50,878: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:48:00,650: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:48:00,654: INFO/command_plan] set_state: <class 'chroma_core.models.filesystem.ManagedFilesystem'>-2 transitioning from stopped to removed
[2017-05-10 13:48:00,719: INFO/command_plan] Created command 35 (Remove file system testfs from configuration) with 5 jobs
[2017-05-10 13:48:00,781: INFO/job_scheduler] run_next: 4 runnable jobs of (5 pending, 0 tasked)
[2017-05-10 13:48:01,100: INFO/job_scheduler] Job 120: RunJobThread.run
[2017-05-10 13:48:01,101: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com unconfigure_target_ha {'uuid': u'13921496978265162699', 'ha_label': u'testfs-OST0000_b8032c', 'primary': False}
[2017-05-10 13:48:01,116: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1f9e66b9-064c-4917-8a1c-024f98bebf4a
[2017-05-10 13:48:01,116: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1f9e66b9-064c-4917-8a1c-024f98bebf4a
[2017-05-10 13:48:01,117: INFO/job_scheduler] Job 121: RunJobThread.run
[2017-05-10 13:48:01,118: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unconfigure_target_ha {'uuid': u'e4158786-3ffd-4c1c-9a7d-99e762123ce1', 'ha_label': u'testfs-MDT0000_0770c2', 'primary': False}
[2017-05-10 13:48:01,132: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9de4845a-440f-4de9-9e84-b7f5c7df125a
[2017-05-10 13:48:01,132: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9de4845a-440f-4de9-9e84-b7f5c7df125a
[2017-05-10 13:48:01,133: INFO/job_scheduler] Job 122: RunJobThread.run
[2017-05-10 13:48:01,135: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com unconfigure_target_ha {'uuid': u'1961503762134256363', 'ha_label': u'testfs-OST0001_865fe0', 'primary': False}
[2017-05-10 13:48:01,157: INFO/job_scheduler] Job 123: RunJobThread.run
[2017-05-10 13:48:01,159: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3f46ee61-a598-4bcb-a5ef-0fc16e293f4e
[2017-05-10 13:48:01,159: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3f46ee61-a598-4bcb-a5ef-0fc16e293f4e
[2017-05-10 13:48:01,174: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux'}
[2017-05-10 13:48:01,198: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux'}
[2017-05-10 13:48:01,210: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 87e2d6be-6270-4fd2-8ed4-fdafe15b9f81
[2017-05-10 13:48:01,210: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 87e2d6be-6270-4fd2-8ed4-fdafe15b9f81
[2017-05-10 13:48:01,245: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux'}
[2017-05-10 13:48:01,262: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cc28ea0a-b845-4a44-bb9d-06c7ab54dd9b
[2017-05-10 13:48:01,262: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cc28ea0a-b845-4a44-bb9d-06c7ab54dd9b
[2017-05-10 13:48:01,289: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 50108b0e-e70d-4aec-b414-feacb8bc2247
[2017-05-10 13:48:01,289: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 50108b0e-e70d-4aec-b414-feacb8bc2247
[2017-05-10 13:48:01,303: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:48:01,303: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:48:01,303: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:48:01,303: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:48:01,303: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 87e2d6be-6270-4fd2-8ed4-fdafe15b9f81
[2017-05-10 13:48:01,303: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 87e2d6be-6270-4fd2-8ed4-fdafe15b9f81
[2017-05-10 13:48:01,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:48:01,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:48:01,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:48:01,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:48:01,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cc28ea0a-b845-4a44-bb9d-06c7ab54dd9b
[2017-05-10 13:48:01,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cc28ea0a-b845-4a44-bb9d-06c7ab54dd9b
[2017-05-10 13:48:01,341: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cc28ea0a-b845-4a44-bb9d-06c7ab54dd9b
[2017-05-10 13:48:01,341: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cc28ea0a-b845-4a44-bb9d-06c7ab54dd9b
[2017-05-10 13:48:01,351: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 87e2d6be-6270-4fd2-8ed4-fdafe15b9f81
[2017-05-10 13:48:01,351: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 87e2d6be-6270-4fd2-8ed4-fdafe15b9f81
[2017-05-10 13:48:01,409: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:48:01,409: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:48:01,410: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:48:01,410: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:48:01,410: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 50108b0e-e70d-4aec-b414-feacb8bc2247
[2017-05-10 13:48:01,410: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 50108b0e-e70d-4aec-b414-feacb8bc2247
[2017-05-10 13:48:01,455: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 50108b0e-e70d-4aec-b414-feacb8bc2247
[2017-05-10 13:48:01,455: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 50108b0e-e70d-4aec-b414-feacb8bc2247
[2017-05-10 13:48:01,455: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com import_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'pacemaker_ha_operation': False, 'device_type': 'linux', 'validate_importable': True}
[2017-05-10 13:48:01,502: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9c904771-4636-47e7-ab02-ad9757039dc2
[2017-05-10 13:48:01,502: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9c904771-4636-47e7-ab02-ad9757039dc2
[2017-05-10 13:48:01,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:48:01,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:48:01,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:48:01,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:48:01,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9c904771-4636-47e7-ab02-ad9757039dc2
[2017-05-10 13:48:01,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9c904771-4636-47e7-ab02-ad9757039dc2
[2017-05-10 13:48:01,635: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9c904771-4636-47e7-ab02-ad9757039dc2
[2017-05-10 13:48:01,635: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9c904771-4636-47e7-ab02-ad9757039dc2
[2017-05-10 13:48:01,636: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_target {'ha_label': u'MGS_8e7c4d'}
[2017-05-10 13:48:01,666: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a3d406ba-c231-4518-a7e9-b90320f66445
[2017-05-10 13:48:01,666: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a3d406ba-c231-4518-a7e9-b90320f66445
[2017-05-10 13:48:07,766: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:48:07,766: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:48:07,766: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:48:07,766: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:48:07,766: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a3d406ba-c231-4518-a7e9-b90320f66445
[2017-05-10 13:48:07,766: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a3d406ba-c231-4518-a7e9-b90320f66445
[2017-05-10 13:48:07,793: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a3d406ba-c231-4518-a7e9-b90320f66445
[2017-05-10 13:48:07,793: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a3d406ba-c231-4518-a7e9-b90320f66445
[2017-05-10 13:48:07,793: INFO/job_scheduler] Job 123 finished 1 steps successfully
[2017-05-10 13:48:07,886: INFO/job] StatefulObject.set_state MGS unmounted->mounted (intentional=True) 140430198099344
[2017-05-10 13:48:07,916: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 20
[2017-05-10 13:48:07,940: INFO/job] Job 123: StateChangeJob complete, setting state mounted on MGS
[2017-05-10 13:48:07,942: INFO/job] Job 123: not_deleted=True
[2017-05-10 13:48:07,943: INFO/job_scheduler] Job 123: complete_job: Updating cache
[2017-05-10 13:48:07,943: INFO/job_scheduler] Job 123: locked_item 140430198099344 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:48:07,949: INFO/job_scheduler] Job 123 completing (errored=False, cancelled=False)
[2017-05-10 13:48:08,016: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 13:48:09,100: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 13:48:09,119: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 13:48:12,918: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 13:48:12,952: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 13:48:19,268: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:48:19,268: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:48:19,268: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:48:19,268: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:48:19,268: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1f9e66b9-064c-4917-8a1c-024f98bebf4a
[2017-05-10 13:48:19,268: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1f9e66b9-064c-4917-8a1c-024f98bebf4a
[2017-05-10 13:48:19,292: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1f9e66b9-064c-4917-8a1c-024f98bebf4a
[2017-05-10 13:48:19,292: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1f9e66b9-064c-4917-8a1c-024f98bebf4a
[2017-05-10 13:48:19,294: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com unconfigure_target_ha {'uuid': u'13921496978265162699', 'ha_label': u'testfs-OST0000_b8032c', 'primary': True}
[2017-05-10 13:48:19,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc af292ccd-0289-40df-9708-4d73f284ce07
[2017-05-10 13:48:19,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc af292ccd-0289-40df-9708-4d73f284ce07
[2017-05-10 13:48:19,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:48:19,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:48:19,325: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:48:19,325: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:48:19,325: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3f46ee61-a598-4bcb-a5ef-0fc16e293f4e
[2017-05-10 13:48:19,325: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3f46ee61-a598-4bcb-a5ef-0fc16e293f4e
[2017-05-10 13:48:19,326: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:48:19,326: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:48:19,326: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:48:19,326: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:48:19,327: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9de4845a-440f-4de9-9e84-b7f5c7df125a
[2017-05-10 13:48:19,327: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9de4845a-440f-4de9-9e84-b7f5c7df125a
[2017-05-10 13:48:19,344: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 13:48:19,364: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9de4845a-440f-4de9-9e84-b7f5c7df125a
[2017-05-10 13:48:19,364: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9de4845a-440f-4de9-9e84-b7f5c7df125a
[2017-05-10 13:48:19,365: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com unconfigure_target_ha {'uuid': u'e4158786-3ffd-4c1c-9a7d-99e762123ce1', 'ha_label': u'testfs-MDT0000_0770c2', 'primary': True}
[2017-05-10 13:48:19,375: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3f46ee61-a598-4bcb-a5ef-0fc16e293f4e
[2017-05-10 13:48:19,375: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3f46ee61-a598-4bcb-a5ef-0fc16e293f4e
[2017-05-10 13:48:19,377: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com unconfigure_target_ha {'uuid': u'1961503762134256363', 'ha_label': u'testfs-OST0001_865fe0', 'primary': True}
[2017-05-10 13:48:19,384: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 13:48:19,403: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5f662fae-bddc-4a10-b1a3-2128a647b28d
[2017-05-10 13:48:19,403: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5f662fae-bddc-4a10-b1a3-2128a647b28d
[2017-05-10 13:48:19,426: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 26b18319-dcee-4c1f-a353-eb24b305e67b
[2017-05-10 13:48:19,426: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 26b18319-dcee-4c1f-a353-eb24b305e67b
[2017-05-10 13:48:20,124: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 13:48:20,140: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 13:48:20,974: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 13:48:20,990: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 13:48:20,992: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 13:48:20,997: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 13:48:21,467: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 13:48:21,470: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 13:48:22,137: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 13:48:22,146: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 13:48:22,972: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 13:48:23,005: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 13:48:24,720: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 13:48:24,738: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 3 tasked)
[2017-05-10 13:48:38,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:48:38,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:48:38,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:48:38,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:48:38,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc af292ccd-0289-40df-9708-4d73f284ce07
[2017-05-10 13:48:38,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc af292ccd-0289-40df-9708-4d73f284ce07
[2017-05-10 13:48:38,534: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:48:38,534: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:48:38,535: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:48:38,535: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:48:38,535: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5f662fae-bddc-4a10-b1a3-2128a647b28d
[2017-05-10 13:48:38,535: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5f662fae-bddc-4a10-b1a3-2128a647b28d
[2017-05-10 13:48:38,536: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5f662fae-bddc-4a10-b1a3-2128a647b28d
[2017-05-10 13:48:38,536: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5f662fae-bddc-4a10-b1a3-2128a647b28d
[2017-05-10 13:48:38,542: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unconfigure_target_store {'uuid': u'e4158786-3ffd-4c1c-9a7d-99e762123ce1'}
[2017-05-10 13:48:38,554: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc af292ccd-0289-40df-9708-4d73f284ce07
[2017-05-10 13:48:38,554: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc af292ccd-0289-40df-9708-4d73f284ce07
[2017-05-10 13:48:38,556: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com unconfigure_target_store {'uuid': u'13921496978265162699'}
[2017-05-10 13:48:38,557: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9663b028-f60f-4847-b9c0-834a5f43a315
[2017-05-10 13:48:38,557: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9663b028-f60f-4847-b9c0-834a5f43a315
[2017-05-10 13:48:38,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bb815427-f4af-408b-b261-174488a43921
[2017-05-10 13:48:38,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bb815427-f4af-408b-b261-174488a43921
[2017-05-10 13:48:38,615: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:48:38,615: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:48:38,615: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:48:38,615: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:48:38,615: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 26b18319-dcee-4c1f-a353-eb24b305e67b
[2017-05-10 13:48:38,615: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 26b18319-dcee-4c1f-a353-eb24b305e67b
[2017-05-10 13:48:38,618: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 26b18319-dcee-4c1f-a353-eb24b305e67b
[2017-05-10 13:48:38,618: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 26b18319-dcee-4c1f-a353-eb24b305e67b
[2017-05-10 13:48:38,619: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com unconfigure_target_store {'uuid': u'1961503762134256363'}
[2017-05-10 13:48:38,639: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e8b1c1f0-d73c-44fb-a2d5-84965c29eaf0
[2017-05-10 13:48:38,639: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e8b1c1f0-d73c-44fb-a2d5-84965c29eaf0
[2017-05-10 13:48:38,640: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:48:38,640: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:48:38,640: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:48:38,640: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:48:38,640: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9663b028-f60f-4847-b9c0-834a5f43a315
[2017-05-10 13:48:38,640: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9663b028-f60f-4847-b9c0-834a5f43a315
[2017-05-10 13:48:38,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:48:38,661: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:48:38,662: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:48:38,662: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:48:38,664: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bb815427-f4af-408b-b261-174488a43921
[2017-05-10 13:48:38,664: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bb815427-f4af-408b-b261-174488a43921
[2017-05-10 13:48:38,682: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9663b028-f60f-4847-b9c0-834a5f43a315
[2017-05-10 13:48:38,682: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9663b028-f60f-4847-b9c0-834a5f43a315
[2017-05-10 13:48:38,683: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com unconfigure_target_store {'uuid': u'e4158786-3ffd-4c1c-9a7d-99e762123ce1'}
[2017-05-10 13:48:38,698: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0fe8d124-e1a3-44ce-85f9-890e2a34d340
[2017-05-10 13:48:38,698: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0fe8d124-e1a3-44ce-85f9-890e2a34d340
[2017-05-10 13:48:38,698: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bb815427-f4af-408b-b261-174488a43921
[2017-05-10 13:48:38,698: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bb815427-f4af-408b-b261-174488a43921
[2017-05-10 13:48:38,699: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com unconfigure_target_store {'uuid': u'13921496978265162699'}
[2017-05-10 13:48:38,713: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 010da2f7-420c-4c8c-b7f3-da9213d485de
[2017-05-10 13:48:38,713: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 010da2f7-420c-4c8c-b7f3-da9213d485de
[2017-05-10 13:48:38,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:48:38,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:48:38,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:48:38,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:48:38,782: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e8b1c1f0-d73c-44fb-a2d5-84965c29eaf0
[2017-05-10 13:48:38,782: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e8b1c1f0-d73c-44fb-a2d5-84965c29eaf0
[2017-05-10 13:48:38,783: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:48:38,783: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:48:38,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:48:38,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:48:38,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 010da2f7-420c-4c8c-b7f3-da9213d485de
[2017-05-10 13:48:38,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 010da2f7-420c-4c8c-b7f3-da9213d485de
[2017-05-10 13:48:38,786: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:48:38,786: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:48:38,786: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:48:38,786: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:48:38,786: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0fe8d124-e1a3-44ce-85f9-890e2a34d340
[2017-05-10 13:48:38,786: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0fe8d124-e1a3-44ce-85f9-890e2a34d340
[2017-05-10 13:48:38,812: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e8b1c1f0-d73c-44fb-a2d5-84965c29eaf0
[2017-05-10 13:48:38,812: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e8b1c1f0-d73c-44fb-a2d5-84965c29eaf0
[2017-05-10 13:48:38,813: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com unconfigure_target_store {'uuid': u'1961503762134256363'}
[2017-05-10 13:48:38,822: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0fe8d124-e1a3-44ce-85f9-890e2a34d340
[2017-05-10 13:48:38,822: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0fe8d124-e1a3-44ce-85f9-890e2a34d340
[2017-05-10 13:48:38,823: INFO/job_scheduler] Job 121 finished 4 steps successfully
[2017-05-10 13:48:38,830: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 010da2f7-420c-4c8c-b7f3-da9213d485de
[2017-05-10 13:48:38,830: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 010da2f7-420c-4c8c-b7f3-da9213d485de
[2017-05-10 13:48:38,831: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 04acdf24-6d2d-4418-8517-5486616b6ef4
[2017-05-10 13:48:38,831: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 04acdf24-6d2d-4418-8517-5486616b6ef4
[2017-05-10 13:48:38,830: INFO/job_scheduler] Job 120 finished 4 steps successfully
[2017-05-10 13:48:38,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:48:38,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:48:38,909: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:48:38,909: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:48:38,909: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 04acdf24-6d2d-4418-8517-5486616b6ef4
[2017-05-10 13:48:38,909: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 04acdf24-6d2d-4418-8517-5486616b6ef4
[2017-05-10 13:48:38,945: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 04acdf24-6d2d-4418-8517-5486616b6ef4
[2017-05-10 13:48:38,945: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 04acdf24-6d2d-4418-8517-5486616b6ef4
[2017-05-10 13:48:38,946: INFO/job_scheduler] Job 122 finished 4 steps successfully
[2017-05-10 13:48:39,032: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.target.ManagedTarget'> 6
[2017-05-10 13:48:39,033: INFO/job] StatefulObject.set_state testfs-MDT0000 unmounted->removed (intentional=True) 140430198620304
[2017-05-10 13:48:39,056: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm6.lotus.hpdd.lab.intel.com at severity 20
[2017-05-10 13:48:39,075: INFO/job] Job 121: StateChangeJob complete, setting state removed on testfs-MDT0000
[2017-05-10 13:48:39,075: INFO/job] Job 121: not_deleted=None
[2017-05-10 13:48:39,075: INFO/job_scheduler] Job 121: complete_job: Updating cache
[2017-05-10 13:48:39,075: INFO/job_scheduler] Job 121: locked_item 140430198620304 <class 'chroma_core.models.target.ManagedTarget'> False None
[2017-05-10 13:48:39,075: INFO/job_scheduler] Job 121 completing (errored=False, cancelled=False)
[2017-05-10 13:48:39,136: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-05-10 13:48:39,150: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.target.ManagedTarget'> 7
[2017-05-10 13:48:39,150: INFO/job] StatefulObject.set_state testfs-OST0000 unmounted->removed (intentional=True) 140430197677712
[2017-05-10 13:48:39,213: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 20
[2017-05-10 13:48:39,251: INFO/job] Job 120: StateChangeJob complete, setting state removed on testfs-OST0000
[2017-05-10 13:48:39,252: INFO/job] Job 120: not_deleted=None
[2017-05-10 13:48:39,252: INFO/job_scheduler] Job 120: complete_job: Updating cache
[2017-05-10 13:48:39,252: INFO/job_scheduler] Job 120: locked_item 140430197677712 <class 'chroma_core.models.target.ManagedTarget'> False None
[2017-05-10 13:48:39,252: INFO/job_scheduler] Job 120 completing (errored=False, cancelled=False)
[2017-05-10 13:48:39,320: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 13:48:39,322: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 13:48:39,395: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.target.ManagedTarget'> 8
[2017-05-10 13:48:39,395: INFO/job] StatefulObject.set_state testfs-OST0001 unmounted->removed (intentional=True) 140430198145872
[2017-05-10 13:48:39,447: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm8.lotus.hpdd.lab.intel.com at severity 20
[2017-05-10 13:48:39,457: INFO/job] Job 122: StateChangeJob complete, setting state removed on testfs-OST0001
[2017-05-10 13:48:39,457: INFO/job] Job 122: not_deleted=None
[2017-05-10 13:48:39,457: INFO/job_scheduler] Job 122: complete_job: Updating cache
[2017-05-10 13:48:39,457: INFO/job_scheduler] Job 122: locked_item 140430198145872 <class 'chroma_core.models.target.ManagedTarget'> False None
[2017-05-10 13:48:39,457: INFO/job_scheduler] Job 122 completing (errored=False, cancelled=False)
[2017-05-10 13:48:39,544: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 13:48:39,605: INFO/job_scheduler] Job 124: RunJobThread.run
[2017-05-10 13:48:39,606: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com purge_configuration {'mgs_device_path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'mgs_device_type': 'linux', 'filesystem_name': u'testfs'}
[2017-05-10 13:48:39,621: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2c3d52a3-5abf-4c35-bde5-8eb6b63b7538
[2017-05-10 13:48:39,621: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2c3d52a3-5abf-4c35-bde5-8eb6b63b7538
[2017-05-10 13:48:40,744: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:48:40,744: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:48:40,744: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:48:40,744: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:48:40,744: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2c3d52a3-5abf-4c35-bde5-8eb6b63b7538
[2017-05-10 13:48:40,744: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2c3d52a3-5abf-4c35-bde5-8eb6b63b7538
[2017-05-10 13:48:40,745: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2c3d52a3-5abf-4c35-bde5-8eb6b63b7538
[2017-05-10 13:48:40,745: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2c3d52a3-5abf-4c35-bde5-8eb6b63b7538
[2017-05-10 13:48:40,745: INFO/job_scheduler] Job 124 finished 1 steps successfully
[2017-05-10 13:48:40,807: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.filesystem.ManagedFilesystem'> 2
[2017-05-10 13:48:40,807: INFO/job] StatefulObject.set_state testfs stopped->removed (intentional=True) 140430198432848
[2017-05-10 13:48:40,809: INFO/job] Job 124: StateChangeJob complete, setting state removed on testfs
[2017-05-10 13:48:40,810: INFO/job] Job 124: not_deleted=None
[2017-05-10 13:48:40,810: INFO/job_scheduler] Job 124: complete_job: Updating cache
[2017-05-10 13:48:40,810: INFO/job_scheduler] Job 124: locked_item 140430198432848 <class 'chroma_core.models.filesystem.ManagedFilesystem'> False None
[2017-05-10 13:48:40,810: INFO/job_scheduler] Job 124: locked_item 140430198621968 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:48:40,813: INFO/job_scheduler] Job 124 completing (errored=False, cancelled=False)
[2017-05-10 13:48:40,913: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:48:41,071: INFO/job_scheduler] notify: Dropping update of 5 (MGS) because it has been updated since
[2017-05-10 13:48:41,071: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:48:41,951: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:48:41,970: INFO/command_plan] set_state: <class 'chroma_core.models.target.ManagedTarget'>-5 transitioning from mounted to removed
[2017-05-10 13:48:41,996: INFO/command_plan] Created command 36 (Remove target MGS from the filesystem
*** WARNING *** this is destructive to your filesystem and irreversible) with 2 jobs
[2017-05-10 13:48:42,038: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 0 tasked)
[2017-05-10 13:48:42,103: INFO/job_scheduler] Job 125: RunJobThread.run
[2017-05-10 13:48:42,105: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_target {'ha_label': u'MGS_8e7c4d'}
[2017-05-10 13:48:42,130: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 60f8cf1c-abeb-41b6-b81c-a7041756ff27
[2017-05-10 13:48:42,130: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 60f8cf1c-abeb-41b6-b81c-a7041756ff27
[2017-05-10 13:48:51,030: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 13:48:52,290: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:48:52,290: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:48:52,290: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:48:52,290: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:48:52,291: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 60f8cf1c-abeb-41b6-b81c-a7041756ff27
[2017-05-10 13:48:52,291: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 60f8cf1c-abeb-41b6-b81c-a7041756ff27
[2017-05-10 13:48:52,295: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 60f8cf1c-abeb-41b6-b81c-a7041756ff27
[2017-05-10 13:48:52,295: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 60f8cf1c-abeb-41b6-b81c-a7041756ff27
[2017-05-10 13:48:52,296: INFO/job_scheduler] Job 125 finished 1 steps successfully
[2017-05-10 13:48:52,500: INFO/job] StatefulObject.set_state MGS mounted->unmounted (intentional=True) 140430198629584
[2017-05-10 13:48:52,508: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on MGS at severity 30
[2017-05-10 13:48:52,520: INFO/job] Job 125: StateChangeJob complete, setting state unmounted on MGS
[2017-05-10 13:48:52,522: INFO/job] Job 125: not_deleted=True
[2017-05-10 13:48:52,530: INFO/job_scheduler] Job 125: complete_job: Updating cache
[2017-05-10 13:48:52,530: INFO/job_scheduler] Job 125: locked_item 140430198629584 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-05-10 13:48:52,533: INFO/job_scheduler] Job 125 completing (errored=False, cancelled=False)
[2017-05-10 13:48:52,554: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 13:48:52,598: INFO/job_scheduler] Job 126: RunJobThread.run
[2017-05-10 13:48:52,599: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com unconfigure_target_ha {'uuid': u'e903f0d3-6ff8-4196-9b86-1ce2f7cd2cc4', 'ha_label': u'MGS_8e7c4d', 'primary': False}
[2017-05-10 13:48:52,645: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b81bd992-82e4-4e62-94c7-6203c89f7dde
[2017-05-10 13:48:52,645: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b81bd992-82e4-4e62-94c7-6203c89f7dde
[2017-05-10 13:48:59,231: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 13:49:10,789: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:10,789: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:10,789: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:10,789: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:10,789: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b81bd992-82e4-4e62-94c7-6203c89f7dde
[2017-05-10 13:49:10,789: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b81bd992-82e4-4e62-94c7-6203c89f7dde
[2017-05-10 13:49:10,812: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b81bd992-82e4-4e62-94c7-6203c89f7dde
[2017-05-10 13:49:10,812: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b81bd992-82e4-4e62-94c7-6203c89f7dde
[2017-05-10 13:49:10,814: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unconfigure_target_ha {'uuid': u'e903f0d3-6ff8-4196-9b86-1ce2f7cd2cc4', 'ha_label': u'MGS_8e7c4d', 'primary': True}
[2017-05-10 13:49:10,828: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3540d031-69b0-4292-bf1c-f0d5878577f8
[2017-05-10 13:49:10,828: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3540d031-69b0-4292-bf1c-f0d5878577f8
[2017-05-10 13:49:29,967: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:29,967: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:29,967: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:29,967: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:29,967: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3540d031-69b0-4292-bf1c-f0d5878577f8
[2017-05-10 13:49:29,967: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3540d031-69b0-4292-bf1c-f0d5878577f8
[2017-05-10 13:49:29,998: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3540d031-69b0-4292-bf1c-f0d5878577f8
[2017-05-10 13:49:29,998: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3540d031-69b0-4292-bf1c-f0d5878577f8
[2017-05-10 13:49:29,999: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com unconfigure_target_store {'uuid': u'e903f0d3-6ff8-4196-9b86-1ce2f7cd2cc4'}
[2017-05-10 13:49:30,013: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc eac87f2d-0723-4622-8e84-d347957e5f4c
[2017-05-10 13:49:30,013: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc eac87f2d-0723-4622-8e84-d347957e5f4c
[2017-05-10 13:49:30,086: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:30,086: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:30,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:30,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:30,090: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc eac87f2d-0723-4622-8e84-d347957e5f4c
[2017-05-10 13:49:30,090: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc eac87f2d-0723-4622-8e84-d347957e5f4c
[2017-05-10 13:49:30,129: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc eac87f2d-0723-4622-8e84-d347957e5f4c
[2017-05-10 13:49:30,129: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc eac87f2d-0723-4622-8e84-d347957e5f4c
[2017-05-10 13:49:30,130: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unconfigure_target_store {'uuid': u'e903f0d3-6ff8-4196-9b86-1ce2f7cd2cc4'}
[2017-05-10 13:49:30,145: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d6bc0748-f1e5-46bb-a7a6-a240bd395403
[2017-05-10 13:49:30,145: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d6bc0748-f1e5-46bb-a7a6-a240bd395403
[2017-05-10 13:49:30,206: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:30,206: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:30,206: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:30,206: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:30,206: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d6bc0748-f1e5-46bb-a7a6-a240bd395403
[2017-05-10 13:49:30,206: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d6bc0748-f1e5-46bb-a7a6-a240bd395403
[2017-05-10 13:49:30,211: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d6bc0748-f1e5-46bb-a7a6-a240bd395403
[2017-05-10 13:49:30,211: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d6bc0748-f1e5-46bb-a7a6-a240bd395403
[2017-05-10 13:49:30,211: INFO/job_scheduler] Job 126 finished 4 steps successfully
[2017-05-10 13:49:30,243: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.target.ManagedTarget'> 5
[2017-05-10 13:49:30,243: INFO/job] StatefulObject.set_state MGS unmounted->removed (intentional=True) 140430198629584
[2017-05-10 13:49:30,291: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 20
[2017-05-10 13:49:30,303: INFO/job] Job 126: StateChangeJob complete, setting state removed on MGS
[2017-05-10 13:49:30,304: INFO/job] Job 126: not_deleted=None
[2017-05-10 13:49:30,304: INFO/job_scheduler] Job 126: complete_job: Updating cache
[2017-05-10 13:49:30,304: INFO/job_scheduler] Job 126: locked_item 140430198629584 <class 'chroma_core.models.target.ManagedTarget'> False None
[2017-05-10 13:49:30,304: INFO/job_scheduler] Job 126 completing (errored=False, cancelled=False)
[2017-05-10 13:49:30,366: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:49:31,646: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:49:31,648: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-1 transitioning from managed to removed
[2017-05-10 13:49:31,835: INFO/command_plan] Created command 37 (Remove host lotus-32vm5.lotus.hpdd.lab.intel.com from configuration) with 9 jobs
[2017-05-10 13:49:31,901: INFO/job_scheduler] run_next: 4 runnable jobs of (9 pending, 0 tasked)
[2017-05-10 13:49:31,998: INFO/job_scheduler] Job 128: RunJobThread.run
[2017-05-10 13:49:31,999: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unconfigure_rsyslog {}
[2017-05-10 13:49:32,032: INFO/job_scheduler] Job 129: RunJobThread.run
[2017-05-10 13:49:32,032: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_lnet {}
[2017-05-10 13:49:32,050: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 24813ff4-f924-4a81-b116-cf5e58a71506
[2017-05-10 13:49:32,050: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 24813ff4-f924-4a81-b116-cf5e58a71506
[2017-05-10 13:49:32,063: INFO/job_scheduler] Job 130: RunJobThread.run
[2017-05-10 13:49:32,064: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unconfigure_ntp {}
[2017-05-10 13:49:32,083: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3f2301f9-4951-4f2b-bd15-6c0b1212359a
[2017-05-10 13:49:32,083: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3f2301f9-4951-4f2b-bd15-6c0b1212359a
[2017-05-10 13:49:32,100: INFO/job_scheduler] Job 127: RunJobThread.run
[2017-05-10 13:49:32,100: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-10 13:49:32,121: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9a0051cd-c4f1-4643-8d26-f16e7fe8bfab
[2017-05-10 13:49:32,121: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9a0051cd-c4f1-4643-8d26-f16e7fe8bfab
[2017-05-10 13:49:32,202: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f89ae4d6-07d4-42de-900a-48f7b36bc75c
[2017-05-10 13:49:32,202: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f89ae4d6-07d4-42de-900a-48f7b36bc75c
[2017-05-10 13:49:32,490: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:49:32,493: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-2 transitioning from managed to removed
[2017-05-10 13:49:32,647: INFO/command_plan] Created command 38 (Remove host lotus-32vm6.lotus.hpdd.lab.intel.com from configuration) with 9 jobs
[2017-05-10 13:49:32,734: INFO/job_scheduler] run_next: 4 runnable jobs of (14 pending, 4 tasked)
[2017-05-10 13:49:32,800: INFO/job_scheduler] Job 136: RunJobThread.run
[2017-05-10 13:49:32,801: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com unconfigure_rsyslog {}
[2017-05-10 13:49:32,813: INFO/job_scheduler] Job 137: RunJobThread.run
[2017-05-10 13:49:32,814: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com stop_lnet {}
[2017-05-10 13:49:32,817: INFO/job_scheduler] Job 138: RunJobThread.run
[2017-05-10 13:49:32,817: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-10 13:49:32,820: INFO/job_scheduler] Job 139: RunJobThread.run
[2017-05-10 13:49:32,820: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com unconfigure_ntp {}
[2017-05-10 13:49:32,832: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 59f708e9-1562-472a-9b80-74da053411d9
[2017-05-10 13:49:32,832: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 59f708e9-1562-472a-9b80-74da053411d9
[2017-05-10 13:49:32,921: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d5af240d-cbc5-460c-811f-9ca3569d5cc7
[2017-05-10 13:49:32,921: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d5af240d-cbc5-460c-811f-9ca3569d5cc7
[2017-05-10 13:49:32,955: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 615020eb-ddaa-4a9c-90af-e64e4ff26541
[2017-05-10 13:49:32,955: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 615020eb-ddaa-4a9c-90af-e64e4ff26541
[2017-05-10 13:49:33,001: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a296ee0b-cdb6-4fab-8dea-4ea60f28f70e
[2017-05-10 13:49:33,001: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a296ee0b-cdb6-4fab-8dea-4ea60f28f70e
[2017-05-10 13:49:33,060: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:49:33,081: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-3 transitioning from managed to removed
[2017-05-10 13:49:33,216: INFO/command_plan] Created command 39 (Remove host lotus-32vm7.lotus.hpdd.lab.intel.com from configuration) with 9 jobs
[2017-05-10 13:49:33,284: INFO/job_scheduler] run_next: 4 runnable jobs of (19 pending, 8 tasked)
[2017-05-10 13:49:33,312: INFO/job_scheduler] Job 145: RunJobThread.run
[2017-05-10 13:49:33,312: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com unconfigure_ntp {}
[2017-05-10 13:49:33,326: INFO/job_scheduler] Job 146: RunJobThread.run
[2017-05-10 13:49:33,327: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com stop_lnet {}
[2017-05-10 13:49:33,336: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0ebdb009-5e75-42af-9de0-bb681c70fd8d
[2017-05-10 13:49:33,336: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0ebdb009-5e75-42af-9de0-bb681c70fd8d
[2017-05-10 13:49:33,361: INFO/job_scheduler] Job 147: RunJobThread.run
[2017-05-10 13:49:33,362: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-10 13:49:33,362: INFO/job_scheduler] Job 148: RunJobThread.run
[2017-05-10 13:49:33,363: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com unconfigure_rsyslog {}
[2017-05-10 13:49:33,375: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 59b7b8dc-b8ee-4e65-a7df-c864efeb3a71
[2017-05-10 13:49:33,375: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 59b7b8dc-b8ee-4e65-a7df-c864efeb3a71
[2017-05-10 13:49:33,397: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4a476ea0-e051-40af-a454-2703e3259a08
[2017-05-10 13:49:33,397: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4a476ea0-e051-40af-a454-2703e3259a08
[2017-05-10 13:49:33,422: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2745d06a-1ad4-4ae4-8385-aaeb5ac08188
[2017-05-10 13:49:33,422: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2745d06a-1ad4-4ae4-8385-aaeb5ac08188
[2017-05-10 13:49:33,580: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:49:33,582: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-4 transitioning from managed to removed
[2017-05-10 13:49:33,723: INFO/command_plan] Created command 40 (Remove host lotus-32vm8.lotus.hpdd.lab.intel.com from configuration) with 9 jobs
[2017-05-10 13:49:33,817: INFO/job_scheduler] run_next: 4 runnable jobs of (24 pending, 12 tasked)
[2017-05-10 13:49:33,899: INFO/job_scheduler] Job 154: RunJobThread.run
[2017-05-10 13:49:33,899: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-10 13:49:33,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 07a419fb-49c7-48b9-b3bc-3329f0d79227
[2017-05-10 13:49:33,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 07a419fb-49c7-48b9-b3bc-3329f0d79227
[2017-05-10 13:49:33,943: INFO/job_scheduler] Job 155: RunJobThread.run
[2017-05-10 13:49:33,944: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com stop_lnet {}
[2017-05-10 13:49:33,971: INFO/job_scheduler] Job 156: RunJobThread.run
[2017-05-10 13:49:33,971: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com unconfigure_ntp {}
[2017-05-10 13:49:33,974: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a9c8d3dd-3cd9-4641-a5c4-5b2ee629b5cf
[2017-05-10 13:49:33,974: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a9c8d3dd-3cd9-4641-a5c4-5b2ee629b5cf
[2017-05-10 13:49:34,027: INFO/job_scheduler] Job 157: RunJobThread.run
[2017-05-10 13:49:34,028: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com unconfigure_rsyslog {}
[2017-05-10 13:49:34,044: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 44e5ad8f-acfa-4ca4-a8d9-ec0683354756
[2017-05-10 13:49:34,044: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 44e5ad8f-acfa-4ca4-a8d9-ec0683354756
[2017-05-10 13:49:34,214: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f9676729-68ed-4876-acf6-2c928f5dcaea
[2017-05-10 13:49:34,214: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f9676729-68ed-4876-acf6-2c928f5dcaea
[2017-05-10 13:49:38,211: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:38,211: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:38,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:38,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:38,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9a0051cd-c4f1-4643-8d26-f16e7fe8bfab
[2017-05-10 13:49:38,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9a0051cd-c4f1-4643-8d26-f16e7fe8bfab
[2017-05-10 13:49:38,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9a0051cd-c4f1-4643-8d26-f16e7fe8bfab
[2017-05-10 13:49:38,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9a0051cd-c4f1-4643-8d26-f16e7fe8bfab
[2017-05-10 13:49:38,214: INFO/job_scheduler] Job 130 finished 1 steps successfully
[2017-05-10 13:49:38,229: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:38,229: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:38,229: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:38,229: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:38,229: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f89ae4d6-07d4-42de-900a-48f7b36bc75c
[2017-05-10 13:49:38,229: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f89ae4d6-07d4-42de-900a-48f7b36bc75c
[2017-05-10 13:49:38,242: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f89ae4d6-07d4-42de-900a-48f7b36bc75c
[2017-05-10 13:49:38,242: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f89ae4d6-07d4-42de-900a-48f7b36bc75c
[2017-05-10 13:49:38,243: INFO/job_scheduler] Job 127 finished 1 steps successfully
[2017-05-10 13:49:38,458: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration configured->unconfigured (intentional=True) 140430198704784
[2017-05-10 13:49:38,462: INFO/job] Job 130: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-10 13:49:38,462: INFO/job] Job 130: not_deleted=True
[2017-05-10 13:49:38,462: INFO/job_scheduler] Job 130: complete_job: Updating cache
[2017-05-10 13:49:38,463: INFO/job_scheduler] Job 130: locked_item 140430198704784 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-10 13:49:38,465: INFO/job_scheduler] Job 130 completing (errored=False, cancelled=False)
[2017-05-10 13:49:38,492: INFO/job_scheduler] run_next: 0 runnable jobs of (20 pending, 15 tasked)
[2017-05-10 13:49:38,734: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration started->stopped (intentional=True) 140430198648400
[2017-05-10 13:49:38,741: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-10 13:49:38,746: INFO/job] Job 127: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 13:49:38,746: INFO/job] Job 127: not_deleted=True
[2017-05-10 13:49:38,746: INFO/job_scheduler] Job 127: complete_job: Updating cache
[2017-05-10 13:49:38,746: INFO/job_scheduler] Job 127: locked_item 140430198648400 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 13:49:38,752: INFO/job_scheduler] Job 127 completing (errored=False, cancelled=False)
[2017-05-10 13:49:38,784: INFO/job_scheduler] run_next: 1 runnable jobs of (20 pending, 14 tasked)
[2017-05-10 13:49:38,800: INFO/job_scheduler] Job 132: RunJobThread.run
[2017-05-10 13:49:38,802: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_corosync2 {}
[2017-05-10 13:49:38,837: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 75838629-7413-4723-9eca-46d02fa7869d
[2017-05-10 13:49:38,837: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 75838629-7413-4723-9eca-46d02fa7869d
[2017-05-10 13:49:39,049: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:39,049: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:39,050: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:39,050: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:39,050: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 615020eb-ddaa-4a9c-90af-e64e4ff26541
[2017-05-10 13:49:39,050: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 615020eb-ddaa-4a9c-90af-e64e4ff26541
[2017-05-10 13:49:39,071: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 615020eb-ddaa-4a9c-90af-e64e4ff26541
[2017-05-10 13:49:39,071: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 615020eb-ddaa-4a9c-90af-e64e4ff26541
[2017-05-10 13:49:39,072: INFO/job_scheduler] Job 139 finished 1 steps successfully
[2017-05-10 13:49:39,151: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration configured->unconfigured (intentional=True) 140430198706000
[2017-05-10 13:49:39,157: INFO/job] Job 139: StateChangeJob complete, setting state unconfigured on lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-10 13:49:39,158: INFO/job] Job 139: not_deleted=True
[2017-05-10 13:49:39,158: INFO/job_scheduler] Job 139: complete_job: Updating cache
[2017-05-10 13:49:39,158: INFO/job_scheduler] Job 139: locked_item 140430198706000 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-10 13:49:39,160: INFO/job_scheduler] Job 139 completing (errored=False, cancelled=False)
[2017-05-10 13:49:39,192: INFO/job_scheduler] run_next: 0 runnable jobs of (19 pending, 14 tasked)
[2017-05-10 13:49:39,196: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:39,196: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:39,196: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:39,196: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:39,196: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 24813ff4-f924-4a81-b116-cf5e58a71506
[2017-05-10 13:49:39,196: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 24813ff4-f924-4a81-b116-cf5e58a71506
[2017-05-10 13:49:39,224: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 24813ff4-f924-4a81-b116-cf5e58a71506
[2017-05-10 13:49:39,224: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 24813ff4-f924-4a81-b116-cf5e58a71506
[2017-05-10 13:49:39,224: INFO/job_scheduler] Job 128 finished 1 steps successfully
[2017-05-10 13:49:39,310: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration configured->unconfigured (intentional=True) 140430198565648
[2017-05-10 13:49:39,314: INFO/job] Job 128: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-10 13:49:39,314: INFO/job] Job 128: not_deleted=True
[2017-05-10 13:49:39,314: INFO/job_scheduler] Job 128: complete_job: Updating cache
[2017-05-10 13:49:39,314: INFO/job_scheduler] Job 128: locked_item 140430198565648 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-10 13:49:39,317: INFO/job_scheduler] Job 128 completing (errored=False, cancelled=False)
[2017-05-10 13:49:39,334: INFO/job_scheduler] run_next: 0 runnable jobs of (19 pending, 13 tasked)
[2017-05-10 13:49:39,452: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:39,452: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:39,453: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:39,453: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:39,453: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0ebdb009-5e75-42af-9de0-bb681c70fd8d
[2017-05-10 13:49:39,453: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0ebdb009-5e75-42af-9de0-bb681c70fd8d
[2017-05-10 13:49:39,461: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0ebdb009-5e75-42af-9de0-bb681c70fd8d
[2017-05-10 13:49:39,461: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0ebdb009-5e75-42af-9de0-bb681c70fd8d
[2017-05-10 13:49:39,462: INFO/job_scheduler] Job 145 finished 1 steps successfully
[2017-05-10 13:49:39,542: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com NTP configuration configured->unconfigured (intentional=True) 140430198624528
[2017-05-10 13:49:39,545: INFO/job] Job 145: StateChangeJob complete, setting state unconfigured on lotus-32vm7.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-10 13:49:39,546: INFO/job] Job 145: not_deleted=True
[2017-05-10 13:49:39,546: INFO/job_scheduler] Job 145: complete_job: Updating cache
[2017-05-10 13:49:39,546: INFO/job_scheduler] Job 145: locked_item 140430198624528 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-10 13:49:39,548: INFO/job_scheduler] Job 145 completing (errored=False, cancelled=False)
[2017-05-10 13:49:39,562: INFO/job_scheduler] run_next: 0 runnable jobs of (19 pending, 12 tasked)
[2017-05-10 13:49:39,931: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:39,931: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:39,931: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:39,931: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:39,931: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 59f708e9-1562-472a-9b80-74da053411d9
[2017-05-10 13:49:39,931: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 59f708e9-1562-472a-9b80-74da053411d9
[2017-05-10 13:49:39,948: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 59f708e9-1562-472a-9b80-74da053411d9
[2017-05-10 13:49:39,948: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 59f708e9-1562-472a-9b80-74da053411d9
[2017-05-10 13:49:39,949: INFO/job_scheduler] Job 136 finished 1 steps successfully
[2017-05-10 13:49:40,030: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration configured->unconfigured (intentional=True) 140430198703184
[2017-05-10 13:49:40,033: INFO/job] Job 136: StateChangeJob complete, setting state unconfigured on lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-10 13:49:40,033: INFO/job] Job 136: not_deleted=True
[2017-05-10 13:49:40,034: INFO/job_scheduler] Job 136: complete_job: Updating cache
[2017-05-10 13:49:40,034: INFO/job_scheduler] Job 136: locked_item 140430198703184 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-10 13:49:40,037: INFO/job_scheduler] Job 136 completing (errored=False, cancelled=False)
[2017-05-10 13:49:40,049: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:40,049: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:40,050: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:40,050: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:40,050: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a296ee0b-cdb6-4fab-8dea-4ea60f28f70e
[2017-05-10 13:49:40,050: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a296ee0b-cdb6-4fab-8dea-4ea60f28f70e
[2017-05-10 13:49:40,054: INFO/job_scheduler] run_next: 0 runnable jobs of (19 pending, 11 tasked)
[2017-05-10 13:49:40,069: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a296ee0b-cdb6-4fab-8dea-4ea60f28f70e
[2017-05-10 13:49:40,069: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a296ee0b-cdb6-4fab-8dea-4ea60f28f70e
[2017-05-10 13:49:40,069: INFO/job_scheduler] Job 138 finished 1 steps successfully
[2017-05-10 13:49:40,121: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration started->stopped (intentional=True) 140430198458960
[2017-05-10 13:49:40,132: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-10 13:49:40,144: INFO/job] Job 138: StateChangeJob complete, setting state stopped on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 13:49:40,145: INFO/job] Job 138: not_deleted=True
[2017-05-10 13:49:40,145: INFO/job_scheduler] Job 138: complete_job: Updating cache
[2017-05-10 13:49:40,149: INFO/job_scheduler] Job 138: locked_item 140430198458960 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 13:49:40,161: INFO/job_scheduler] Job 138 completing (errored=False, cancelled=False)
[2017-05-10 13:49:40,178: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:40,178: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:40,178: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:40,178: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:40,178: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 44e5ad8f-acfa-4ca4-a8d9-ec0683354756
[2017-05-10 13:49:40,178: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 44e5ad8f-acfa-4ca4-a8d9-ec0683354756
[2017-05-10 13:49:40,188: INFO/job_scheduler] run_next: 1 runnable jobs of (19 pending, 10 tasked)
[2017-05-10 13:49:40,194: INFO/job_scheduler] Job 141: RunJobThread.run
[2017-05-10 13:49:40,195: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com stop_corosync2 {}
[2017-05-10 13:49:40,215: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 44e5ad8f-acfa-4ca4-a8d9-ec0683354756
[2017-05-10 13:49:40,215: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 44e5ad8f-acfa-4ca4-a8d9-ec0683354756
[2017-05-10 13:49:40,216: INFO/job_scheduler] run_next: 0 runnable jobs of (18 pending, 11 tasked)
[2017-05-10 13:49:40,219: INFO/job_scheduler] Job 156 finished 1 steps successfully
[2017-05-10 13:49:40,220: INFO/job_scheduler] run_next: 0 runnable jobs of (18 pending, 11 tasked)
[2017-05-10 13:49:40,228: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6ed8761d-9e67-409c-a505-d150431ecc7b
[2017-05-10 13:49:40,228: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6ed8761d-9e67-409c-a505-d150431ecc7b
[2017-05-10 13:49:40,338: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com NTP configuration configured->unconfigured (intentional=True) 140430198668816
[2017-05-10 13:49:40,342: INFO/job] Job 156: StateChangeJob complete, setting state unconfigured on lotus-32vm8.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-10 13:49:40,342: INFO/job] Job 156: not_deleted=True
[2017-05-10 13:49:40,342: INFO/job_scheduler] Job 156: complete_job: Updating cache
[2017-05-10 13:49:40,343: INFO/job_scheduler] Job 156: locked_item 140430198668816 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-10 13:49:40,346: INFO/job_scheduler] Job 156 completing (errored=False, cancelled=False)
[2017-05-10 13:49:40,379: INFO/job_scheduler] run_next: 0 runnable jobs of (18 pending, 10 tasked)
[2017-05-10 13:49:40,477: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:40,477: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:40,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:40,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:40,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4a476ea0-e051-40af-a454-2703e3259a08
[2017-05-10 13:49:40,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4a476ea0-e051-40af-a454-2703e3259a08
[2017-05-10 13:49:40,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4a476ea0-e051-40af-a454-2703e3259a08
[2017-05-10 13:49:40,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4a476ea0-e051-40af-a454-2703e3259a08
[2017-05-10 13:49:40,515: INFO/job_scheduler] Job 147 finished 1 steps successfully
[2017-05-10 13:49:40,550: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:40,550: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:40,552: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:40,552: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:40,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2745d06a-1ad4-4ae4-8385-aaeb5ac08188
[2017-05-10 13:49:40,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2745d06a-1ad4-4ae4-8385-aaeb5ac08188
[2017-05-10 13:49:40,588: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration started->stopped (intentional=True) 140430198627344
[2017-05-10 13:49:40,594: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2745d06a-1ad4-4ae4-8385-aaeb5ac08188
[2017-05-10 13:49:40,594: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2745d06a-1ad4-4ae4-8385-aaeb5ac08188
[2017-05-10 13:49:40,595: INFO/job_scheduler] Job 148 finished 1 steps successfully
[2017-05-10 13:49:40,602: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-10 13:49:40,607: INFO/job] Job 147: StateChangeJob complete, setting state stopped on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 13:49:40,607: INFO/job] Job 147: not_deleted=True
[2017-05-10 13:49:40,607: INFO/job_scheduler] Job 147: complete_job: Updating cache
[2017-05-10 13:49:40,607: INFO/job_scheduler] Job 147: locked_item 140430198627344 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 13:49:40,610: INFO/job_scheduler] Job 147 completing (errored=False, cancelled=False)
[2017-05-10 13:49:40,633: INFO/job_scheduler] run_next: 1 runnable jobs of (18 pending, 9 tasked)
[2017-05-10 13:49:40,642: INFO/job_scheduler] Job 149: RunJobThread.run
[2017-05-10 13:49:40,644: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com stop_corosync2 {}
[2017-05-10 13:49:40,670: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ce155b63-b350-411c-a049-c5c53920662b
[2017-05-10 13:49:40,670: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ce155b63-b350-411c-a049-c5c53920662b
[2017-05-10 13:49:40,721: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com RSyslog configuration configured->unconfigured (intentional=True) 140430198625168
[2017-05-10 13:49:40,730: INFO/job] Job 148: StateChangeJob complete, setting state unconfigured on lotus-32vm7.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-10 13:49:40,730: INFO/job] Job 148: not_deleted=True
[2017-05-10 13:49:40,730: INFO/job_scheduler] Job 148: complete_job: Updating cache
[2017-05-10 13:49:40,730: INFO/job_scheduler] Job 148: locked_item 140430198625168 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-10 13:49:40,735: INFO/job_scheduler] Job 148 completing (errored=False, cancelled=False)
[2017-05-10 13:49:40,782: INFO/job_scheduler] run_next: 0 runnable jobs of (17 pending, 9 tasked)
[2017-05-10 13:49:40,785: INFO/job_scheduler] run_next: 0 runnable jobs of (17 pending, 9 tasked)
[2017-05-10 13:49:40,790: INFO/job_scheduler] run_next: 0 runnable jobs of (17 pending, 9 tasked)
[2017-05-10 13:49:41,349: INFO/job_scheduler] run_next: 0 runnable jobs of (17 pending, 9 tasked)
[2017-05-10 13:49:41,368: INFO/job_scheduler] run_next: 0 runnable jobs of (17 pending, 9 tasked)
[2017-05-10 13:49:42,218: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:42,218: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:42,218: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:42,218: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:42,219: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f9676729-68ed-4876-acf6-2c928f5dcaea
[2017-05-10 13:49:42,219: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f9676729-68ed-4876-acf6-2c928f5dcaea
[2017-05-10 13:49:42,220: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f9676729-68ed-4876-acf6-2c928f5dcaea
[2017-05-10 13:49:42,220: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f9676729-68ed-4876-acf6-2c928f5dcaea
[2017-05-10 13:49:42,220: INFO/job_scheduler] Job 157 finished 1 steps successfully
[2017-05-10 13:49:42,321: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com RSyslog configuration configured->unconfigured (intentional=True) 140430198668880
[2017-05-10 13:49:42,324: INFO/job] Job 157: StateChangeJob complete, setting state unconfigured on lotus-32vm8.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-10 13:49:42,324: INFO/job] Job 157: not_deleted=True
[2017-05-10 13:49:42,324: INFO/job_scheduler] Job 157: complete_job: Updating cache
[2017-05-10 13:49:42,325: INFO/job_scheduler] Job 157: locked_item 140430198668880 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-10 13:49:42,328: INFO/job_scheduler] Job 157 completing (errored=False, cancelled=False)
[2017-05-10 13:49:42,346: INFO/job_scheduler] run_next: 0 runnable jobs of (17 pending, 8 tasked)
[2017-05-10 13:49:43,069: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:43,069: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:43,070: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:43,070: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:43,070: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 07a419fb-49c7-48b9-b3bc-3329f0d79227
[2017-05-10 13:49:43,070: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 07a419fb-49c7-48b9-b3bc-3329f0d79227
[2017-05-10 13:49:43,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 07a419fb-49c7-48b9-b3bc-3329f0d79227
[2017-05-10 13:49:43,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 07a419fb-49c7-48b9-b3bc-3329f0d79227
[2017-05-10 13:49:43,088: INFO/job_scheduler] Job 154 finished 1 steps successfully
[2017-05-10 13:49:43,145: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration started->stopped (intentional=True) 140430199164368
[2017-05-10 13:49:43,154: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-10 13:49:43,158: INFO/job] Job 154: StateChangeJob complete, setting state stopped on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 13:49:43,158: INFO/job] Job 154: not_deleted=True
[2017-05-10 13:49:43,158: INFO/job_scheduler] Job 154: complete_job: Updating cache
[2017-05-10 13:49:43,159: INFO/job_scheduler] Job 154: locked_item 140430199164368 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 13:49:43,169: INFO/job_scheduler] Job 154 completing (errored=False, cancelled=False)
[2017-05-10 13:49:43,197: INFO/job_scheduler] run_next: 1 runnable jobs of (17 pending, 7 tasked)
[2017-05-10 13:49:43,209: INFO/job_scheduler] Job 158: RunJobThread.run
[2017-05-10 13:49:43,210: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com stop_corosync2 {}
[2017-05-10 13:49:43,235: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d4c751cd-e5d2-435c-9299-6c88710f8cd4
[2017-05-10 13:49:43,235: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d4c751cd-e5d2-435c-9299-6c88710f8cd4
[2017-05-10 13:49:43,317: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 8 tasked)
[2017-05-10 13:49:43,334: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 8 tasked)
[2017-05-10 13:49:46,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:46,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:46,923: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:46,923: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:46,923: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 75838629-7413-4723-9eca-46d02fa7869d
[2017-05-10 13:49:46,923: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 75838629-7413-4723-9eca-46d02fa7869d
[2017-05-10 13:49:46,953: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 75838629-7413-4723-9eca-46d02fa7869d
[2017-05-10 13:49:46,953: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 75838629-7413-4723-9eca-46d02fa7869d
[2017-05-10 13:49:46,954: INFO/job_scheduler] Job 132 finished 1 steps successfully
[2017-05-10 13:49:46,995: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration started->stopped (intentional=True) 140430198620944
[2017-05-10 13:49:47,004: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-10 13:49:47,044: INFO/job] Job 132: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:49:47,044: INFO/job] Job 132: not_deleted=True
[2017-05-10 13:49:47,044: INFO/job_scheduler] Job 132: complete_job: Updating cache
[2017-05-10 13:49:47,044: INFO/job_scheduler] Job 132: locked_item 140430198620944 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 13:49:47,053: INFO/job_scheduler] Job 132 completing (errored=False, cancelled=False)
[2017-05-10 13:49:47,095: INFO/job_scheduler] run_next: 1 runnable jobs of (16 pending, 7 tasked)
[2017-05-10 13:49:47,100: INFO/job_scheduler] Job 134: RunJobThread.run
[2017-05-10 13:49:47,103: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unconfigure_corosync2 {'mcast_port': 53491, 'host_fqdn': u'lotus-32vm5.lotus.hpdd.lab.intel.com'}
[2017-05-10 13:49:47,128: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e9cbdac6-2d1e-4f10-99c2-acdfca43178b
[2017-05-10 13:49:47,128: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e9cbdac6-2d1e-4f10-99c2-acdfca43178b
[2017-05-10 13:49:48,326: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:48,326: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:48,327: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:48,327: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:48,327: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6ed8761d-9e67-409c-a505-d150431ecc7b
[2017-05-10 13:49:48,327: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6ed8761d-9e67-409c-a505-d150431ecc7b
[2017-05-10 13:49:48,333: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:48,333: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:48,333: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:48,333: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:48,333: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3f2301f9-4951-4f2b-bd15-6c0b1212359a
[2017-05-10 13:49:48,333: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3f2301f9-4951-4f2b-bd15-6c0b1212359a
[2017-05-10 13:49:48,338: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3f2301f9-4951-4f2b-bd15-6c0b1212359a
[2017-05-10 13:49:48,338: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3f2301f9-4951-4f2b-bd15-6c0b1212359a
[2017-05-10 13:49:48,340: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 13:49:48,351: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6ed8761d-9e67-409c-a505-d150431ecc7b
[2017-05-10 13:49:48,351: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6ed8761d-9e67-409c-a505-d150431ecc7b
[2017-05-10 13:49:48,354: INFO/job_scheduler] Job 141 finished 1 steps successfully
[2017-05-10 13:49:48,360: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e9dc5dfe-c14f-46dd-a730-ae8b542c4a50
[2017-05-10 13:49:48,360: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e9dc5dfe-c14f-46dd-a730-ae8b542c4a50
[2017-05-10 13:49:48,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:48,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:48,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:48,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:48,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e9dc5dfe-c14f-46dd-a730-ae8b542c4a50
[2017-05-10 13:49:48,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e9dc5dfe-c14f-46dd-a730-ae8b542c4a50
[2017-05-10 13:49:48,474: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e9dc5dfe-c14f-46dd-a730-ae8b542c4a50
[2017-05-10 13:49:48,474: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e9dc5dfe-c14f-46dd-a730-ae8b542c4a50
[2017-05-10 13:49:48,512: INFO/job_scheduler] Job 129 finished 2 steps successfully
[2017-05-10 13:49:48,703: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration started->stopped (intentional=True) 140430198704080
[2017-05-10 13:49:48,713: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-10 13:49:48,723: INFO/job] Job 141: StateChangeJob complete, setting state stopped on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:49:48,723: INFO/job] Job 141: not_deleted=True
[2017-05-10 13:49:48,725: INFO/job_scheduler] Job 141: complete_job: Updating cache
[2017-05-10 13:49:48,726: INFO/job_scheduler] Job 141: locked_item 140430198704080 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 13:49:48,745: INFO/job_scheduler] Job 141 completing (errored=False, cancelled=False)
[2017-05-10 13:49:48,774: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:48,774: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:48,774: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:48,774: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:48,774: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ce155b63-b350-411c-a049-c5c53920662b
[2017-05-10 13:49:48,774: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ce155b63-b350-411c-a049-c5c53920662b
[2017-05-10 13:49:48,793: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ce155b63-b350-411c-a049-c5c53920662b
[2017-05-10 13:49:48,793: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ce155b63-b350-411c-a049-c5c53920662b
[2017-05-10 13:49:48,794: INFO/job_scheduler] Job 149 finished 1 steps successfully
[2017-05-10 13:49:48,795: INFO/job_scheduler] run_next: 1 runnable jobs of (15 pending, 7 tasked)
[2017-05-10 13:49:48,799: INFO/job_scheduler] Job 142: RunJobThread.run
[2017-05-10 13:49:48,856: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_up->lnet_down (intentional=True) 140430198688080
[2017-05-10 13:49:48,861: INFO/job] Job 129: StateChangeJob complete, setting state lnet_down on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:49:48,862: INFO/job] Job 129: not_deleted=True
[2017-05-10 13:49:48,862: INFO/job_scheduler] Job 129: complete_job: Updating cache
[2017-05-10 13:49:48,862: INFO/job_scheduler] Job 129: locked_item 140430198688080 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:49:48,867: INFO/job_scheduler] Job 129 completing (errored=False, cancelled=False)
[2017-05-10 13:49:48,889: INFO/job_scheduler] run_next: 1 runnable jobs of (14 pending, 7 tasked)
[2017-05-10 13:49:48,895: INFO/job_scheduler] Job 131: RunJobThread.run
[2017-05-10 13:49:48,896: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unload_lnet {}
[2017-05-10 13:49:48,924: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d1b1e092-6342-44ce-818a-ec636e25b0d2
[2017-05-10 13:49:48,924: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d1b1e092-6342-44ce-818a-ec636e25b0d2
[2017-05-10 13:49:48,945: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration started->stopped (intentional=True) 140430198783632
[2017-05-10 13:49:48,956: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-10 13:49:48,992: INFO/job] Job 149: StateChangeJob complete, setting state stopped on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:49:48,992: INFO/job] Job 149: not_deleted=True
[2017-05-10 13:49:48,992: INFO/job_scheduler] Job 149: complete_job: Updating cache
[2017-05-10 13:49:48,993: INFO/job_scheduler] Job 149: locked_item 140430198783632 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 13:49:48,999: INFO/job_scheduler] Job 149 completing (errored=False, cancelled=False)
[2017-05-10 13:49:49,022: INFO/job_scheduler] run_next: 1 runnable jobs of (13 pending, 7 tasked)
[2017-05-10 13:49:49,029: INFO/job_scheduler] Job 152: RunJobThread.run
[2017-05-10 13:49:49,029: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com unconfigure_corosync2 {'mcast_port': 51853, 'host_fqdn': u'lotus-32vm7.lotus.hpdd.lab.intel.com'}
[2017-05-10 13:49:49,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 90fe08da-1f9c-41c2-b997-6a7a64680882
[2017-05-10 13:49:49,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 90fe08da-1f9c-41c2-b997-6a7a64680882
[2017-05-10 13:49:50,228: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 8 tasked)
[2017-05-10 13:49:50,260: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 8 tasked)
[2017-05-10 13:49:50,861: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 8 tasked)
[2017-05-10 13:49:50,881: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 8 tasked)
[2017-05-10 13:49:51,023: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:51,023: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:51,023: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:51,023: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:51,023: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d1b1e092-6342-44ce-818a-ec636e25b0d2
[2017-05-10 13:49:51,023: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d1b1e092-6342-44ce-818a-ec636e25b0d2
[2017-05-10 13:49:51,042: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d1b1e092-6342-44ce-818a-ec636e25b0d2
[2017-05-10 13:49:51,042: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d1b1e092-6342-44ce-818a-ec636e25b0d2
[2017-05-10 13:49:51,043: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 13:49:51,065: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8db8bc8b-591e-413b-a5f2-f7400a7753d0
[2017-05-10 13:49:51,065: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8db8bc8b-591e-413b-a5f2-f7400a7753d0
[2017-05-10 13:49:51,157: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:51,157: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:51,157: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:51,157: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:51,157: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8db8bc8b-591e-413b-a5f2-f7400a7753d0
[2017-05-10 13:49:51,157: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8db8bc8b-591e-413b-a5f2-f7400a7753d0
[2017-05-10 13:49:51,180: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8db8bc8b-591e-413b-a5f2-f7400a7753d0
[2017-05-10 13:49:51,180: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8db8bc8b-591e-413b-a5f2-f7400a7753d0
[2017-05-10 13:49:51,294: INFO/job_scheduler] Job 131 finished 2 steps successfully
[2017-05-10 13:49:51,327: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:51,327: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:51,328: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:51,328: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:51,328: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d4c751cd-e5d2-435c-9299-6c88710f8cd4
[2017-05-10 13:49:51,328: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d4c751cd-e5d2-435c-9299-6c88710f8cd4
[2017-05-10 13:49:51,344: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_unloaded (intentional=True) 140430198688080
[2017-05-10 13:49:51,352: INFO/job] Job 131: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:49:51,353: INFO/job] Job 131: not_deleted=True
[2017-05-10 13:49:51,353: INFO/job_scheduler] Job 131: complete_job: Updating cache
[2017-05-10 13:49:51,353: INFO/job_scheduler] Job 131: locked_item 140430198688080 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:49:51,356: INFO/job_scheduler] Job 131 completing (errored=False, cancelled=False)
[2017-05-10 13:49:51,357: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d4c751cd-e5d2-435c-9299-6c88710f8cd4
[2017-05-10 13:49:51,357: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d4c751cd-e5d2-435c-9299-6c88710f8cd4
[2017-05-10 13:49:51,357: INFO/job_scheduler] Job 158 finished 1 steps successfully
[2017-05-10 13:49:51,370: INFO/job_scheduler] run_next: 1 runnable jobs of (12 pending, 7 tasked)
[2017-05-10 13:49:51,391: INFO/job_scheduler] Job 133: RunJobThread.run
[2017-05-10 13:49:51,391: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unconfigure_lnet {}
[2017-05-10 13:49:51,407: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 95168bc9-ca7d-42d9-abc1-eb9d11863f3e
[2017-05-10 13:49:51,407: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 95168bc9-ca7d-42d9-abc1-eb9d11863f3e
[2017-05-10 13:49:51,436: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration started->stopped (intentional=True) 140430198669008
[2017-05-10 13:49:51,443: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-10 13:49:51,447: INFO/job] Job 158: StateChangeJob complete, setting state stopped on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:49:51,447: INFO/job] Job 158: not_deleted=True
[2017-05-10 13:49:51,449: INFO/job_scheduler] Job 158: complete_job: Updating cache
[2017-05-10 13:49:51,449: INFO/job_scheduler] Job 158: locked_item 140430198669008 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 13:49:51,470: INFO/job_scheduler] Job 158 completing (errored=False, cancelled=False)
[2017-05-10 13:49:51,504: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:51,504: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:51,504: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:51,504: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:51,505: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 95168bc9-ca7d-42d9-abc1-eb9d11863f3e
[2017-05-10 13:49:51,505: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 95168bc9-ca7d-42d9-abc1-eb9d11863f3e
[2017-05-10 13:49:51,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 95168bc9-ca7d-42d9-abc1-eb9d11863f3e
[2017-05-10 13:49:51,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 95168bc9-ca7d-42d9-abc1-eb9d11863f3e
[2017-05-10 13:49:51,525: INFO/job] Invalid result None fixed up on called to unconfigure_lnet with args {}
[2017-05-10 13:49:51,526: INFO/job_scheduler] Job 133 finished 1 steps successfully
[2017-05-10 13:49:51,570: INFO/job_scheduler] run_next: 1 runnable jobs of (11 pending, 7 tasked)
[2017-05-10 13:49:51,575: INFO/job_scheduler] Job 160: RunJobThread.run
[2017-05-10 13:49:51,577: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:51,577: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:51,577: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:51,577: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:51,577: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 59b7b8dc-b8ee-4e65-a7df-c864efeb3a71
[2017-05-10 13:49:51,577: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 59b7b8dc-b8ee-4e65-a7df-c864efeb3a71
[2017-05-10 13:49:51,622: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 59b7b8dc-b8ee-4e65-a7df-c864efeb3a71
[2017-05-10 13:49:51,622: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 59b7b8dc-b8ee-4e65-a7df-c864efeb3a71
[2017-05-10 13:49:51,624: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 13:49:51,662: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8b7e8cf7-7650-4980-b0ca-490efd6d479e
[2017-05-10 13:49:51,662: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8b7e8cf7-7650-4980-b0ca-490efd6d479e
[2017-05-10 13:49:51,682: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->unconfigured (intentional=True) 140430198688080
[2017-05-10 13:49:51,688: INFO/job] Job 133: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:49:51,688: INFO/job] Job 133: not_deleted=True
[2017-05-10 13:49:51,688: INFO/job_scheduler] Job 133: complete_job: Updating cache
[2017-05-10 13:49:51,688: INFO/job_scheduler] Job 133: locked_item 140430198688080 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:49:51,692: INFO/job_scheduler] Job 133 completing (errored=False, cancelled=False)
[2017-05-10 13:49:51,711: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 7 tasked)
[2017-05-10 13:49:52,750: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:52,750: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:52,750: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:52,750: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:52,750: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8b7e8cf7-7650-4980-b0ca-490efd6d479e
[2017-05-10 13:49:52,750: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8b7e8cf7-7650-4980-b0ca-490efd6d479e
[2017-05-10 13:49:52,776: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8b7e8cf7-7650-4980-b0ca-490efd6d479e
[2017-05-10 13:49:52,776: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8b7e8cf7-7650-4980-b0ca-490efd6d479e
[2017-05-10 13:49:52,997: INFO/job_scheduler] Job 146 finished 2 steps successfully
[2017-05-10 13:49:53,051: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration lnet_up->lnet_down (intentional=True) 140430198692560
[2017-05-10 13:49:53,058: INFO/job] Job 146: StateChangeJob complete, setting state lnet_down on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:49:53,058: INFO/job] Job 146: not_deleted=True
[2017-05-10 13:49:53,058: INFO/job_scheduler] Job 146: complete_job: Updating cache
[2017-05-10 13:49:53,059: INFO/job_scheduler] Job 146: locked_item 140430198692560 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:49:53,067: INFO/job_scheduler] Job 146 completing (errored=False, cancelled=False)
[2017-05-10 13:49:53,095: INFO/job_scheduler] run_next: 1 runnable jobs of (10 pending, 6 tasked)
[2017-05-10 13:49:53,111: INFO/job_scheduler] Job 150: RunJobThread.run
[2017-05-10 13:49:53,112: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com unload_lnet {}
[2017-05-10 13:49:53,150: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5f27aa86-afed-4e8c-a941-8fc0f0302ab1
[2017-05-10 13:49:53,150: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5f27aa86-afed-4e8c-a941-8fc0f0302ab1
[2017-05-10 13:49:53,406: INFO/job_scheduler] run_next: 0 runnable jobs of (9 pending, 7 tasked)
[2017-05-10 13:49:53,459: INFO/job_scheduler] run_next: 0 runnable jobs of (9 pending, 7 tasked)
[2017-05-10 13:49:54,159: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:54,159: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:54,160: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:54,160: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:54,160: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a9c8d3dd-3cd9-4641-a5c4-5b2ee629b5cf
[2017-05-10 13:49:54,160: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a9c8d3dd-3cd9-4641-a5c4-5b2ee629b5cf
[2017-05-10 13:49:54,161: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a9c8d3dd-3cd9-4641-a5c4-5b2ee629b5cf
[2017-05-10 13:49:54,161: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a9c8d3dd-3cd9-4641-a5c4-5b2ee629b5cf
[2017-05-10 13:49:54,163: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 13:49:54,183: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 378e61ba-366f-4a2c-8d70-819a8ffab647
[2017-05-10 13:49:54,183: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 378e61ba-366f-4a2c-8d70-819a8ffab647
[2017-05-10 13:49:54,285: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:54,285: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:54,285: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:54,285: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:54,285: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e9cbdac6-2d1e-4f10-99c2-acdfca43178b
[2017-05-10 13:49:54,285: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e9cbdac6-2d1e-4f10-99c2-acdfca43178b
[2017-05-10 13:49:54,320: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e9cbdac6-2d1e-4f10-99c2-acdfca43178b
[2017-05-10 13:49:54,320: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e9cbdac6-2d1e-4f10-99c2-acdfca43178b
[2017-05-10 13:49:54,321: INFO/job_scheduler] Job 134 finished 1 steps successfully
[2017-05-10 13:49:54,321: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com unconfigure_corosync2 {'mcast_port': 53491, 'host_fqdn': u'lotus-32vm6.lotus.hpdd.lab.intel.com'}
[2017-05-10 13:49:54,338: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5b95ea2c-adc2-4a55-8d78-aadbe21d4c66
[2017-05-10 13:49:54,338: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5b95ea2c-adc2-4a55-8d78-aadbe21d4c66
[2017-05-10 13:49:54,710: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->unconfigured (intentional=True) 140430198620944
[2017-05-10 13:49:54,719: INFO/job] Job 134: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:49:54,720: INFO/job] Job 134: not_deleted=True
[2017-05-10 13:49:54,720: INFO/job_scheduler] Job 134: complete_job: Updating cache
[2017-05-10 13:49:54,720: INFO/job_scheduler] Job 134: locked_item 140430198620944 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 13:49:54,725: INFO/job_scheduler] Job 134 completing (errored=False, cancelled=False)
[2017-05-10 13:49:54,742: INFO/job_scheduler] run_next: 1 runnable jobs of (9 pending, 6 tasked)
[2017-05-10 13:49:54,771: INFO/job_scheduler] Job 135: RunJobThread.run
[2017-05-10 13:49:54,774: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com deregister_server {}
[2017-05-10 13:49:54,803: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 629fa2da-8736-4462-b7fe-7af5aa9a452f
[2017-05-10 13:49:54,803: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 629fa2da-8736-4462-b7fe-7af5aa9a452f
[2017-05-10 13:49:54,878: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:54,878: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:54,878: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:54,878: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:49:54,878: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 629fa2da-8736-4462-b7fe-7af5aa9a452f
[2017-05-10 13:49:54,878: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 629fa2da-8736-4462-b7fe-7af5aa9a452f
[2017-05-10 13:49:54,921: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 629fa2da-8736-4462-b7fe-7af5aa9a452f
[2017-05-10 13:49:54,921: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 629fa2da-8736-4462-b7fe-7af5aa9a452f
[2017-05-10 13:49:54,957: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:49:54,958: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration unconfigured->removed (intentional=False) 140430198152080
[2017-05-10 13:49:54,971: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.ntp.NTPConfiguration'> 1
[2017-05-10 13:49:54,985: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:49:54,985: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->removed (intentional=False) 140430198730320
[2017-05-10 13:49:54,996: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.corosync2.Corosync2Configuration'> 1
[2017-05-10 13:49:55,010: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:49:55,010: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->removed (intentional=False) 140430198401296
[2017-05-10 13:49:55,017: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> 1
[2017-05-10 13:49:55,033: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:49:55,034: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration unconfigured->removed (intentional=False) 140430198268816
[2017-05-10 13:49:55,044: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> 1
[2017-05-10 13:49:55,059: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:49:55,059: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->removed (intentional=False) 140430198365904
[2017-05-10 13:49:55,069: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> 1
[2017-05-10 13:49:55,219: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:55,219: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:55,219: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:55,219: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:55,219: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5f27aa86-afed-4e8c-a941-8fc0f0302ab1
[2017-05-10 13:49:55,219: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5f27aa86-afed-4e8c-a941-8fc0f0302ab1
[2017-05-10 13:49:55,220: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5f27aa86-afed-4e8c-a941-8fc0f0302ab1
[2017-05-10 13:49:55,220: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5f27aa86-afed-4e8c-a941-8fc0f0302ab1
[2017-05-10 13:49:55,220: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 13:49:55,245: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 403eb099-4910-4d8a-b896-038d807f6a1f
[2017-05-10 13:49:55,245: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 403eb099-4910-4d8a-b896-038d807f6a1f
[2017-05-10 13:49:55,314: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:55,314: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:55,314: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:55,314: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:55,314: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 378e61ba-366f-4a2c-8d70-819a8ffab647
[2017-05-10 13:49:55,314: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 378e61ba-366f-4a2c-8d70-819a8ffab647
[2017-05-10 13:49:55,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:55,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:55,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:55,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:55,323: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 403eb099-4910-4d8a-b896-038d807f6a1f
[2017-05-10 13:49:55,323: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 403eb099-4910-4d8a-b896-038d807f6a1f
[2017-05-10 13:49:55,328: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 403eb099-4910-4d8a-b896-038d807f6a1f
[2017-05-10 13:49:55,328: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 403eb099-4910-4d8a-b896-038d807f6a1f
[2017-05-10 13:49:55,364: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 378e61ba-366f-4a2c-8d70-819a8ffab647
[2017-05-10 13:49:55,364: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 378e61ba-366f-4a2c-8d70-819a8ffab647
[2017-05-10 13:49:55,950: INFO/job_scheduler] Job 150 finished 2 steps successfully
[2017-05-10 13:49:56,012: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_unloaded (intentional=True) 140430198692560
[2017-05-10 13:49:56,018: INFO/job] Job 150: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:49:56,018: INFO/job] Job 150: not_deleted=True
[2017-05-10 13:49:56,018: INFO/job_scheduler] Job 150: complete_job: Updating cache
[2017-05-10 13:49:56,018: INFO/job_scheduler] Job 150: locked_item 140430198692560 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:49:56,027: INFO/job_scheduler] Job 150 completing (errored=False, cancelled=False)
[2017-05-10 13:49:56,060: INFO/job_scheduler] run_next: 1 runnable jobs of (8 pending, 6 tasked)
[2017-05-10 13:49:56,078: INFO/job_scheduler] Job 151: RunJobThread.run
[2017-05-10 13:49:56,078: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com unconfigure_lnet {}
[2017-05-10 13:49:56,108: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5a650814-198f-41ac-bb0d-b248490c781a
[2017-05-10 13:49:56,108: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5a650814-198f-41ac-bb0d-b248490c781a
[2017-05-10 13:49:56,165: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:56,165: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:56,165: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:56,165: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:56,165: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 90fe08da-1f9c-41c2-b997-6a7a64680882
[2017-05-10 13:49:56,165: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 90fe08da-1f9c-41c2-b997-6a7a64680882
[2017-05-10 13:49:56,171: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:56,171: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:56,171: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:56,171: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:56,171: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d5af240d-cbc5-460c-811f-9ca3569d5cc7
[2017-05-10 13:49:56,171: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d5af240d-cbc5-460c-811f-9ca3569d5cc7
[2017-05-10 13:49:56,173: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 90fe08da-1f9c-41c2-b997-6a7a64680882
[2017-05-10 13:49:56,173: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 90fe08da-1f9c-41c2-b997-6a7a64680882
[2017-05-10 13:49:56,174: INFO/job_scheduler] Job 152 finished 1 steps successfully
[2017-05-10 13:49:56,174: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com unconfigure_corosync2 {'mcast_port': 51853, 'host_fqdn': u'lotus-32vm8.lotus.hpdd.lab.intel.com'}
[2017-05-10 13:49:56,209: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3795b416-f42c-41fb-86c8-6ab6c15219ae
[2017-05-10 13:49:56,209: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3795b416-f42c-41fb-86c8-6ab6c15219ae
[2017-05-10 13:49:56,209: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:56,209: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:56,210: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:56,210: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:56,210: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5a650814-198f-41ac-bb0d-b248490c781a
[2017-05-10 13:49:56,210: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5a650814-198f-41ac-bb0d-b248490c781a
[2017-05-10 13:49:56,215: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d5af240d-cbc5-460c-811f-9ca3569d5cc7
[2017-05-10 13:49:56,215: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d5af240d-cbc5-460c-811f-9ca3569d5cc7
[2017-05-10 13:49:56,217: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 13:49:56,236: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5a650814-198f-41ac-bb0d-b248490c781a
[2017-05-10 13:49:56,236: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5a650814-198f-41ac-bb0d-b248490c781a
[2017-05-10 13:49:56,236: INFO/job] Invalid result None fixed up on called to unconfigure_lnet with args {}
[2017-05-10 13:49:56,237: INFO/job_scheduler] Job 151 finished 1 steps successfully
[2017-05-10 13:49:56,264: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f1af61b5-7575-43f2-8bb1-5f83bfc5ac11
[2017-05-10 13:49:56,264: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f1af61b5-7575-43f2-8bb1-5f83bfc5ac11
[2017-05-10 13:49:56,345: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->unconfigured (intentional=True) 140430198783632
[2017-05-10 13:49:56,352: INFO/job] Job 152: StateChangeJob complete, setting state unconfigured on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:49:56,352: INFO/job] Job 152: not_deleted=True
[2017-05-10 13:49:56,352: INFO/job_scheduler] Job 152: complete_job: Updating cache
[2017-05-10 13:49:56,352: INFO/job_scheduler] Job 152: locked_item 140430198783632 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 13:49:56,364: INFO/job_scheduler] Job 152 completing (errored=False, cancelled=False)
[2017-05-10 13:49:56,377: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 6 tasked)
[2017-05-10 13:49:56,423: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:56,423: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:56,423: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:56,423: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:56,423: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f1af61b5-7575-43f2-8bb1-5f83bfc5ac11
[2017-05-10 13:49:56,423: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f1af61b5-7575-43f2-8bb1-5f83bfc5ac11
[2017-05-10 13:49:56,434: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.host.ManagedHost'> 1
[2017-05-10 13:49:56,444: INFO/job_scheduler] Job 135 finished 2 steps successfully
[2017-05-10 13:49:56,446: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f1af61b5-7575-43f2-8bb1-5f83bfc5ac11
[2017-05-10 13:49:56,446: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f1af61b5-7575-43f2-8bb1-5f83bfc5ac11
[2017-05-10 13:49:56,665: INFO/job_scheduler] Job 155 finished 2 steps successfully
[2017-05-10 13:49:56,738: INFO/job_scheduler] Job 137 finished 2 steps successfully
[2017-05-10 13:49:56,889: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->unconfigured (intentional=True) 140430198692560
[2017-05-10 13:49:56,894: INFO/job] Job 151: StateChangeJob complete, setting state unconfigured on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:49:56,895: INFO/job] Job 151: not_deleted=True
[2017-05-10 13:49:56,895: INFO/job_scheduler] Job 151: complete_job: Updating cache
[2017-05-10 13:49:56,895: INFO/job_scheduler] Job 151: locked_item 140430198692560 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:49:56,898: INFO/job_scheduler] Job 151 completing (errored=False, cancelled=False)
[2017-05-10 13:49:56,919: INFO/job_scheduler] run_next: 1 runnable jobs of (7 pending, 5 tasked)
[2017-05-10 13:49:56,950: INFO/job_scheduler] Job 153: RunJobThread.run
[2017-05-10 13:49:56,952: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com deregister_server {}
[2017-05-10 13:49:56,970: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ca5f47ba-ca9a-4948-a4d1-8dacf23bff0d
[2017-05-10 13:49:56,970: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ca5f47ba-ca9a-4948-a4d1-8dacf23bff0d
[2017-05-10 13:49:57,000: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com managed->removed (intentional=True) 140430198687696
[2017-05-10 13:49:57,030: INFO/job] Job 135: StateChangeJob complete, setting state removed on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:49:57,030: INFO/job] Job 135: not_deleted=None
[2017-05-10 13:49:57,030: INFO/job_scheduler] Job 135: complete_job: Updating cache
[2017-05-10 13:49:57,030: INFO/job_scheduler] Job 135: locked_item 140430198687696 <class 'chroma_core.models.host.ManagedHost'> True None
[2017-05-10 13:49:57,030: INFO/job_scheduler] Job 135 completing (errored=False, cancelled=False)
[2017-05-10 13:49:57,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:57,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:57,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:57,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/97f8d31f-0a2d-4812-a9d6-a411e016c719
[2017-05-10 13:49:57,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ca5f47ba-ca9a-4948-a4d1-8dacf23bff0d
[2017-05-10 13:49:57,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ca5f47ba-ca9a-4948-a4d1-8dacf23bff0d
[2017-05-10 13:49:57,134: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 5 tasked)
[2017-05-10 13:49:57,144: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ca5f47ba-ca9a-4948-a4d1-8dacf23bff0d
[2017-05-10 13:49:57,144: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ca5f47ba-ca9a-4948-a4d1-8dacf23bff0d
[2017-05-10 13:49:57,169: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration lnet_up->lnet_down (intentional=True) 140430198793168
[2017-05-10 13:49:57,193: INFO/job] Job 155: StateChangeJob complete, setting state lnet_down on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:49:57,194: INFO/job] Job 155: not_deleted=True
[2017-05-10 13:49:57,194: INFO/job_scheduler] Job 155: complete_job: Updating cache
[2017-05-10 13:49:57,194: INFO/job_scheduler] Job 155: locked_item 140430198793168 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:49:57,202: INFO/job_scheduler] Job 155 completing (errored=False, cancelled=False)
[2017-05-10 13:49:57,219: INFO/job] Deleting dependent lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration for host lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:49:57,223: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->removed (intentional=False) 140430198795792
[2017-05-10 13:49:57,245: INFO/job_scheduler] run_next: 1 runnable jobs of (6 pending, 4 tasked)
[2017-05-10 13:49:57,255: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> 3
[2017-05-10 13:49:57,304: INFO/job_scheduler] Job 159: RunJobThread.run
[2017-05-10 13:49:57,304: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com unload_lnet {}
[2017-05-10 13:49:57,339: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 13044718-ec42-496d-b490-209eda4d9d88
[2017-05-10 13:49:57,339: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 13044718-ec42-496d-b490-209eda4d9d88
[2017-05-10 13:49:57,380: INFO/job] Deleting dependent lotus-32vm7.lotus.hpdd.lab.intel.com RSyslog configuration for host lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:49:57,380: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->removed (intentional=False) 140430198795152
[2017-05-10 13:49:57,425: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration lnet_up->lnet_down (intentional=True) 140430197676176
[2017-05-10 13:49:57,434: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> 3
[2017-05-10 13:49:57,448: INFO/job] Job 137: StateChangeJob complete, setting state lnet_down on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:49:57,448: INFO/job] Job 137: not_deleted=True
[2017-05-10 13:49:57,448: INFO/job_scheduler] Job 137: complete_job: Updating cache
[2017-05-10 13:49:57,448: INFO/job_scheduler] Job 137: locked_item 140430197676176 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:49:57,458: INFO/job_scheduler] Job 137 completing (errored=False, cancelled=False)
[2017-05-10 13:49:57,469: INFO/job] Deleting dependent lotus-32vm7.lotus.hpdd.lab.intel.com NTP configuration for host lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:49:57,469: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com NTP configuration unconfigured->removed (intentional=False) 140430198792272
[2017-05-10 13:49:57,476: INFO/job_scheduler] run_next: 1 runnable jobs of (5 pending, 4 tasked)
[2017-05-10 13:49:57,480: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.ntp.NTPConfiguration'> 3
[2017-05-10 13:49:57,495: INFO/job_scheduler] Job 140: RunJobThread.run
[2017-05-10 13:49:57,496: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com unload_lnet {}
[2017-05-10 13:49:57,506: INFO/job] Deleting dependent lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration for host lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:49:57,506: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration unconfigured->removed (intentional=False) 140430198794448
[2017-05-10 13:49:57,537: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> 3
[2017-05-10 13:49:57,556: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d3b6aac5-0ef0-4bac-9ec5-ef8117372bc2
[2017-05-10 13:49:57,556: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d3b6aac5-0ef0-4bac-9ec5-ef8117372bc2
[2017-05-10 13:49:57,576: INFO/job] Deleting dependent lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration for host lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:49:57,576: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->removed (intentional=False) 140430198793936
[2017-05-10 13:49:57,605: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.corosync2.Corosync2Configuration'> 3
[2017-05-10 13:49:58,550: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.host.ManagedHost'> 3
[2017-05-10 13:49:58,572: INFO/job_scheduler] Job 153 finished 2 steps successfully
[2017-05-10 13:49:58,608: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com managed->removed (intentional=True) 140430198690640
[2017-05-10 13:49:58,614: INFO/job] Job 153: StateChangeJob complete, setting state removed on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-05-10 13:49:58,614: INFO/job] Job 153: not_deleted=None
[2017-05-10 13:49:58,614: INFO/job_scheduler] Job 153: complete_job: Updating cache
[2017-05-10 13:49:58,614: INFO/job_scheduler] Job 153: locked_item 140430198690640 <class 'chroma_core.models.host.ManagedHost'> True None
[2017-05-10 13:49:58,614: INFO/job_scheduler] Job 153 completing (errored=False, cancelled=False)
[2017-05-10 13:49:58,662: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:58,662: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:58,662: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:58,662: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:58,663: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d3b6aac5-0ef0-4bac-9ec5-ef8117372bc2
[2017-05-10 13:49:58,663: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d3b6aac5-0ef0-4bac-9ec5-ef8117372bc2
[2017-05-10 13:49:58,688: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d3b6aac5-0ef0-4bac-9ec5-ef8117372bc2
[2017-05-10 13:49:58,688: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d3b6aac5-0ef0-4bac-9ec5-ef8117372bc2
[2017-05-10 13:49:58,689: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 13:49:58,723: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 4 tasked)
[2017-05-10 13:49:58,751: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9ba79dd7-c35a-4d8d-97c2-017adb001908
[2017-05-10 13:49:58,751: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9ba79dd7-c35a-4d8d-97c2-017adb001908
[2017-05-10 13:49:59,453: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:59,453: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:59,453: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:59,453: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:59,453: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 13044718-ec42-496d-b490-209eda4d9d88
[2017-05-10 13:49:59,453: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 13044718-ec42-496d-b490-209eda4d9d88
[2017-05-10 13:49:59,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 13044718-ec42-496d-b490-209eda4d9d88
[2017-05-10 13:49:59,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 13044718-ec42-496d-b490-209eda4d9d88
[2017-05-10 13:49:59,460: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 13:49:59,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 42f2ba70-5093-4dc7-bfe0-6916c5104207
[2017-05-10 13:49:59,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 42f2ba70-5093-4dc7-bfe0-6916c5104207
[2017-05-10 13:49:59,572: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:59,572: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:59,573: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:59,573: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:59,573: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 42f2ba70-5093-4dc7-bfe0-6916c5104207
[2017-05-10 13:49:59,573: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 42f2ba70-5093-4dc7-bfe0-6916c5104207
[2017-05-10 13:49:59,592: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 42f2ba70-5093-4dc7-bfe0-6916c5104207
[2017-05-10 13:49:59,592: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 42f2ba70-5093-4dc7-bfe0-6916c5104207
[2017-05-10 13:49:59,723: INFO/job_scheduler] Job 159 finished 2 steps successfully
[2017-05-10 13:49:59,757: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_unloaded (intentional=True) 140430198793168
[2017-05-10 13:49:59,769: INFO/job] Job 159: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:49:59,769: INFO/job] Job 159: not_deleted=True
[2017-05-10 13:49:59,769: INFO/job_scheduler] Job 159: complete_job: Updating cache
[2017-05-10 13:49:59,770: INFO/job_scheduler] Job 159: locked_item 140430198793168 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:49:59,773: INFO/job_scheduler] Job 159 completing (errored=False, cancelled=False)
[2017-05-10 13:49:59,793: INFO/job_scheduler] run_next: 1 runnable jobs of (4 pending, 3 tasked)
[2017-05-10 13:49:59,826: INFO/job_scheduler] Job 161: RunJobThread.run
[2017-05-10 13:49:59,827: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com unconfigure_lnet {}
[2017-05-10 13:49:59,855: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9e2d37bf-007d-4d0c-a497-9ce070af44a7
[2017-05-10 13:49:59,855: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9e2d37bf-007d-4d0c-a497-9ce070af44a7
[2017-05-10 13:49:59,858: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:59,858: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:59,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:59,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:49:59,861: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9ba79dd7-c35a-4d8d-97c2-017adb001908
[2017-05-10 13:49:59,861: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9ba79dd7-c35a-4d8d-97c2-017adb001908
[2017-05-10 13:49:59,869: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9ba79dd7-c35a-4d8d-97c2-017adb001908
[2017-05-10 13:49:59,869: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9ba79dd7-c35a-4d8d-97c2-017adb001908
[2017-05-10 13:49:59,919: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:59,919: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:59,920: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:59,920: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:49:59,920: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9e2d37bf-007d-4d0c-a497-9ce070af44a7
[2017-05-10 13:49:59,920: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9e2d37bf-007d-4d0c-a497-9ce070af44a7
[2017-05-10 13:49:59,932: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9e2d37bf-007d-4d0c-a497-9ce070af44a7
[2017-05-10 13:49:59,932: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9e2d37bf-007d-4d0c-a497-9ce070af44a7
[2017-05-10 13:49:59,932: INFO/job] Invalid result None fixed up on called to unconfigure_lnet with args {}
[2017-05-10 13:49:59,932: INFO/job_scheduler] Job 161 finished 1 steps successfully
[2017-05-10 13:49:59,957: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->unconfigured (intentional=True) 140430198793168
[2017-05-10 13:49:59,969: INFO/job] Job 161: StateChangeJob complete, setting state unconfigured on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:49:59,969: INFO/job] Job 161: not_deleted=True
[2017-05-10 13:49:59,969: INFO/job_scheduler] Job 161: complete_job: Updating cache
[2017-05-10 13:49:59,969: INFO/job_scheduler] Job 161: locked_item 140430198793168 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:49:59,973: INFO/job_scheduler] Job 161 completing (errored=False, cancelled=False)
[2017-05-10 13:49:59,991: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 3 tasked)
[2017-05-10 13:50:00,043: INFO/job_scheduler] Job 140 finished 2 steps successfully
[2017-05-10 13:50:00,083: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_unloaded (intentional=True) 140430197676176
[2017-05-10 13:50:00,089: INFO/job] Job 140: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:50:00,089: INFO/job] Job 140: not_deleted=True
[2017-05-10 13:50:00,089: INFO/job_scheduler] Job 140: complete_job: Updating cache
[2017-05-10 13:50:00,089: INFO/job_scheduler] Job 140: locked_item 140430197676176 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:50:00,092: INFO/job_scheduler] Job 140 completing (errored=False, cancelled=False)
[2017-05-10 13:50:00,110: INFO/job_scheduler] run_next: 1 runnable jobs of (3 pending, 2 tasked)
[2017-05-10 13:50:00,122: INFO/job_scheduler] Job 143: RunJobThread.run
[2017-05-10 13:50:00,123: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com unconfigure_lnet {}
[2017-05-10 13:50:00,156: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 52c5faca-78b3-4d17-903a-f93a7bd4a892
[2017-05-10 13:50:00,156: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 52c5faca-78b3-4d17-903a-f93a7bd4a892
[2017-05-10 13:50:00,208: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:50:00,208: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:50:00,218: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:50:00,218: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:50:00,219: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 52c5faca-78b3-4d17-903a-f93a7bd4a892
[2017-05-10 13:50:00,219: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 52c5faca-78b3-4d17-903a-f93a7bd4a892
[2017-05-10 13:50:00,223: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 52c5faca-78b3-4d17-903a-f93a7bd4a892
[2017-05-10 13:50:00,223: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 52c5faca-78b3-4d17-903a-f93a7bd4a892
[2017-05-10 13:50:00,223: INFO/job] Invalid result None fixed up on called to unconfigure_lnet with args {}
[2017-05-10 13:50:00,223: INFO/job_scheduler] Job 143 finished 1 steps successfully
[2017-05-10 13:50:00,291: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->unconfigured (intentional=True) 140430197676176
[2017-05-10 13:50:00,296: INFO/job] Job 143: StateChangeJob complete, setting state unconfigured on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:50:00,296: INFO/job] Job 143: not_deleted=True
[2017-05-10 13:50:00,296: INFO/job_scheduler] Job 143: complete_job: Updating cache
[2017-05-10 13:50:00,296: INFO/job_scheduler] Job 143: locked_item 140430197676176 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:50:00,299: INFO/job_scheduler] Job 143 completing (errored=False, cancelled=False)
[2017-05-10 13:50:00,316: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 2 tasked)
[2017-05-10 13:50:01,464: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:50:01,464: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:50:01,465: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:50:01,465: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:50:01,465: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5b95ea2c-adc2-4a55-8d78-aadbe21d4c66
[2017-05-10 13:50:01,465: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5b95ea2c-adc2-4a55-8d78-aadbe21d4c66
[2017-05-10 13:50:01,509: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5b95ea2c-adc2-4a55-8d78-aadbe21d4c66
[2017-05-10 13:50:01,509: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5b95ea2c-adc2-4a55-8d78-aadbe21d4c66
[2017-05-10 13:50:01,510: INFO/job_scheduler] Job 142 finished 1 steps successfully
[2017-05-10 13:50:01,591: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->unconfigured (intentional=True) 140430198704080
[2017-05-10 13:50:01,597: INFO/job] Job 142: StateChangeJob complete, setting state unconfigured on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:50:01,597: INFO/job] Job 142: not_deleted=True
[2017-05-10 13:50:01,597: INFO/job_scheduler] Job 142: complete_job: Updating cache
[2017-05-10 13:50:01,597: INFO/job_scheduler] Job 142: locked_item 140430198704080 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 13:50:01,603: INFO/job_scheduler] Job 142 completing (errored=False, cancelled=False)
[2017-05-10 13:50:01,616: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 1 tasked)
[2017-05-10 13:50:01,639: INFO/job_scheduler] Job 144: RunJobThread.run
[2017-05-10 13:50:01,641: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com deregister_server {}
[2017-05-10 13:50:01,660: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 44e7753b-84bb-4afd-b332-aa5682541f14
[2017-05-10 13:50:01,660: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 44e7753b-84bb-4afd-b332-aa5682541f14
[2017-05-10 13:50:01,736: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:50:01,736: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:50:01,737: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:50:01,737: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:50:01,737: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 44e7753b-84bb-4afd-b332-aa5682541f14
[2017-05-10 13:50:01,737: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 44e7753b-84bb-4afd-b332-aa5682541f14
[2017-05-10 13:50:01,773: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 44e7753b-84bb-4afd-b332-aa5682541f14
[2017-05-10 13:50:01,773: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 44e7753b-84bb-4afd-b332-aa5682541f14
[2017-05-10 13:50:01,798: INFO/job] Deleting dependent lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration for host lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:50:01,799: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration unconfigured->removed (intentional=False) 140430198689296
[2017-05-10 13:50:01,828: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> 2
[2017-05-10 13:50:01,851: INFO/job] Deleting dependent lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration for host lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:50:01,851: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->removed (intentional=False) 140430199167440
[2017-05-10 13:50:01,871: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.corosync2.Corosync2Configuration'> 2
[2017-05-10 13:50:01,893: INFO/job] Deleting dependent lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration for host lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:50:01,893: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration unconfigured->removed (intentional=False) 140430198637904
[2017-05-10 13:50:01,914: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.ntp.NTPConfiguration'> 2
[2017-05-10 13:50:01,936: INFO/job] Deleting dependent lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration for host lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:50:01,936: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->removed (intentional=False) 140429660354320
[2017-05-10 13:50:01,959: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> 2
[2017-05-10 13:50:01,984: INFO/job] Deleting dependent lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration for host lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:50:01,985: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->removed (intentional=False) 140430198049744
[2017-05-10 13:50:02,001: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> 2
[2017-05-10 13:50:02,580: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.host.ManagedHost'> 2
[2017-05-10 13:50:02,592: INFO/job_scheduler] Job 144 finished 2 steps successfully
[2017-05-10 13:50:02,627: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com managed->removed (intentional=True) 140430198269456
[2017-05-10 13:50:02,635: INFO/job] Job 144: StateChangeJob complete, setting state removed on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:50:02,635: INFO/job] Job 144: not_deleted=None
[2017-05-10 13:50:02,635: INFO/job_scheduler] Job 144: complete_job: Updating cache
[2017-05-10 13:50:02,635: INFO/job_scheduler] Job 144: locked_item 140430198269456 <class 'chroma_core.models.host.ManagedHost'> True None
[2017-05-10 13:50:02,635: INFO/job_scheduler] Job 144 completing (errored=False, cancelled=False)
[2017-05-10 13:50:02,724: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 13:50:03,329: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:50:03,329: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:50:03,329: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:50:03,329: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:50:03,329: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3795b416-f42c-41fb-86c8-6ab6c15219ae
[2017-05-10 13:50:03,329: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3795b416-f42c-41fb-86c8-6ab6c15219ae
[2017-05-10 13:50:03,330: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3795b416-f42c-41fb-86c8-6ab6c15219ae
[2017-05-10 13:50:03,330: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3795b416-f42c-41fb-86c8-6ab6c15219ae
[2017-05-10 13:50:03,330: INFO/job_scheduler] Job 160 finished 1 steps successfully
[2017-05-10 13:50:03,432: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->unconfigured (intentional=True) 140430198669008
[2017-05-10 13:50:03,440: INFO/job] Job 160: StateChangeJob complete, setting state unconfigured on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:50:03,440: INFO/job] Job 160: not_deleted=True
[2017-05-10 13:50:03,440: INFO/job_scheduler] Job 160: complete_job: Updating cache
[2017-05-10 13:50:03,440: INFO/job_scheduler] Job 160: locked_item 140430198669008 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 13:50:03,445: INFO/job_scheduler] Job 160 completing (errored=False, cancelled=False)
[2017-05-10 13:50:03,457: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 13:50:03,479: INFO/job_scheduler] Job 162: RunJobThread.run
[2017-05-10 13:50:03,482: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com deregister_server {}
[2017-05-10 13:50:03,503: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9c0b31f7-4c5f-4e78-9913-9bad06d344ac
[2017-05-10 13:50:03,503: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9c0b31f7-4c5f-4e78-9913-9bad06d344ac
[2017-05-10 13:50:03,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:50:03,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:50:03,600: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:50:03,600: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/91278809-f64b-4ec7-8847-724c8f44d4ca
[2017-05-10 13:50:03,600: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9c0b31f7-4c5f-4e78-9913-9bad06d344ac
[2017-05-10 13:50:03,600: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9c0b31f7-4c5f-4e78-9913-9bad06d344ac
[2017-05-10 13:50:03,617: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9c0b31f7-4c5f-4e78-9913-9bad06d344ac
[2017-05-10 13:50:03,617: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9c0b31f7-4c5f-4e78-9913-9bad06d344ac
[2017-05-10 13:50:03,657: INFO/job] Deleting dependent lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration for host lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:50:03,657: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->removed (intentional=False) 140430198761488
[2017-05-10 13:50:03,667: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> 4
[2017-05-10 13:50:03,684: INFO/job] Deleting dependent lotus-32vm8.lotus.hpdd.lab.intel.com NTP configuration for host lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:50:03,684: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com NTP configuration unconfigured->removed (intentional=False) 140430198688528
[2017-05-10 13:50:03,690: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.ntp.NTPConfiguration'> 4
[2017-05-10 13:50:03,708: INFO/job] Deleting dependent lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration for host lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:50:03,709: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->removed (intentional=False) 140430198281104
[2017-05-10 13:50:03,719: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.corosync2.Corosync2Configuration'> 4
[2017-05-10 13:50:03,733: INFO/job] Deleting dependent lotus-32vm8.lotus.hpdd.lab.intel.com RSyslog configuration for host lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:50:03,733: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->removed (intentional=False) 140430199243344
[2017-05-10 13:50:03,739: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> 4
[2017-05-10 13:50:03,757: INFO/job] Deleting dependent lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration for host lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:50:03,757: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration unconfigured->removed (intentional=False) 140430199242960
[2017-05-10 13:50:03,766: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> 4
[2017-05-10 13:50:04,370: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.host.ManagedHost'> 4
[2017-05-10 13:50:04,379: INFO/job_scheduler] Job 162 finished 2 steps successfully
[2017-05-10 13:50:04,404: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com managed->removed (intentional=True) 140430198761872
[2017-05-10 13:50:04,408: INFO/job] Job 162: StateChangeJob complete, setting state removed on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-05-10 13:50:04,408: INFO/job] Job 162: not_deleted=None
[2017-05-10 13:50:04,408: INFO/job_scheduler] Job 162: complete_job: Updating cache
[2017-05-10 13:50:04,408: INFO/job_scheduler] Job 162: locked_item 140430198761872 <class 'chroma_core.models.host.ManagedHost'> True None
[2017-05-10 13:50:04,409: INFO/job_scheduler] Job 162 completing (errored=False, cancelled=False)
[2017-05-10 13:50:04,455: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:50:29,676: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:50:29,677: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 13:50:29,691: INFO/command_plan] add_jobs: created Job 163 (Test for host connectivity)
[2017-05-10 13:50:29,728: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 13:50:29,742: INFO/job_scheduler] Job 163: RunJobThread.run
[2017-05-10 13:50:33,426: INFO/job_scheduler] Job 163 finished 1 steps successfully
[2017-05-10 13:50:33,482: INFO/job_scheduler] Job 163: complete_job: Updating cache
[2017-05-10 13:50:33,482: INFO/job_scheduler] Job 163 completing (errored=False, cancelled=False)
[2017-05-10 13:50:33,534: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:50:34,635: INFO/job_scheduler] Getting FQDN for 'lotus-32vm5': lotus-32vm5
lotus-32vm5.lotus.hpdd.lab.intel.com

[2017-05-10 13:50:34,678: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:50:34,687: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-5 transitioning from undeployed to unconfigured
[2017-05-10 13:50:34,725: INFO/command_plan] Created command 42 (Deploying agent to host lotus-32vm5.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-05-10 13:50:34,782: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 13:50:34,831: INFO/job_scheduler] Job 164: RunJobThread.run
[2017-05-10 13:50:35,440: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 13:50:37,061: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:50:37,062: INFO/job_scheduler]   Job 164 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-10 13:50:37,062: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 13:50:37,110: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:50:37,110: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bca91641-4133-4fc3-af92-8c8a7c08e3dc
[2017-05-10 13:50:37,113: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/ddb9e595-0304-44db-abc4-ebf80e0d1eb3
[2017-05-10 13:50:37,113: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/ddb9e595-0304-44db-abc4-ebf80e0d1eb3
[2017-05-10 13:50:37,138: INFO/job_scheduler] Job 164 finished 1 steps successfully
[2017-05-10 13:50:37,184: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 140430198511760
[2017-05-10 13:50:37,191: INFO/job] Job 164: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:50:37,191: INFO/job] Job 164: not_deleted=True
[2017-05-10 13:50:37,191: INFO/job_scheduler] Job 164: complete_job: Updating cache
[2017-05-10 13:50:37,192: INFO/job_scheduler] Job 164: locked_item 140430198511760 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 13:50:37,195: INFO/job_scheduler] Job 164 completing (errored=False, cancelled=False)
[2017-05-10 13:50:37,262: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-5
[2017-05-10 13:50:37,262: INFO/job_scheduler] _notify: Updating .boot_time of item 5 (lotus-32vm5.lotus.hpdd.lab.intel.com) from None to 2017-05-03 16:42:07+00:00
[2017-05-10 13:50:37,285: INFO/job_scheduler] _notify: Set boot_time=2017-05-03 16:42:07+00:00 on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-5) and saved
[2017-05-10 13:50:37,299: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:50:47,633: INFO/job_scheduler] _notify: Updating .properties of item 5 (lotus-32vm5.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-05-10 13:50:47,640: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-5) and saved
[2017-05-10 13:50:47,647: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:50:47,839: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:50:48,237: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:50:48,240: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-5 transitioning from unconfigured to managed
[2017-05-10 13:50:48,411: INFO/command_plan] Created command 43 (Setup managed host lotus-32vm5.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-05-10 13:50:48,510: INFO/job_scheduler] run_next: 1 runnable jobs of (11 pending, 0 tasked)
[2017-05-10 13:50:48,547: INFO/job_scheduler] Job 165: RunJobThread.run
[2017-05-10 13:50:48,548: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-05-10 13:50:48,573: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6ccc1855-dbc6-4ef4-9a93-2666fe34b77c
[2017-05-10 13:50:48,573: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6ccc1855-dbc6-4ef4-9a93-2666fe34b77c
[2017-05-10 13:50:49,634: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/ddb9e595-0304-44db-abc4-ebf80e0d1eb3
[2017-05-10 13:50:49,634: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/ddb9e595-0304-44db-abc4-ebf80e0d1eb3
[2017-05-10 13:50:49,635: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/ddb9e595-0304-44db-abc4-ebf80e0d1eb3
[2017-05-10 13:50:49,635: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/ddb9e595-0304-44db-abc4-ebf80e0d1eb3
[2017-05-10 13:50:49,635: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6ccc1855-dbc6-4ef4-9a93-2666fe34b77c
[2017-05-10 13:50:49,635: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6ccc1855-dbc6-4ef4-9a93-2666fe34b77c
[2017-05-10 13:50:49,644: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6ccc1855-dbc6-4ef4-9a93-2666fe34b77c
[2017-05-10 13:50:49,644: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6ccc1855-dbc6-4ef4-9a93-2666fe34b77c
[2017-05-10 13:50:49,645: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-10 13:50:49.645076+00:00:
[2017-05-10 13:50:49,645: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-10 13:50:49.645076+00:00:
[2017-05-10 13:50:49,645: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 13:50:49,645: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 13:50:49,673: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-10 13:50:49.673663+00:00:
[2017-05-10 13:50:49,673: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-05-10 13:50:49.673663+00:00:
[2017-05-10 13:50:49,674: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 13:50:49,674: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 13:50:49,688: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:50:49,688: INFO/job_scheduler]   Job 165 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 13:50:49,688: INFO/job_scheduler]   Job 166 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:50:49,688: INFO/job_scheduler]   Job 167 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:50:49,688: INFO/job_scheduler]   Job 168 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:50:49,688: INFO/job_scheduler]   Job 169 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:50:49,689: INFO/job_scheduler]   Job 173 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:50:49,689: INFO/job_scheduler]   Job 175 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:50:49,689: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 1 tasked)
[2017-05-10 13:50:49,693: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:50:49,693: INFO/job_scheduler]   Job 165 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 13:50:49,693: INFO/job_scheduler]   Job 166 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:50:49,693: INFO/job_scheduler]   Job 167 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:50:49,693: INFO/job_scheduler]   Job 168 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:50:49,694: INFO/job_scheduler]   Job 169 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:50:49,694: INFO/job_scheduler]   Job 173 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:50:49,694: INFO/job_scheduler]   Job 175 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:50:49,694: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 1 tasked)
[2017-05-10 13:50:49,695: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com restart_agent {}
[2017-05-10 13:50:49,708: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d938392d-2495-4cb1-9b09-de88cf410589
[2017-05-10 13:50:49,708: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d938392d-2495-4cb1-9b09-de88cf410589
[2017-05-10 13:50:49,820: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/ddb9e595-0304-44db-abc4-ebf80e0d1eb3
[2017-05-10 13:50:49,820: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/ddb9e595-0304-44db-abc4-ebf80e0d1eb3
[2017-05-10 13:50:49,821: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/ddb9e595-0304-44db-abc4-ebf80e0d1eb3
[2017-05-10 13:50:49,821: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/ddb9e595-0304-44db-abc4-ebf80e0d1eb3
[2017-05-10 13:50:49,821: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d938392d-2495-4cb1-9b09-de88cf410589
[2017-05-10 13:50:49,821: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d938392d-2495-4cb1-9b09-de88cf410589
[2017-05-10 13:50:49,822: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d938392d-2495-4cb1-9b09-de88cf410589
[2017-05-10 13:50:49,822: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d938392d-2495-4cb1-9b09-de88cf410589
[2017-05-10 13:50:49,822: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old ddb9e595-0304-44db-abc4-ebf80e0d1eb3)
[2017-05-10 13:50:49,822: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old ddb9e595-0304-44db-abc4-ebf80e0d1eb3)
[2017-05-10 13:51:19,601: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/ddb9e595-0304-44db-abc4-ebf80e0d1eb3
[2017-05-10 13:51:19,601: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/ddb9e595-0304-44db-abc4-ebf80e0d1eb3
[2017-05-10 13:51:19,601: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session ddb9e595-0304-44db-abc4-ebf80e0d1eb3 because session terminated
[2017-05-10 13:51:19,601: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session ddb9e595-0304-44db-abc4-ebf80e0d1eb3 because session terminated
[2017-05-10 13:51:19,608: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:51:19,608: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:51:19,852: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new 6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:51:19,852: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new 6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:51:19,853: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 13:51:19,869: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c67e35f9-e022-436e-b1c7-b200002a16ae
[2017-05-10 13:51:19,869: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c67e35f9-e022-436e-b1c7-b200002a16ae
[2017-05-10 13:51:19,934: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:51:19,934: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:51:19,934: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:51:19,934: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:51:19,934: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c67e35f9-e022-436e-b1c7-b200002a16ae
[2017-05-10 13:51:19,934: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c67e35f9-e022-436e-b1c7-b200002a16ae
[2017-05-10 13:51:19,934: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c67e35f9-e022-436e-b1c7-b200002a16ae
[2017-05-10 13:51:19,934: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c67e35f9-e022-436e-b1c7-b200002a16ae
[2017-05-10 13:51:19,935: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-05-10 13:51:19,952: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2cebac36-c528-4718-82a9-e477e0dca753
[2017-05-10 13:51:19,952: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2cebac36-c528-4718-82a9-e477e0dca753
[2017-05-10 13:51:29,545: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:51:29,545: INFO/job_scheduler]   Job 165 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 13:51:29,546: INFO/job_scheduler]   Job 166 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:51:29,546: INFO/job_scheduler]   Job 167 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:51:29,546: INFO/job_scheduler]   Job 168 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:51:29,546: INFO/job_scheduler]   Job 169 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:51:29,546: INFO/job_scheduler]   Job 173 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:51:29,547: INFO/job_scheduler]   Job 175 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:51:29,547: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 1 tasked)
[2017-05-10 13:52:06,934: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:06,934: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:06,935: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:06,935: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:06,935: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2cebac36-c528-4718-82a9-e477e0dca753
[2017-05-10 13:52:06,935: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2cebac36-c528-4718-82a9-e477e0dca753
[2017-05-10 13:52:06,940: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2cebac36-c528-4718-82a9-e477e0dca753
[2017-05-10 13:52:06,940: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2cebac36-c528-4718-82a9-e477e0dca753
[2017-05-10 13:52:07,768: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com install_packages {'repos': [u'e2fsprogs', u'iml-agent', u'lustre', u'zfs'], 'packages': [u'chroma-agent-management', u'kernel-devel-3.10.0-514.10.2.el7_lustre', u'lustre-osd-zfs', u'lustre-osd-ldiskfs', u'lustre-modules', u'lustre', u'zfs']}
[2017-05-10 13:52:07,786: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f154e70c-33b0-43ce-bd8b-887e568d9631
[2017-05-10 13:52:07,786: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f154e70c-33b0-43ce-bd8b-887e568d9631
[2017-05-10 13:52:25,268: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:25,268: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:25,268: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:25,268: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:25,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f154e70c-33b0-43ce-bd8b-887e568d9631
[2017-05-10 13:52:25,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f154e70c-33b0-43ce-bd8b-887e568d9631
[2017-05-10 13:52:25,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f154e70c-33b0-43ce-bd8b-887e568d9631
[2017-05-10 13:52:25,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f154e70c-33b0-43ce-bd8b-887e568d9631
[2017-05-10 13:52:25,822: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com kernel_status {}
[2017-05-10 13:52:25,838: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 834e62cc-707e-4a6b-b362-3ad44c7a5ea2
[2017-05-10 13:52:25,838: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 834e62cc-707e-4a6b-b362-3ad44c7a5ea2
[2017-05-10 13:52:27,946: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:27,946: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:27,947: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:27,947: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:27,947: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 834e62cc-707e-4a6b-b362-3ad44c7a5ea2
[2017-05-10 13:52:27,947: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 834e62cc-707e-4a6b-b362-3ad44c7a5ea2
[2017-05-10 13:52:27,952: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 834e62cc-707e-4a6b-b362-3ad44c7a5ea2
[2017-05-10 13:52:27,952: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 834e62cc-707e-4a6b-b362-3ad44c7a5ea2
[2017-05-10 13:52:27,952: INFO/job_scheduler] Job 165 finished 4 steps successfully
[2017-05-10 13:52:28,053: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 140430198510992
[2017-05-10 13:52:28,058: INFO/job] Job 165: StateChangeJob complete, setting state packages_installed on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:52:28,058: INFO/job] Job 165: not_deleted=True
[2017-05-10 13:52:28,058: INFO/job_scheduler] Job 165: complete_job: Updating cache
[2017-05-10 13:52:28,058: INFO/job_scheduler] Job 165: locked_item 140430199263312 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 13:52:28,061: INFO/job_scheduler] Job 165 completing (errored=False, cancelled=False)
[2017-05-10 13:52:28,072: INFO/job_scheduler] run_next: 4 runnable jobs of (10 pending, 0 tasked)
[2017-05-10 13:52:28,097: INFO/job_scheduler] Job 166: RunJobThread.run
[2017-05-10 13:52:28,098: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-05-10 13:52:28,109: INFO/job_scheduler] Job 167: open DB connection during completion
[2017-05-10 13:52:28,128: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b8f57e87-69eb-4e88-9ffd-e5f923495395
[2017-05-10 13:52:28,128: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b8f57e87-69eb-4e88-9ffd-e5f923495395
[2017-05-10 13:52:28,130: INFO/job_scheduler] Job 168: RunJobThread.run
[2017-05-10 13:52:28,134: INFO/job_scheduler] Job 169: RunJobThread.run
[2017-05-10 13:52:28,135: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-05-10 13:52:28,146: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-05-10 13:52:28,158: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 60ba4da1-07c4-4b49-a936-e10fc621e882
[2017-05-10 13:52:28,158: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 60ba4da1-07c4-4b49-a936-e10fc621e882
[2017-05-10 13:52:28,169: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 140430198687696
[2017-05-10 13:52:28,180: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 13:52:28,189: INFO/job] Job 167: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:52:28,189: INFO/job] Job 167: not_deleted=True
[2017-05-10 13:52:28,189: INFO/job_scheduler] Job 167: complete_job: Updating cache
[2017-05-10 13:52:28,190: INFO/job_scheduler] Job 167: locked_item 140430198687696 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:52:28,207: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 10f1b63d-a58e-4ac6-b704-518ddb255b32
[2017-05-10 13:52:28,207: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 10f1b63d-a58e-4ac6-b704-518ddb255b32
[2017-05-10 13:52:28,213: INFO/job_scheduler] Job 167 completing (errored=False, cancelled=False)
[2017-05-10 13:52:28,257: INFO/job_scheduler] run_next: 1 runnable jobs of (6 pending, 3 tasked)
[2017-05-10 13:52:28,273: INFO/job_scheduler] Job 171: RunJobThread.run
[2017-05-10 13:52:28,274: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com load_lnet {}
[2017-05-10 13:52:28,307: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0b32417f-e02c-4336-96ca-5afe079f6d7b
[2017-05-10 13:52:28,307: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0b32417f-e02c-4336-96ca-5afe079f6d7b
[2017-05-10 13:52:29,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:29,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:29,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:29,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:29,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0b32417f-e02c-4336-96ca-5afe079f6d7b
[2017-05-10 13:52:29,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0b32417f-e02c-4336-96ca-5afe079f6d7b
[2017-05-10 13:52:29,475: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0b32417f-e02c-4336-96ca-5afe079f6d7b
[2017-05-10 13:52:29,475: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0b32417f-e02c-4336-96ca-5afe079f6d7b
[2017-05-10 13:52:29,476: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 13:52:29,497: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 24a0b54b-c8fb-433b-890f-0d77cbc9865e
[2017-05-10 13:52:29,497: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 24a0b54b-c8fb-433b-890f-0d77cbc9865e
[2017-05-10 13:52:29,698: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:29,698: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:29,698: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:29,698: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:29,699: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 24a0b54b-c8fb-433b-890f-0d77cbc9865e
[2017-05-10 13:52:29,699: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 24a0b54b-c8fb-433b-890f-0d77cbc9865e
[2017-05-10 13:52:29,711: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 24a0b54b-c8fb-433b-890f-0d77cbc9865e
[2017-05-10 13:52:29,711: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 24a0b54b-c8fb-433b-890f-0d77cbc9865e
[2017-05-10 13:52:29,979: INFO/job_scheduler] Job 171 finished 2 steps successfully
[2017-05-10 13:52:30,096: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 140430198687696
[2017-05-10 13:52:30,110: INFO/job] Job 171: StateChangeJob complete, setting state lnet_down on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:52:30,110: INFO/job] Job 171: not_deleted=True
[2017-05-10 13:52:30,110: INFO/job_scheduler] Job 171: complete_job: Updating cache
[2017-05-10 13:52:30,110: INFO/job_scheduler] Job 171: locked_item 140430198687696 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:52:30,114: INFO/job_scheduler] Job 171 completing (errored=False, cancelled=False)
[2017-05-10 13:52:30,132: INFO/job_scheduler] run_next: 1 runnable jobs of (5 pending, 3 tasked)
[2017-05-10 13:52:30,152: INFO/job_scheduler] Job 172: RunJobThread.run
[2017-05-10 13:52:30,153: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_lnet {}
[2017-05-10 13:52:30,186: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 50552190-68b8-4462-a94e-4a0634cc5675
[2017-05-10 13:52:30,186: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 50552190-68b8-4462-a94e-4a0634cc5675
[2017-05-10 13:52:31,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:31,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:31,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:31,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:31,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 50552190-68b8-4462-a94e-4a0634cc5675
[2017-05-10 13:52:31,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 50552190-68b8-4462-a94e-4a0634cc5675
[2017-05-10 13:52:31,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 50552190-68b8-4462-a94e-4a0634cc5675
[2017-05-10 13:52:31,451: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 50552190-68b8-4462-a94e-4a0634cc5675
[2017-05-10 13:52:31,452: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 13:52:31,480: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 898c0576-8a71-4225-a5e2-0122d3cac980
[2017-05-10 13:52:31,480: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 898c0576-8a71-4225-a5e2-0122d3cac980
[2017-05-10 13:52:32,696: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:32,696: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:32,697: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:32,697: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:32,697: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 898c0576-8a71-4225-a5e2-0122d3cac980
[2017-05-10 13:52:32,697: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 898c0576-8a71-4225-a5e2-0122d3cac980
[2017-05-10 13:52:32,698: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 898c0576-8a71-4225-a5e2-0122d3cac980
[2017-05-10 13:52:32,698: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 898c0576-8a71-4225-a5e2-0122d3cac980
[2017-05-10 13:52:32,739: INFO/job_scheduler] Job 172 finished 2 steps successfully
[2017-05-10 13:52:32,752: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 140430198687696
[2017-05-10 13:52:32,757: INFO/job] Job 172: StateChangeJob complete, setting state lnet_up on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:52:32,757: INFO/job] Job 172: not_deleted=True
[2017-05-10 13:52:32,757: INFO/job_scheduler] Job 172: complete_job: Updating cache
[2017-05-10 13:52:32,757: INFO/job_scheduler] Job 172: locked_item 140430198687696 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:52:32,760: INFO/job_scheduler] Job 172 completing (errored=False, cancelled=False)
[2017-05-10 13:52:32,770: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 3 tasked)
[2017-05-10 13:52:33,530: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:33,530: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:33,531: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:33,531: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:33,531: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 10f1b63d-a58e-4ac6-b704-518ddb255b32
[2017-05-10 13:52:33,531: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 10f1b63d-a58e-4ac6-b704-518ddb255b32
[2017-05-10 13:52:33,533: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:33,533: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:33,533: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:33,533: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:52:33,534: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b8f57e87-69eb-4e88-9ffd-e5f923495395
[2017-05-10 13:52:33,534: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b8f57e87-69eb-4e88-9ffd-e5f923495395
[2017-05-10 13:52:33,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b8f57e87-69eb-4e88-9ffd-e5f923495395
[2017-05-10 13:52:33,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b8f57e87-69eb-4e88-9ffd-e5f923495395
[2017-05-10 13:52:33,546: INFO/job_scheduler] Job 166 finished 1 steps successfully
[2017-05-10 13:52:33,579: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 10f1b63d-a58e-4ac6-b704-518ddb255b32
[2017-05-10 13:52:33,579: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 10f1b63d-a58e-4ac6-b704-518ddb255b32
[2017-05-10 13:52:33,580: INFO/job_scheduler] Job 168 finished 1 steps successfully
[2017-05-10 13:52:33,677: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 140430198687760
[2017-05-10 13:52:33,681: INFO/job] Job 166: StateChangeJob complete, setting state configured on lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-10 13:52:33,681: INFO/job] Job 166: not_deleted=True
[2017-05-10 13:52:33,681: INFO/job_scheduler] Job 166: complete_job: Updating cache
[2017-05-10 13:52:33,681: INFO/job_scheduler] Job 166: locked_item 140430198687760 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-10 13:52:33,684: INFO/job_scheduler] Job 166 completing (errored=False, cancelled=False)
[2017-05-10 13:52:33,703: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-05-10 13:52:33,752: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 140430198687376
[2017-05-10 13:52:33,757: INFO/job] Job 168: StateChangeJob complete, setting state configured on lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-10 13:52:33,757: INFO/job] Job 168: not_deleted=True
[2017-05-10 13:52:33,757: INFO/job_scheduler] Job 168: complete_job: Updating cache
[2017-05-10 13:52:33,757: INFO/job_scheduler] Job 168: locked_item 140430198687376 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-10 13:52:33,760: INFO/job_scheduler] Job 168 completing (errored=False, cancelled=False)
[2017-05-10 13:52:33,796: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 1 tasked)
[2017-05-10 13:53:10,430: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:53:10,430: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:53:10,430: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:53:10,430: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:53:10,430: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 60ba4da1-07c4-4b49-a936-e10fc621e882
[2017-05-10 13:53:10,430: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 60ba4da1-07c4-4b49-a936-e10fc621e882
[2017-05-10 13:53:10,440: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 60ba4da1-07c4-4b49-a936-e10fc621e882
[2017-05-10 13:53:10,440: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 60ba4da1-07c4-4b49-a936-e10fc621e882
[2017-05-10 13:53:10,441: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.224', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-05-10 13:53:10,456: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 324e5387-8e88-48d8-83b4-103c3e23093a
[2017-05-10 13:53:10,456: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 324e5387-8e88-48d8-83b4-103c3e23093a
[2017-05-10 13:53:10,552: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:53:10,552: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:53:10,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:53:10,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:53:10,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 324e5387-8e88-48d8-83b4-103c3e23093a
[2017-05-10 13:53:10,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 324e5387-8e88-48d8-83b4-103c3e23093a
[2017-05-10 13:53:10,571: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 324e5387-8e88-48d8-83b4-103c3e23093a
[2017-05-10 13:53:10,571: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 324e5387-8e88-48d8-83b4-103c3e23093a
[2017-05-10 13:53:10,590: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 62535, 'pcs_password': 'HSNG6u6mmlA7xKcMtNZS'}
[2017-05-10 13:53:10,608: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 405f8b5b-074f-4db9-9fca-068f215593fe
[2017-05-10 13:53:10,608: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 405f8b5b-074f-4db9-9fca-068f215593fe
[2017-05-10 13:53:22,723: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:53:22,723: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:53:22,723: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:53:22,723: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:53:22,723: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 405f8b5b-074f-4db9-9fca-068f215593fe
[2017-05-10 13:53:22,723: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 405f8b5b-074f-4db9-9fca-068f215593fe
[2017-05-10 13:53:22,728: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 405f8b5b-074f-4db9-9fca-068f215593fe
[2017-05-10 13:53:22,728: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 405f8b5b-074f-4db9-9fca-068f215593fe
[2017-05-10 13:53:22,729: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': True, 'mcast_port': 62535, 'new_node_fqdn': u'lotus-32vm5.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'HSNG6u6mmlA7xKcMtNZS'}
[2017-05-10 13:53:22,743: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9ab5c919-668e-4464-9ffa-23523dd059eb
[2017-05-10 13:53:22,743: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9ab5c919-668e-4464-9ffa-23523dd059eb
[2017-05-10 13:53:28,822: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:53:28,822: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:53:28,822: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:53:28,822: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:53:28,823: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9ab5c919-668e-4464-9ffa-23523dd059eb
[2017-05-10 13:53:28,823: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9ab5c919-668e-4464-9ffa-23523dd059eb
[2017-05-10 13:53:28,862: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9ab5c919-668e-4464-9ffa-23523dd059eb
[2017-05-10 13:53:28,862: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9ab5c919-668e-4464-9ffa-23523dd059eb
[2017-05-10 13:53:28,863: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 13:53:28.863487+00:00:
[2017-05-10 13:53:28,863: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 13:53:28.863487+00:00:
[2017-05-10 13:53:28,863: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->62535
[2017-05-10 13:53:28,863: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->62535
[2017-05-10 13:53:28,865: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 13:53:28,865: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 13:53:28,881: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:53:28,882: INFO/job_scheduler]   Job 169 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-05-10 13:53:28,882: INFO/job_scheduler]   Job 170 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-05-10 13:53:28,882: INFO/job_scheduler]   Job 173 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:53:28,882: INFO/job_scheduler]   Job 174 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:53:28,882: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 1 tasked)
[2017-05-10 13:53:28,884: INFO/job_scheduler] Job 169 finished 1 steps successfully
[2017-05-10 13:53:28,903: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 140430199167952
[2017-05-10 13:53:28,911: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-10 13:53:28,920: INFO/job] Job 169: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:53:28,921: INFO/job] Job 169: not_deleted=True
[2017-05-10 13:53:28,922: INFO/job_scheduler] Job 169: complete_job: Updating cache
[2017-05-10 13:53:28,923: INFO/job_scheduler] Job 169: locked_item 140430199167952 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 13:53:28,943: INFO/job_scheduler] Job 169 completing (errored=False, cancelled=False)
[2017-05-10 13:53:28,974: INFO/job_scheduler] run_next: 1 runnable jobs of (4 pending, 0 tasked)
[2017-05-10 13:53:28,993: INFO/job_scheduler] Job 170: RunJobThread.run
[2017-05-10 13:53:28,994: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-05-10 13:53:29,030: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 67f5c6ca-7d23-4f15-a2e7-daa48522b952
[2017-05-10 13:53:29,030: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 67f5c6ca-7d23-4f15-a2e7-daa48522b952
[2017-05-10 13:53:34,305: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-05-10 13:53:34,323: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-05-10 13:53:37,166: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:53:37,166: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:53:37,166: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:53:37,166: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:53:37,166: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 67f5c6ca-7d23-4f15-a2e7-daa48522b952
[2017-05-10 13:53:37,166: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 67f5c6ca-7d23-4f15-a2e7-daa48522b952
[2017-05-10 13:53:37,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 67f5c6ca-7d23-4f15-a2e7-daa48522b952
[2017-05-10 13:53:37,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 67f5c6ca-7d23-4f15-a2e7-daa48522b952
[2017-05-10 13:53:37,198: INFO/job_scheduler] Job 170 finished 1 steps successfully
[2017-05-10 13:53:37,277: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 140430199167952
[2017-05-10 13:53:37,315: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 13:53:37,331: INFO/job] Job 170: StateChangeJob complete, setting state started on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:53:37,331: INFO/job] Job 170: not_deleted=True
[2017-05-10 13:53:37,332: INFO/job_scheduler] Job 170: complete_job: Updating cache
[2017-05-10 13:53:37,332: INFO/job_scheduler] Job 170: locked_item 140430199167952 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 13:53:37,338: INFO/job_scheduler] Job 170 completing (errored=False, cancelled=False)
[2017-05-10 13:53:37,356: INFO/job_scheduler] run_next: 1 runnable jobs of (3 pending, 0 tasked)
[2017-05-10 13:53:37,390: INFO/job_scheduler] Job 173: RunJobThread.run
[2017-05-10 13:53:37,391: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 13:53:37,422: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e931eb81-cc5f-4a86-9736-2d5f4f5d2c1e
[2017-05-10 13:53:37,422: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e931eb81-cc5f-4a86-9736-2d5f4f5d2c1e
[2017-05-10 13:53:44,345: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-05-10 13:53:44,362: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-05-10 13:53:44,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:53:44,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:53:44,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:53:44,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:53:44,546: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e931eb81-cc5f-4a86-9736-2d5f4f5d2c1e
[2017-05-10 13:53:44,546: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e931eb81-cc5f-4a86-9736-2d5f4f5d2c1e
[2017-05-10 13:53:44,593: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e931eb81-cc5f-4a86-9736-2d5f4f5d2c1e
[2017-05-10 13:53:44,593: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e931eb81-cc5f-4a86-9736-2d5f4f5d2c1e
[2017-05-10 13:53:44,594: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-05-10 13:53:44,607: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4a7e1b15-4278-4dc1-918c-9d19d5af2c4f
[2017-05-10 13:53:44,607: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4a7e1b15-4278-4dc1-918c-9d19d5af2c4f
[2017-05-10 13:53:54,328: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-05-10 13:53:54,345: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-05-10 13:54:14,400: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-05-10 13:54:14,417: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-05-10 13:54:34,497: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:54:34,498: INFO/job_scheduler]   Job 173 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:54:34,498: INFO/job_scheduler]   Job 174 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:54:34,498: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-05-10 13:54:44,522: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:54:44,522: INFO/job_scheduler]   Job 173 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:54:44,523: INFO/job_scheduler]   Job 174 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:54:44,523: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-05-10 13:54:54,564: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:54:54,565: INFO/job_scheduler]   Job 173 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:54:54,565: INFO/job_scheduler]   Job 174 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:54:54,565: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-05-10 13:55:04,577: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:55:04,577: INFO/job_scheduler]   Job 173 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:55:04,578: INFO/job_scheduler]   Job 174 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:55:04,578: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-05-10 13:55:06,018: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:55:06,018: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:55:06,019: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:55:06,019: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:55:06,019: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4a7e1b15-4278-4dc1-918c-9d19d5af2c4f
[2017-05-10 13:55:06,019: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4a7e1b15-4278-4dc1-918c-9d19d5af2c4f
[2017-05-10 13:55:06,050: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4a7e1b15-4278-4dc1-918c-9d19d5af2c4f
[2017-05-10 13:55:06,050: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4a7e1b15-4278-4dc1-918c-9d19d5af2c4f
[2017-05-10 13:55:06,053: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-10 13:55:06,072: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8176a864-4dd7-466e-bd67-1c293258ead9
[2017-05-10 13:55:06,072: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8176a864-4dd7-466e-bd67-1c293258ead9
[2017-05-10 13:55:13,204: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:55:13,204: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:55:13,204: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:55:13,204: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:55:13,205: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8176a864-4dd7-466e-bd67-1c293258ead9
[2017-05-10 13:55:13,205: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8176a864-4dd7-466e-bd67-1c293258ead9
[2017-05-10 13:55:13,244: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8176a864-4dd7-466e-bd67-1c293258ead9
[2017-05-10 13:55:13,244: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8176a864-4dd7-466e-bd67-1c293258ead9
[2017-05-10 13:55:13,244: INFO/job_scheduler] Job 173 finished 3 steps successfully
[2017-05-10 13:55:13,325: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 140430198689424
[2017-05-10 13:55:13,332: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-10 13:55:13,340: INFO/job] Job 173: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 13:55:13,341: INFO/job] Job 173: not_deleted=True
[2017-05-10 13:55:13,342: INFO/job_scheduler] Job 173: complete_job: Updating cache
[2017-05-10 13:55:13,342: INFO/job_scheduler] Job 173: locked_item 140430198689424 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 13:55:13,350: INFO/job_scheduler] Job 173 completing (errored=False, cancelled=False)
[2017-05-10 13:55:13,378: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 0 tasked)
[2017-05-10 13:55:13,402: INFO/job_scheduler] Job 174: RunJobThread.run
[2017-05-10 13:55:13,403: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 13:55:13,432: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b3d6757d-e3de-4a4d-bb12-63a614573d72
[2017-05-10 13:55:13,432: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b3d6757d-e3de-4a4d-bb12-63a614573d72
[2017-05-10 13:55:20,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:55:20,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:55:20,567: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:55:20,567: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:55:20,567: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b3d6757d-e3de-4a4d-bb12-63a614573d72
[2017-05-10 13:55:20,567: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b3d6757d-e3de-4a4d-bb12-63a614573d72
[2017-05-10 13:55:20,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b3d6757d-e3de-4a4d-bb12-63a614573d72
[2017-05-10 13:55:20,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b3d6757d-e3de-4a4d-bb12-63a614573d72
[2017-05-10 13:55:20,599: INFO/job_scheduler] Job 174 finished 1 steps successfully
[2017-05-10 13:55:20,640: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 140430198689424
[2017-05-10 13:55:20,665: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 13:55:20,681: INFO/job] Job 174: StateChangeJob complete, setting state started on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 13:55:20,682: INFO/job] Job 174: not_deleted=True
[2017-05-10 13:55:20,682: INFO/job_scheduler] Job 174: complete_job: Updating cache
[2017-05-10 13:55:20,682: INFO/job_scheduler] Job 174: locked_item 140430198689424 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 13:55:20,694: INFO/job_scheduler] Job 174 completing (errored=False, cancelled=False)
[2017-05-10 13:55:20,783: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-5
[2017-05-10 13:55:20,784: INFO/job_scheduler] _notify: Updating .mcast_port of item 5 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 62535
[2017-05-10 13:55:20,792: INFO/job_scheduler] _notify: Set mcast_port=62535 on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-5) and saved
[2017-05-10 13:55:20,797: INFO/job_scheduler] _notify: Updating .network_interfaces of item 5 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-05-10 13:55:20,820: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-5) and saved
[2017-05-10 13:55:20,829: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 5 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-10 13:55:20,837: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-5) and saved
[2017-05-10 13:55:20,844: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 13:55:20,879: INFO/job_scheduler] Job 175: RunJobThread.run
[2017-05-10 13:55:20,882: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-05-10 13:55:20,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 058d30f8-9a09-4d71-85c8-1aa4b28f3d9d
[2017-05-10 13:55:20,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 058d30f8-9a09-4d71-85c8-1aa4b28f3d9d
[2017-05-10 13:55:24,650: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 5 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from True to False
[2017-05-10 13:55:24,655: INFO/job_scheduler] _notify: Set corosync_reported_up=False on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-5) and saved
[2017-05-10 13:55:24,666: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 13:55:44,711: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 5 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-10 13:55:44,717: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-5) and saved
[2017-05-10 13:55:44,726: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 13:55:48,232: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:55:48,232: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:55:48,232: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:55:48,232: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:55:48,232: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 058d30f8-9a09-4d71-85c8-1aa4b28f3d9d
[2017-05-10 13:55:48,232: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 058d30f8-9a09-4d71-85c8-1aa4b28f3d9d
[2017-05-10 13:55:48,274: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 058d30f8-9a09-4d71-85c8-1aa4b28f3d9d
[2017-05-10 13:55:48,274: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 058d30f8-9a09-4d71-85c8-1aa4b28f3d9d
[2017-05-10 13:55:48,276: INFO/job_scheduler] Job 175 finished 1 steps successfully
[2017-05-10 13:55:48,740: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 140430199263312
[2017-05-10 13:55:48,746: INFO/job] Job 175: StateChangeJob complete, setting state managed on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 13:55:48,747: INFO/job] Job 175: not_deleted=True
[2017-05-10 13:55:48,747: INFO/job_scheduler] Job 175: complete_job: Updating cache
[2017-05-10 13:55:48,747: INFO/job_scheduler] Job 175: locked_item 140430199263312 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 13:55:48,750: INFO/job_scheduler] Job 175 completing (errored=False, cancelled=False)
[2017-05-10 13:55:48,797: INFO/job_scheduler] Replaying 3 buffered notifications for ManagedHost-5
[2017-05-10 13:55:48,807: INFO/job_scheduler] _notify: Updating .immutable_state of item 5 (lotus-32vm5.lotus.hpdd.lab.intel.com) from True to False
[2017-05-10 13:55:48,820: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-5) and saved
[2017-05-10 13:55:48,873: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:56:51,172: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:56:51,172: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/6aa4cc37-8386-4294-8bb0-2297e9ddeb60
[2017-05-10 13:56:51,172: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 6aa4cc37-8386-4294-8bb0-2297e9ddeb60 because session terminated
[2017-05-10 13:56:51,172: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 6aa4cc37-8386-4294-8bb0-2297e9ddeb60 because session terminated
[2017-05-10 13:57:02,453: INFO/job_scheduler] _notify: Updating .boot_time of item 5 (lotus-32vm5.lotus.hpdd.lab.intel.com) from 2017-05-03 16:42:07+00:00 to 2017-05-10 06:56:09+00:00
[2017-05-10 13:57:02,458: INFO/job_scheduler] _notify: Set boot_time=2017-05-10 06:56:09+00:00 on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-5) and saved
[2017-05-10 13:57:02,464: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:57:02,510: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 13:57:02,510: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 13:57:06,543: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:57:06,572: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:57:10,933: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:57:10,934: INFO/command_plan] add_jobs: done checking dependencies
[2017-05-10 13:57:10,960: INFO/command_plan] add_jobs: created Job 176 (Test for host connectivity)
[2017-05-10 13:57:11,021: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 13:57:11,023: INFO/job_scheduler] Job 176: RunJobThread.run
[2017-05-10 13:57:11,607: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 13:57:14,648: INFO/job_scheduler] Job 176 finished 1 steps successfully
[2017-05-10 13:57:14,685: INFO/job_scheduler] Job 176: complete_job: Updating cache
[2017-05-10 13:57:14,685: INFO/job_scheduler] Job 176 completing (errored=False, cancelled=False)
[2017-05-10 13:57:14,755: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:57:15,955: INFO/job_scheduler] Getting FQDN for 'lotus-32vm6.lotus.hpdd.lab.intel.com': lotus-32vm6
lotus-32vm6.lotus.hpdd.lab.intel.com

[2017-05-10 13:57:15,993: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:57:16,000: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-6 transitioning from undeployed to unconfigured
[2017-05-10 13:57:16,052: INFO/command_plan] Created command 45 (Deploying agent to host lotus-32vm6.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-05-10 13:57:16,072: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 13:57:16,138: INFO/job_scheduler] Job 177: RunJobThread.run
[2017-05-10 13:57:16,703: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 13:57:18,289: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:57:18,289: INFO/job_scheduler]   Job 177 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-05-10 13:57:18,289: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 13:57:18,308: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:57:18,308: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/bef11f0b-d982-4908-a376-2809b0a8d2db
[2017-05-10 13:57:18,311: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/0e074fbc-34a5-4f0d-861f-aed7fce2a770
[2017-05-10 13:57:18,311: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/0e074fbc-34a5-4f0d-861f-aed7fce2a770
[2017-05-10 13:57:18,318: INFO/job_scheduler] Job 177 finished 1 steps successfully
[2017-05-10 13:57:18,331: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 140430198773776
[2017-05-10 13:57:18,337: INFO/job] Job 177: StateChangeJob complete, setting state unconfigured on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:57:18,337: INFO/job] Job 177: not_deleted=True
[2017-05-10 13:57:18,337: INFO/job_scheduler] Job 177: complete_job: Updating cache
[2017-05-10 13:57:18,337: INFO/job_scheduler] Job 177: locked_item 140430198773776 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 13:57:18,341: INFO/job_scheduler] Job 177 completing (errored=False, cancelled=False)
[2017-05-10 13:57:18,395: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-6
[2017-05-10 13:57:18,396: INFO/job_scheduler] _notify: Updating .boot_time of item 6 (lotus-32vm6.lotus.hpdd.lab.intel.com) from None to 2017-04-25 07:09:19+00:00
[2017-05-10 13:57:18,408: INFO/job_scheduler] _notify: Set boot_time=2017-04-25 07:09:19+00:00 on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-6) and saved
[2017-05-10 13:57:18,419: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:57:27,934: INFO/job_scheduler] _notify: Updating .properties of item 6 (lotus-32vm6.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-05-10 13:57:27,948: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-6) and saved
[2017-05-10 13:57:27,962: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:57:28,129: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 13:57:29,672: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 13:57:29,684: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-6 transitioning from unconfigured to managed
[2017-05-10 13:57:29,845: INFO/command_plan] Created command 46 (Setup managed host lotus-32vm6.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-05-10 13:57:29,918: INFO/job_scheduler] run_next: 1 runnable jobs of (11 pending, 0 tasked)
[2017-05-10 13:57:30,012: INFO/job_scheduler] Job 178: RunJobThread.run
[2017-05-10 13:57:30,013: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-05-10 13:57:30,033: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a40a64a7-636a-417e-8a10-7105181a0cdf
[2017-05-10 13:57:30,033: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a40a64a7-636a-417e-8a10-7105181a0cdf
[2017-05-10 13:57:31,082: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/0e074fbc-34a5-4f0d-861f-aed7fce2a770
[2017-05-10 13:57:31,082: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/0e074fbc-34a5-4f0d-861f-aed7fce2a770
[2017-05-10 13:57:31,082: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/0e074fbc-34a5-4f0d-861f-aed7fce2a770
[2017-05-10 13:57:31,082: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/0e074fbc-34a5-4f0d-861f-aed7fce2a770
[2017-05-10 13:57:31,082: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a40a64a7-636a-417e-8a10-7105181a0cdf
[2017-05-10 13:57:31,082: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a40a64a7-636a-417e-8a10-7105181a0cdf
[2017-05-10 13:57:31,098: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a40a64a7-636a-417e-8a10-7105181a0cdf
[2017-05-10 13:57:31,098: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a40a64a7-636a-417e-8a10-7105181a0cdf
[2017-05-10 13:57:31,098: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-05-10 13:57:31.098323+00:00:
[2017-05-10 13:57:31,098: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-05-10 13:57:31.098323+00:00:
[2017-05-10 13:57:31,098: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 13:57:31,098: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-05-10 13:57:31,123: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-05-10 13:57:31.123149+00:00:
[2017-05-10 13:57:31,123: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-05-10 13:57:31.123149+00:00:
[2017-05-10 13:57:31,124: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:57:31,124: INFO/job_scheduler]   Job 178 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 13:57:31,124: INFO/job_scheduler]   Job 179 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:57:31,123: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 13:57:31,125: INFO/job_scheduler]   Job 180 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:57:31,125: INFO/job_scheduler]   Job 181 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:57:31,125: INFO/job_scheduler]   Job 182 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:57:31,126: INFO/job_scheduler]   Job 186 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:57:31,126: INFO/job_scheduler]   Job 188 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:57:31,126: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 1 tasked)
[2017-05-10 13:57:31,123: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-05-10 13:57:31,146: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:57:31,146: INFO/job_scheduler]   Job 178 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 13:57:31,146: INFO/job_scheduler]   Job 179 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:57:31,146: INFO/job_scheduler]   Job 180 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:57:31,146: INFO/job_scheduler]   Job 181 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:57:31,147: INFO/job_scheduler]   Job 182 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:57:31,147: INFO/job_scheduler]   Job 186 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:57:31,147: INFO/job_scheduler]   Job 188 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:57:31,147: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 1 tasked)
[2017-05-10 13:57:31,148: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com restart_agent {}
[2017-05-10 13:57:31,164: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0d86452c-2424-4906-bb9a-7fb5e03b34f7
[2017-05-10 13:57:31,164: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0d86452c-2424-4906-bb9a-7fb5e03b34f7
[2017-05-10 13:57:31,253: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/0e074fbc-34a5-4f0d-861f-aed7fce2a770
[2017-05-10 13:57:31,253: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/0e074fbc-34a5-4f0d-861f-aed7fce2a770
[2017-05-10 13:57:31,253: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/0e074fbc-34a5-4f0d-861f-aed7fce2a770
[2017-05-10 13:57:31,253: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/0e074fbc-34a5-4f0d-861f-aed7fce2a770
[2017-05-10 13:57:31,253: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0d86452c-2424-4906-bb9a-7fb5e03b34f7
[2017-05-10 13:57:31,253: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0d86452c-2424-4906-bb9a-7fb5e03b34f7
[2017-05-10 13:57:31,280: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0d86452c-2424-4906-bb9a-7fb5e03b34f7
[2017-05-10 13:57:31,280: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0d86452c-2424-4906-bb9a-7fb5e03b34f7
[2017-05-10 13:57:31,280: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm6.lotus.hpdd.lab.intel.com (old 0e074fbc-34a5-4f0d-861f-aed7fce2a770)
[2017-05-10 13:57:31,280: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm6.lotus.hpdd.lab.intel.com (old 0e074fbc-34a5-4f0d-861f-aed7fce2a770)
[2017-05-10 13:58:01,052: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/0e074fbc-34a5-4f0d-861f-aed7fce2a770
[2017-05-10 13:58:01,052: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/0e074fbc-34a5-4f0d-861f-aed7fce2a770
[2017-05-10 13:58:01,052: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 0e074fbc-34a5-4f0d-861f-aed7fce2a770 because session terminated
[2017-05-10 13:58:01,052: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 0e074fbc-34a5-4f0d-861f-aed7fce2a770 because session terminated
[2017-05-10 13:58:01,054: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:58:01,054: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:58:01,314: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm6.lotus.hpdd.lab.intel.com new e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:58:01,314: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm6.lotus.hpdd.lab.intel.com new e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:58:01,315: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 13:58:01,330: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7e093a93-f19f-4af2-95c4-090f9365df1b
[2017-05-10 13:58:01,330: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7e093a93-f19f-4af2-95c4-090f9365df1b
[2017-05-10 13:58:02,447: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:58:02,447: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:58:02,447: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:58:02,447: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:58:02,447: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7e093a93-f19f-4af2-95c4-090f9365df1b
[2017-05-10 13:58:02,447: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7e093a93-f19f-4af2-95c4-090f9365df1b
[2017-05-10 13:58:02,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7e093a93-f19f-4af2-95c4-090f9365df1b
[2017-05-10 13:58:02,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7e093a93-f19f-4af2-95c4-090f9365df1b
[2017-05-10 13:58:02,495: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-05-10 13:58:02,508: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fa6f7fee-b3ae-4126-86f5-906bc5085bf5
[2017-05-10 13:58:02,508: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fa6f7fee-b3ae-4126-86f5-906bc5085bf5
[2017-05-10 13:58:11,112: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:58:11,113: INFO/job_scheduler]   Job 178 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-05-10 13:58:11,113: INFO/job_scheduler]   Job 179 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:58:11,113: INFO/job_scheduler]   Job 180 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:58:11,113: INFO/job_scheduler]   Job 181 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:58:11,113: INFO/job_scheduler]   Job 182 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:58:11,113: INFO/job_scheduler]   Job 186 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:58:11,114: INFO/job_scheduler]   Job 188 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:58:11,114: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 1 tasked)
[2017-05-10 13:58:47,190: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:58:47,190: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:58:47,191: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:58:47,191: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:58:47,191: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fa6f7fee-b3ae-4126-86f5-906bc5085bf5
[2017-05-10 13:58:47,191: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fa6f7fee-b3ae-4126-86f5-906bc5085bf5
[2017-05-10 13:58:47,195: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fa6f7fee-b3ae-4126-86f5-906bc5085bf5
[2017-05-10 13:58:47,195: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fa6f7fee-b3ae-4126-86f5-906bc5085bf5
[2017-05-10 13:58:47,361: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com install_packages {'repos': [u'e2fsprogs', u'iml-agent', u'lustre', u'zfs'], 'packages': [u'chroma-agent-management', u'kernel-devel-3.10.0-514.10.2.el7_lustre', u'lustre-osd-zfs', u'lustre-osd-ldiskfs', u'lustre-modules', u'lustre', u'zfs']}
[2017-05-10 13:58:47,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e3d61d89-7b9d-4c1d-8e16-356b4598b2e6
[2017-05-10 13:58:47,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e3d61d89-7b9d-4c1d-8e16-356b4598b2e6
[2017-05-10 13:59:05,178: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:05,178: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:05,179: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:05,179: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:05,179: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e3d61d89-7b9d-4c1d-8e16-356b4598b2e6
[2017-05-10 13:59:05,179: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e3d61d89-7b9d-4c1d-8e16-356b4598b2e6
[2017-05-10 13:59:05,207: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e3d61d89-7b9d-4c1d-8e16-356b4598b2e6
[2017-05-10 13:59:05,207: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e3d61d89-7b9d-4c1d-8e16-356b4598b2e6
[2017-05-10 13:59:05,673: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com kernel_status {}
[2017-05-10 13:59:05,688: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7600e66a-5c34-4272-a3a2-5a69a6fef019
[2017-05-10 13:59:05,688: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7600e66a-5c34-4272-a3a2-5a69a6fef019
[2017-05-10 13:59:07,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:07,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:07,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:07,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:07,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7600e66a-5c34-4272-a3a2-5a69a6fef019
[2017-05-10 13:59:07,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7600e66a-5c34-4272-a3a2-5a69a6fef019
[2017-05-10 13:59:07,806: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7600e66a-5c34-4272-a3a2-5a69a6fef019
[2017-05-10 13:59:07,806: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7600e66a-5c34-4272-a3a2-5a69a6fef019
[2017-05-10 13:59:07,806: INFO/job_scheduler] Job 178 finished 4 steps successfully
[2017-05-10 13:59:07,886: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 140430199250896
[2017-05-10 13:59:07,891: INFO/job] Job 178: StateChangeJob complete, setting state packages_installed on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:59:07,891: INFO/job] Job 178: not_deleted=True
[2017-05-10 13:59:07,891: INFO/job_scheduler] Job 178: complete_job: Updating cache
[2017-05-10 13:59:07,891: INFO/job_scheduler] Job 178: locked_item 140430198773712 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 13:59:07,895: INFO/job_scheduler] Job 178 completing (errored=False, cancelled=False)
[2017-05-10 13:59:07,912: INFO/job_scheduler] run_next: 4 runnable jobs of (10 pending, 0 tasked)
[2017-05-10 13:59:07,935: INFO/job_scheduler] Job 179: open DB connection during completion
[2017-05-10 13:59:07,953: INFO/job_scheduler] Job 180: RunJobThread.run
[2017-05-10 13:59:07,954: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-05-10 13:59:07,977: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 35c6ee68-87da-4ddf-b204-dbec72c1e947
[2017-05-10 13:59:07,977: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 35c6ee68-87da-4ddf-b204-dbec72c1e947
[2017-05-10 13:59:07,982: INFO/job_scheduler] Job 181: RunJobThread.run
[2017-05-10 13:59:07,983: INFO/job_scheduler] Job 182: RunJobThread.run
[2017-05-10 13:59:07,984: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-05-10 13:59:07,985: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 140430199291472
[2017-05-10 13:59:07,986: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-05-10 13:59:08,004: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-05-10 13:59:08,007: INFO/job] Job 179: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:59:08,008: INFO/job] Job 179: not_deleted=True
[2017-05-10 13:59:08,008: INFO/job_scheduler] Job 179: complete_job: Updating cache
[2017-05-10 13:59:08,008: INFO/job_scheduler] Job 179: locked_item 140430199291472 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:59:08,011: INFO/job_scheduler] Job 179 completing (errored=False, cancelled=False)
[2017-05-10 13:59:08,035: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6e25a0fc-2269-4898-bbd1-80ddaf89cf78
[2017-05-10 13:59:08,035: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6e25a0fc-2269-4898-bbd1-80ddaf89cf78
[2017-05-10 13:59:08,048: INFO/job_scheduler] run_next: 1 runnable jobs of (6 pending, 3 tasked)
[2017-05-10 13:59:08,056: INFO/job_scheduler] Job 183: RunJobThread.run
[2017-05-10 13:59:08,057: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com load_lnet {}
[2017-05-10 13:59:08,066: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fc88be3a-89e0-4587-b077-85031e0c8f82
[2017-05-10 13:59:08,066: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fc88be3a-89e0-4587-b077-85031e0c8f82
[2017-05-10 13:59:08,127: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b472527c-658f-4d0a-9fad-12dcc62f66ba
[2017-05-10 13:59:08,127: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b472527c-658f-4d0a-9fad-12dcc62f66ba
[2017-05-10 13:59:09,157: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:09,157: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:09,158: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:09,158: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:09,158: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b472527c-658f-4d0a-9fad-12dcc62f66ba
[2017-05-10 13:59:09,158: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b472527c-658f-4d0a-9fad-12dcc62f66ba
[2017-05-10 13:59:09,159: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b472527c-658f-4d0a-9fad-12dcc62f66ba
[2017-05-10 13:59:09,159: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b472527c-658f-4d0a-9fad-12dcc62f66ba
[2017-05-10 13:59:09,160: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 13:59:09,176: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 28429743-94a1-45b8-99df-93bdf8330337
[2017-05-10 13:59:09,176: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 28429743-94a1-45b8-99df-93bdf8330337
[2017-05-10 13:59:09,284: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:09,284: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:09,285: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:09,285: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:09,285: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 28429743-94a1-45b8-99df-93bdf8330337
[2017-05-10 13:59:09,285: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 28429743-94a1-45b8-99df-93bdf8330337
[2017-05-10 13:59:09,294: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 28429743-94a1-45b8-99df-93bdf8330337
[2017-05-10 13:59:09,294: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 28429743-94a1-45b8-99df-93bdf8330337
[2017-05-10 13:59:09,536: INFO/job_scheduler] Job 183 finished 2 steps successfully
[2017-05-10 13:59:09,574: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 140430199291472
[2017-05-10 13:59:09,584: INFO/job] Job 183: StateChangeJob complete, setting state lnet_down on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:59:09,585: INFO/job] Job 183: not_deleted=True
[2017-05-10 13:59:09,585: INFO/job_scheduler] Job 183: complete_job: Updating cache
[2017-05-10 13:59:09,585: INFO/job_scheduler] Job 183: locked_item 140430199291472 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:59:09,588: INFO/job_scheduler] Job 183 completing (errored=False, cancelled=False)
[2017-05-10 13:59:09,604: INFO/job_scheduler] run_next: 1 runnable jobs of (5 pending, 3 tasked)
[2017-05-10 13:59:09,608: INFO/job_scheduler] Job 185: RunJobThread.run
[2017-05-10 13:59:09,609: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_lnet {}
[2017-05-10 13:59:09,623: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 16071388-d226-4df8-bfb6-60aa4063228d
[2017-05-10 13:59:09,623: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 16071388-d226-4df8-bfb6-60aa4063228d
[2017-05-10 13:59:11,760: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:11,760: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:11,760: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:11,760: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:11,761: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 16071388-d226-4df8-bfb6-60aa4063228d
[2017-05-10 13:59:11,761: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 16071388-d226-4df8-bfb6-60aa4063228d
[2017-05-10 13:59:11,788: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 16071388-d226-4df8-bfb6-60aa4063228d
[2017-05-10 13:59:11,788: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 16071388-d226-4df8-bfb6-60aa4063228d
[2017-05-10 13:59:11,789: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 13:59:11,802: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d09aed4b-5e5d-4b8f-b788-a026d4a00f36
[2017-05-10 13:59:11,802: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d09aed4b-5e5d-4b8f-b788-a026d4a00f36
[2017-05-10 13:59:12,081: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:12,081: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:12,081: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:12,081: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:12,082: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d09aed4b-5e5d-4b8f-b788-a026d4a00f36
[2017-05-10 13:59:12,082: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d09aed4b-5e5d-4b8f-b788-a026d4a00f36
[2017-05-10 13:59:12,118: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d09aed4b-5e5d-4b8f-b788-a026d4a00f36
[2017-05-10 13:59:12,118: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d09aed4b-5e5d-4b8f-b788-a026d4a00f36
[2017-05-10 13:59:12,331: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:12,331: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:12,332: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:12,332: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:12,332: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fc88be3a-89e0-4587-b077-85031e0c8f82
[2017-05-10 13:59:12,332: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fc88be3a-89e0-4587-b077-85031e0c8f82
[2017-05-10 13:59:12,339: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fc88be3a-89e0-4587-b077-85031e0c8f82
[2017-05-10 13:59:12,339: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fc88be3a-89e0-4587-b077-85031e0c8f82
[2017-05-10 13:59:12,340: INFO/job_scheduler] Job 181 finished 1 steps successfully
[2017-05-10 13:59:12,386: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 140430199288784
[2017-05-10 13:59:12,389: INFO/job] Job 181: StateChangeJob complete, setting state configured on lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-10 13:59:12,389: INFO/job] Job 181: not_deleted=True
[2017-05-10 13:59:12,389: INFO/job_scheduler] Job 181: complete_job: Updating cache
[2017-05-10 13:59:12,390: INFO/job_scheduler] Job 181: locked_item 140430199288784 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-10 13:59:12,392: INFO/job_scheduler] Job 181 completing (errored=False, cancelled=False)
[2017-05-10 13:59:12,422: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 3 tasked)
[2017-05-10 13:59:12,535: INFO/job_scheduler] Job 185 finished 2 steps successfully
[2017-05-10 13:59:12,552: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 140430199291472
[2017-05-10 13:59:12,557: INFO/job] Job 185: StateChangeJob complete, setting state lnet_up on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 13:59:12,557: INFO/job] Job 185: not_deleted=True
[2017-05-10 13:59:12,557: INFO/job_scheduler] Job 185: complete_job: Updating cache
[2017-05-10 13:59:12,558: INFO/job_scheduler] Job 185: locked_item 140430199291472 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 13:59:12,562: INFO/job_scheduler] Job 185 completing (errored=False, cancelled=False)
[2017-05-10 13:59:12,585: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-05-10 13:59:13,288: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:13,288: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:13,288: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:13,288: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:13,288: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6e25a0fc-2269-4898-bbd1-80ddaf89cf78
[2017-05-10 13:59:13,288: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6e25a0fc-2269-4898-bbd1-80ddaf89cf78
[2017-05-10 13:59:13,302: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6e25a0fc-2269-4898-bbd1-80ddaf89cf78
[2017-05-10 13:59:13,302: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6e25a0fc-2269-4898-bbd1-80ddaf89cf78
[2017-05-10 13:59:13,303: INFO/job_scheduler] Job 182 finished 1 steps successfully
[2017-05-10 13:59:13,394: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 140430199288080
[2017-05-10 13:59:13,398: INFO/job] Job 182: StateChangeJob complete, setting state configured on lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-10 13:59:13,398: INFO/job] Job 182: not_deleted=True
[2017-05-10 13:59:13,398: INFO/job_scheduler] Job 182: complete_job: Updating cache
[2017-05-10 13:59:13,398: INFO/job_scheduler] Job 182: locked_item 140430199288080 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-10 13:59:13,401: INFO/job_scheduler] Job 182 completing (errored=False, cancelled=False)
[2017-05-10 13:59:13,421: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 1 tasked)
[2017-05-10 13:59:25,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:25,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:25,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:25,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:25,720: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 35c6ee68-87da-4ddf-b204-dbec72c1e947
[2017-05-10 13:59:25,720: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 35c6ee68-87da-4ddf-b204-dbec72c1e947
[2017-05-10 13:59:25,749: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 35c6ee68-87da-4ddf-b204-dbec72c1e947
[2017-05-10 13:59:25,749: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 35c6ee68-87da-4ddf-b204-dbec72c1e947
[2017-05-10 13:59:25,750: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.225', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-05-10 13:59:25,769: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5b8aba6c-6ed2-4ee9-81d3-d73554444ae9
[2017-05-10 13:59:25,769: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5b8aba6c-6ed2-4ee9-81d3-d73554444ae9
[2017-05-10 13:59:25,890: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:25,890: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:25,890: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:25,890: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:25,890: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5b8aba6c-6ed2-4ee9-81d3-d73554444ae9
[2017-05-10 13:59:25,890: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5b8aba6c-6ed2-4ee9-81d3-d73554444ae9
[2017-05-10 13:59:25,936: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5b8aba6c-6ed2-4ee9-81d3-d73554444ae9
[2017-05-10 13:59:25,936: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5b8aba6c-6ed2-4ee9-81d3-d73554444ae9
[2017-05-10 13:59:25,956: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 62535, 'pcs_password': '9dYBbhWIQMatlYThwUEZ'}
[2017-05-10 13:59:25,974: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b34889f3-f0aa-4490-8d1c-6311996a98df
[2017-05-10 13:59:25,974: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b34889f3-f0aa-4490-8d1c-6311996a98df
[2017-05-10 13:59:38,145: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:38,145: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:38,145: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:38,145: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:38,146: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b34889f3-f0aa-4490-8d1c-6311996a98df
[2017-05-10 13:59:38,146: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b34889f3-f0aa-4490-8d1c-6311996a98df
[2017-05-10 13:59:38,192: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b34889f3-f0aa-4490-8d1c-6311996a98df
[2017-05-10 13:59:38,192: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b34889f3-f0aa-4490-8d1c-6311996a98df
[2017-05-10 13:59:38,193: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': False, 'mcast_port': 62535, 'new_node_fqdn': u'lotus-32vm6.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': '9dYBbhWIQMatlYThwUEZ'}
[2017-05-10 13:59:38,208: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3f76ab09-16d0-4923-8fc7-384178bf942f
[2017-05-10 13:59:38,208: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3f76ab09-16d0-4923-8fc7-384178bf942f
[2017-05-10 13:59:44,513: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 13:59:44,513: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 13:59:44,513: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 13:59:44,513: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 13:59:44,513: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3f76ab09-16d0-4923-8fc7-384178bf942f
[2017-05-10 13:59:44,513: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3f76ab09-16d0-4923-8fc7-384178bf942f
[2017-05-10 13:59:44,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3f76ab09-16d0-4923-8fc7-384178bf942f
[2017-05-10 13:59:44,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3f76ab09-16d0-4923-8fc7-384178bf942f
[2017-05-10 13:59:44,526: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 13:59:44.526767+00:00:
[2017-05-10 13:59:44,526: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-05-10 13:59:44.526767+00:00:
[2017-05-10 13:59:44,527: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->62535
[2017-05-10 13:59:44,527: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->62535
[2017-05-10 13:59:44,529: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 13:59:44,529: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-05-10 13:59:44,543: INFO/job_scheduler] Job 180 finished 1 steps successfully
[2017-05-10 13:59:44,546: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:59:44,546: INFO/job_scheduler]   Job 180 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-05-10 13:59:44,546: INFO/job_scheduler]   Job 184 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-05-10 13:59:44,546: INFO/job_scheduler]   Job 186 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:59:44,546: INFO/job_scheduler]   Job 187 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:59:44,547: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 1 tasked)
[2017-05-10 13:59:44,582: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 140430198283856
[2017-05-10 13:59:44,590: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-10 13:59:44,599: INFO/job] Job 180: StateChangeJob complete, setting state stopped on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:59:44,599: INFO/job] Job 180: not_deleted=True
[2017-05-10 13:59:44,600: INFO/job_scheduler] Job 180: complete_job: Updating cache
[2017-05-10 13:59:44,601: INFO/job_scheduler] Job 180: locked_item 140430198283856 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 13:59:44,622: INFO/job_scheduler] Job 180 completing (errored=False, cancelled=False)
[2017-05-10 13:59:44,644: INFO/job_scheduler] run_next: 1 runnable jobs of (4 pending, 0 tasked)
[2017-05-10 13:59:44,651: INFO/job_scheduler] Job 184: RunJobThread.run
[2017-05-10 13:59:44,652: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-05-10 13:59:44,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3cffe932-ae30-4e57-9998-3a351d1086fe
[2017-05-10 13:59:44,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3cffe932-ae30-4e57-9998-3a351d1086fe
[2017-05-10 13:59:52,818: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:52,818: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:52,819: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:52,819: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 13:59:52,819: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3cffe932-ae30-4e57-9998-3a351d1086fe
[2017-05-10 13:59:52,819: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3cffe932-ae30-4e57-9998-3a351d1086fe
[2017-05-10 13:59:52,838: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3cffe932-ae30-4e57-9998-3a351d1086fe
[2017-05-10 13:59:52,838: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3cffe932-ae30-4e57-9998-3a351d1086fe
[2017-05-10 13:59:52,839: INFO/job_scheduler] Job 184 finished 1 steps successfully
[2017-05-10 13:59:52,888: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 140430198283856
[2017-05-10 13:59:52,921: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm6.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 13:59:52,933: INFO/job] Job 184: StateChangeJob complete, setting state started on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:59:52,933: INFO/job] Job 184: not_deleted=True
[2017-05-10 13:59:52,933: INFO/job_scheduler] Job 184: complete_job: Updating cache
[2017-05-10 13:59:52,933: INFO/job_scheduler] Job 184: locked_item 140430198283856 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 13:59:52,937: INFO/job_scheduler] Job 184 completing (errored=False, cancelled=False)
[2017-05-10 13:59:52,950: INFO/job_scheduler] run_next: 1 runnable jobs of (3 pending, 0 tasked)
[2017-05-10 13:59:52,977: INFO/job_scheduler] Job 186: RunJobThread.run
[2017-05-10 13:59:52,978: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 13:59:52,999: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5afd4d56-9c89-438b-8385-e2098ea96137
[2017-05-10 13:59:52,999: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5afd4d56-9c89-438b-8385-e2098ea96137
[2017-05-10 13:59:55,744: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-05-10 13:59:55,759: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-05-10 13:59:57,035: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 5 (lotus-32vm5.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7fb874278fd0> to [6]
[2017-05-10 13:59:57,048: INFO/job_scheduler] _notify: Set ha_cluster_peers=[6] on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-5) and saved
[2017-05-10 13:59:57,129: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-05-10 13:59:57,136: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 13:59:57,136: INFO/job_scheduler]   Job 186 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:59:57,136: INFO/job_scheduler]   Job 187 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 13:59:57,137: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-05-10 13:59:57,139: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-05-10 13:59:57,139: INFO/job_scheduler]   Job 186 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 13:59:57,140: INFO/job_scheduler]   Job 188 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-05-10 13:59:57,140: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-05-10 14:00:00,067: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 14:00:00,067: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 14:00:00,068: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 14:00:00,068: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 14:00:00,068: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5afd4d56-9c89-438b-8385-e2098ea96137
[2017-05-10 14:00:00,068: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5afd4d56-9c89-438b-8385-e2098ea96137
[2017-05-10 14:00:00,114: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5afd4d56-9c89-438b-8385-e2098ea96137
[2017-05-10 14:00:00,114: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5afd4d56-9c89-438b-8385-e2098ea96137
[2017-05-10 14:00:00,116: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-05-10 14:00:00,138: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c586ab96-ab2a-49ce-8e29-1c2fe96bc834
[2017-05-10 14:00:00,138: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c586ab96-ab2a-49ce-8e29-1c2fe96bc834
[2017-05-10 14:00:05,793: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-05-10 14:00:05,814: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-05-10 14:00:07,045: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 14:00:07,045: INFO/job_scheduler]   Job 186 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 14:00:07,045: INFO/job_scheduler]   Job 187 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 14:00:07,046: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-05-10 14:00:15,805: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-05-10 14:00:15,827: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-05-10 14:00:15,860: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 14:00:15,860: INFO/job_scheduler]   Job 186 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 14:00:15,860: INFO/job_scheduler]   Job 187 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 14:00:15,861: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-05-10 14:00:17,078: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 14:00:17,078: INFO/job_scheduler]   Job 186 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 14:00:17,079: INFO/job_scheduler]   Job 187 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 14:00:17,079: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-05-10 14:00:20,310: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 14:00:20,310: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 14:00:20,310: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 14:00:20,310: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 14:00:20,311: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c586ab96-ab2a-49ce-8e29-1c2fe96bc834
[2017-05-10 14:00:20,311: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c586ab96-ab2a-49ce-8e29-1c2fe96bc834
[2017-05-10 14:00:20,356: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c586ab96-ab2a-49ce-8e29-1c2fe96bc834
[2017-05-10 14:00:20,356: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c586ab96-ab2a-49ce-8e29-1c2fe96bc834
[2017-05-10 14:00:20,358: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-10 14:00:20,371: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9e979db5-69e1-4509-a2be-9445d0f6255a
[2017-05-10 14:00:20,371: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9e979db5-69e1-4509-a2be-9445d0f6255a
[2017-05-10 14:00:25,844: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-05-10 14:00:25,862: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-05-10 14:00:27,084: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 14:00:27,084: INFO/job_scheduler]   Job 186 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 14:00:27,084: INFO/job_scheduler]   Job 187 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 14:00:27,084: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-05-10 14:00:27,466: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 14:00:27,466: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 14:00:27,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 14:00:27,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 14:00:27,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9e979db5-69e1-4509-a2be-9445d0f6255a
[2017-05-10 14:00:27,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9e979db5-69e1-4509-a2be-9445d0f6255a
[2017-05-10 14:00:27,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9e979db5-69e1-4509-a2be-9445d0f6255a
[2017-05-10 14:00:27,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9e979db5-69e1-4509-a2be-9445d0f6255a
[2017-05-10 14:00:27,495: INFO/job_scheduler] Job 186 finished 3 steps successfully
[2017-05-10 14:00:27,550: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 140430199290064
[2017-05-10 14:00:27,558: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-10 14:00:27,563: INFO/job] Job 186: StateChangeJob complete, setting state stopped on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 14:00:27,564: INFO/job] Job 186: not_deleted=True
[2017-05-10 14:00:27,565: INFO/job_scheduler] Job 186: complete_job: Updating cache
[2017-05-10 14:00:27,565: INFO/job_scheduler] Job 186: locked_item 140430199290064 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 14:00:27,571: INFO/job_scheduler] Job 186 completing (errored=False, cancelled=False)
[2017-05-10 14:00:27,607: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 0 tasked)
[2017-05-10 14:00:27,650: INFO/job_scheduler] Job 187: RunJobThread.run
[2017-05-10 14:00:27,651: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-05-10 14:00:27,683: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5e8e5a4a-e770-403d-8ead-f4ce3bbfb138
[2017-05-10 14:00:27,683: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5e8e5a4a-e770-403d-8ead-f4ce3bbfb138
[2017-05-10 14:00:34,748: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 14:00:34,748: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 14:00:34,748: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 14:00:34,748: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 14:00:34,749: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5e8e5a4a-e770-403d-8ead-f4ce3bbfb138
[2017-05-10 14:00:34,749: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5e8e5a4a-e770-403d-8ead-f4ce3bbfb138
[2017-05-10 14:00:34,749: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5e8e5a4a-e770-403d-8ead-f4ce3bbfb138
[2017-05-10 14:00:34,749: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5e8e5a4a-e770-403d-8ead-f4ce3bbfb138
[2017-05-10 14:00:34,750: INFO/job_scheduler] Job 187 finished 1 steps successfully
[2017-05-10 14:00:34,829: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 140430199290064
[2017-05-10 14:00:34,851: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm6.lotus.hpdd.lab.intel.com at severity 30
[2017-05-10 14:00:34,858: INFO/job] Job 187: StateChangeJob complete, setting state started on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 14:00:34,858: INFO/job] Job 187: not_deleted=True
[2017-05-10 14:00:34,858: INFO/job_scheduler] Job 187: complete_job: Updating cache
[2017-05-10 14:00:34,858: INFO/job_scheduler] Job 187: locked_item 140430199290064 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 14:00:34,861: INFO/job_scheduler] Job 187 completing (errored=False, cancelled=False)
[2017-05-10 14:00:34,883: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-6
[2017-05-10 14:00:34,884: INFO/job_scheduler] _notify: Updating .mcast_port of item 6 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 62535
[2017-05-10 14:00:34,893: INFO/job_scheduler] _notify: Set mcast_port=62535 on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-6) and saved
[2017-05-10 14:00:34,895: INFO/job_scheduler] _notify: Updating .network_interfaces of item 6 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-05-10 14:00:34,908: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-6) and saved
[2017-05-10 14:00:34,942: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 6 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-05-10 14:00:34,949: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-6) and saved
[2017-05-10 14:00:34,956: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 14:00:34,990: INFO/job_scheduler] Job 188: RunJobThread.run
[2017-05-10 14:00:34,993: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-05-10 14:00:35,018: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6035788a-6079-4725-965a-e82ee876ac73
[2017-05-10 14:00:35,018: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6035788a-6079-4725-965a-e82ee876ac73
[2017-05-10 14:00:35,860: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 14:00:35,876: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 14:01:00,489: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 14:01:00,489: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 14:01:00,489: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 14:01:00,489: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 14:01:00,489: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6035788a-6079-4725-965a-e82ee876ac73
[2017-05-10 14:01:00,489: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6035788a-6079-4725-965a-e82ee876ac73
[2017-05-10 14:01:00,490: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6035788a-6079-4725-965a-e82ee876ac73
[2017-05-10 14:01:00,490: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6035788a-6079-4725-965a-e82ee876ac73
[2017-05-10 14:01:00,492: INFO/job_scheduler] Job 188 finished 1 steps successfully
[2017-05-10 14:01:01,041: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 140430198773712
[2017-05-10 14:01:01,049: INFO/job] Job 188: StateChangeJob complete, setting state managed on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 14:01:01,049: INFO/job] Job 188: not_deleted=True
[2017-05-10 14:01:01,049: INFO/job_scheduler] Job 188: complete_job: Updating cache
[2017-05-10 14:01:01,049: INFO/job_scheduler] Job 188: locked_item 140430198773712 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-05-10 14:01:01,065: INFO/job_scheduler] Job 188 completing (errored=False, cancelled=False)
[2017-05-10 14:01:01,119: INFO/job_scheduler] Replaying 4 buffered notifications for ManagedHost-6
[2017-05-10 14:01:01,157: INFO/job_scheduler] _notify: Updating .immutable_state of item 6 (lotus-32vm6.lotus.hpdd.lab.intel.com) from True to False
[2017-05-10 14:01:01,171: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-6) and saved
[2017-05-10 14:01:01,227: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 6 (lotus-32vm6.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7fb87424c250> to [5]
[2017-05-10 14:01:01,243: INFO/job_scheduler] _notify: Set ha_cluster_peers=[5] on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-6) and saved
[2017-05-10 14:01:01,335: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 14:07:08,367: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 14:07:08,384: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 14:08:07,271: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 14:08:07,292: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 14:17:20,148: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 14:17:20,164: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 14:18:19,276: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 14:18:19,303: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 14:27:31,976: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 14:27:31,992: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 14:28:31,226: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 14:28:31,237: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 14:37:43,758: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 14:37:43,779: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 14:38:43,200: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 14:38:43,219: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 14:47:55,610: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 14:47:55,641: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 14:48:55,008: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 14:48:55,022: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 14:58:07,391: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 14:58:07,407: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 14:59:06,827: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 14:59:06,849: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 15:08:19,316: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 15:08:19,335: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 15:09:18,665: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 15:09:18,684: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 15:18:31,103: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 15:18:31,134: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 15:19:30,493: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 15:19:30,507: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 15:28:42,785: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 15:28:42,800: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 15:29:42,551: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 15:29:42,580: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 15:38:54,509: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 15:38:54,527: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 15:39:54,452: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 15:39:54,472: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 15:39:59,985: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 15:39:59,989: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-5 transitioning from managed to removed
[2017-05-10 15:40:00,104: INFO/command_plan] Created command 47 (Remove host lotus-32vm5.lotus.hpdd.lab.intel.com from configuration) with 9 jobs
[2017-05-10 15:40:00,135: INFO/job_scheduler] run_next: 4 runnable jobs of (9 pending, 0 tasked)
[2017-05-10 15:40:00,202: INFO/job_scheduler] Job 192: RunJobThread.run
[2017-05-10 15:40:00,203: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_lnet {}
[2017-05-10 15:40:00,226: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d3803366-e87b-4466-8d69-15e7f424451d
[2017-05-10 15:40:00,226: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d3803366-e87b-4466-8d69-15e7f424451d
[2017-05-10 15:40:00,229: INFO/job_scheduler] Job 189: RunJobThread.run
[2017-05-10 15:40:00,230: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unconfigure_ntp {}
[2017-05-10 15:40:00,257: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 98019145-681f-4ed4-83b3-d14ce3d76a24
[2017-05-10 15:40:00,257: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 98019145-681f-4ed4-83b3-d14ce3d76a24
[2017-05-10 15:40:00,266: INFO/job_scheduler] Job 190: RunJobThread.run
[2017-05-10 15:40:00,267: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unconfigure_rsyslog {}
[2017-05-10 15:40:00,293: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dce538ea-0bc3-486a-8d83-87003e0cf1a5
[2017-05-10 15:40:00,293: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dce538ea-0bc3-486a-8d83-87003e0cf1a5
[2017-05-10 15:40:00,294: INFO/job_scheduler] Job 191: RunJobThread.run
[2017-05-10 15:40:00,294: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-10 15:40:00,331: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4cd2731f-bd6c-4e55-b1f7-8760a5685700
[2017-05-10 15:40:00,331: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4cd2731f-bd6c-4e55-b1f7-8760a5685700
[2017-05-10 15:40:04,678: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-10 15:40:04,696: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-05-10 15:40:06,342: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:06,342: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:06,343: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:06,343: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:06,343: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 98019145-681f-4ed4-83b3-d14ce3d76a24
[2017-05-10 15:40:06,343: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 98019145-681f-4ed4-83b3-d14ce3d76a24
[2017-05-10 15:40:06,374: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 98019145-681f-4ed4-83b3-d14ce3d76a24
[2017-05-10 15:40:06,374: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 98019145-681f-4ed4-83b3-d14ce3d76a24
[2017-05-10 15:40:06,375: INFO/job_scheduler] Job 189 finished 1 steps successfully
[2017-05-10 15:40:06,446: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration configured->unconfigured (intentional=True) 140430198768720
[2017-05-10 15:40:06,450: INFO/job] Job 189: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-10 15:40:06,450: INFO/job] Job 189: not_deleted=True
[2017-05-10 15:40:06,450: INFO/job_scheduler] Job 189: complete_job: Updating cache
[2017-05-10 15:40:06,450: INFO/job_scheduler] Job 189: locked_item 140430198768720 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-10 15:40:06,453: INFO/job_scheduler] Job 189 completing (errored=False, cancelled=False)
[2017-05-10 15:40:06,462: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 3 tasked)
[2017-05-10 15:40:07,379: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:07,379: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:07,379: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:07,379: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:07,379: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dce538ea-0bc3-486a-8d83-87003e0cf1a5
[2017-05-10 15:40:07,379: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dce538ea-0bc3-486a-8d83-87003e0cf1a5
[2017-05-10 15:40:07,410: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dce538ea-0bc3-486a-8d83-87003e0cf1a5
[2017-05-10 15:40:07,410: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dce538ea-0bc3-486a-8d83-87003e0cf1a5
[2017-05-10 15:40:07,410: INFO/job_scheduler] Job 190 finished 1 steps successfully
[2017-05-10 15:40:07,521: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration configured->unconfigured (intentional=True) 140430198769424
[2017-05-10 15:40:07,525: INFO/job] Job 190: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-10 15:40:07,525: INFO/job] Job 190: not_deleted=True
[2017-05-10 15:40:07,525: INFO/job_scheduler] Job 190: complete_job: Updating cache
[2017-05-10 15:40:07,525: INFO/job_scheduler] Job 190: locked_item 140430198769424 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-10 15:40:07,528: INFO/job_scheduler] Job 190 completing (errored=False, cancelled=False)
[2017-05-10 15:40:07,537: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 2 tasked)
[2017-05-10 15:40:08,382: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:08,382: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:08,382: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:08,382: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:08,382: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4cd2731f-bd6c-4e55-b1f7-8760a5685700
[2017-05-10 15:40:08,382: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4cd2731f-bd6c-4e55-b1f7-8760a5685700
[2017-05-10 15:40:08,406: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4cd2731f-bd6c-4e55-b1f7-8760a5685700
[2017-05-10 15:40:08,406: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4cd2731f-bd6c-4e55-b1f7-8760a5685700
[2017-05-10 15:40:08,406: INFO/job_scheduler] Job 191 finished 1 steps successfully
[2017-05-10 15:40:08,437: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration started->stopped (intentional=True) 140430199276240
[2017-05-10 15:40:08,445: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-10 15:40:08,462: INFO/job] Job 191: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 15:40:08,462: INFO/job] Job 191: not_deleted=True
[2017-05-10 15:40:08,463: INFO/job_scheduler] Job 191: complete_job: Updating cache
[2017-05-10 15:40:08,464: INFO/job_scheduler] Job 191: locked_item 140430199276240 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 15:40:08,472: INFO/job_scheduler] Job 191 completing (errored=False, cancelled=False)
[2017-05-10 15:40:08,487: INFO/job_scheduler] run_next: 1 runnable jobs of (5 pending, 1 tasked)
[2017-05-10 15:40:08,493: INFO/job_scheduler] Job 194: RunJobThread.run
[2017-05-10 15:40:08,496: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_corosync2 {}
[2017-05-10 15:40:08,522: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 79293c9a-cab0-4693-978e-ae0c262f10af
[2017-05-10 15:40:08,522: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 79293c9a-cab0-4693-978e-ae0c262f10af
[2017-05-10 15:40:12,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:12,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:12,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:12,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:12,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d3803366-e87b-4466-8d69-15e7f424451d
[2017-05-10 15:40:12,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d3803366-e87b-4466-8d69-15e7f424451d
[2017-05-10 15:40:12,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d3803366-e87b-4466-8d69-15e7f424451d
[2017-05-10 15:40:12,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d3803366-e87b-4466-8d69-15e7f424451d
[2017-05-10 15:40:12,495: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 15:40:12,510: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c01095bf-3e3d-4f28-87ee-186ddac1d391
[2017-05-10 15:40:12,510: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c01095bf-3e3d-4f28-87ee-186ddac1d391
[2017-05-10 15:40:12,577: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:12,577: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:12,577: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:12,577: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:12,577: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c01095bf-3e3d-4f28-87ee-186ddac1d391
[2017-05-10 15:40:12,577: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c01095bf-3e3d-4f28-87ee-186ddac1d391
[2017-05-10 15:40:12,624: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c01095bf-3e3d-4f28-87ee-186ddac1d391
[2017-05-10 15:40:12,624: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c01095bf-3e3d-4f28-87ee-186ddac1d391
[2017-05-10 15:40:12,667: INFO/job_scheduler] Job 192 finished 2 steps successfully
[2017-05-10 15:40:12,690: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_up->lnet_down (intentional=True) 140430199249168
[2017-05-10 15:40:12,696: INFO/job] Job 192: StateChangeJob complete, setting state lnet_down on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 15:40:12,696: INFO/job] Job 192: not_deleted=True
[2017-05-10 15:40:12,697: INFO/job_scheduler] Job 192: complete_job: Updating cache
[2017-05-10 15:40:12,697: INFO/job_scheduler] Job 192: locked_item 140430199249168 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 15:40:12,700: INFO/job_scheduler] Job 192 completing (errored=False, cancelled=False)
[2017-05-10 15:40:12,711: INFO/job_scheduler] run_next: 1 runnable jobs of (4 pending, 1 tasked)
[2017-05-10 15:40:12,718: INFO/job_scheduler] Job 193: RunJobThread.run
[2017-05-10 15:40:12,719: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unload_lnet {}
[2017-05-10 15:40:12,734: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7aad6b07-0fd1-4015-a1ed-8e96bc479b46
[2017-05-10 15:40:12,734: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7aad6b07-0fd1-4015-a1ed-8e96bc479b46
[2017-05-10 15:40:13,714: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 2 tasked)
[2017-05-10 15:40:13,728: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 2 tasked)
[2017-05-10 15:40:14,502: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 15:40:14,502: INFO/job_scheduler]   Job 194 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration started->stopped
[2017-05-10 15:40:14,503: INFO/job_scheduler]   Job 195 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->unconfigured
[2017-05-10 15:40:14,503: INFO/job_scheduler]   Job 197 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 15:40:14,503: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 2 tasked)
[2017-05-10 15:40:14,833: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:14,833: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:14,833: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:14,833: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:14,834: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7aad6b07-0fd1-4015-a1ed-8e96bc479b46
[2017-05-10 15:40:14,834: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7aad6b07-0fd1-4015-a1ed-8e96bc479b46
[2017-05-10 15:40:14,848: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7aad6b07-0fd1-4015-a1ed-8e96bc479b46
[2017-05-10 15:40:14,848: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7aad6b07-0fd1-4015-a1ed-8e96bc479b46
[2017-05-10 15:40:14,849: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 15:40:14,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e52b856c-23c5-40db-a340-7378ef53bcbe
[2017-05-10 15:40:14,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e52b856c-23c5-40db-a340-7378ef53bcbe
[2017-05-10 15:40:14,951: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:14,951: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:14,952: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:14,952: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:14,952: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e52b856c-23c5-40db-a340-7378ef53bcbe
[2017-05-10 15:40:14,952: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e52b856c-23c5-40db-a340-7378ef53bcbe
[2017-05-10 15:40:14,982: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e52b856c-23c5-40db-a340-7378ef53bcbe
[2017-05-10 15:40:14,982: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e52b856c-23c5-40db-a340-7378ef53bcbe
[2017-05-10 15:40:15,102: INFO/job_scheduler] Job 193 finished 2 steps successfully
[2017-05-10 15:40:15,157: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_unloaded (intentional=True) 140430199249168
[2017-05-10 15:40:15,162: INFO/job] Job 193: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 15:40:15,162: INFO/job] Job 193: not_deleted=True
[2017-05-10 15:40:15,163: INFO/job_scheduler] Job 193: complete_job: Updating cache
[2017-05-10 15:40:15,163: INFO/job_scheduler] Job 193: locked_item 140430199249168 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 15:40:15,169: INFO/job_scheduler] Job 193 completing (errored=False, cancelled=False)
[2017-05-10 15:40:15,183: INFO/job_scheduler] run_next: 1 runnable jobs of (3 pending, 1 tasked)
[2017-05-10 15:40:15,187: INFO/job_scheduler] Job 196: RunJobThread.run
[2017-05-10 15:40:15,188: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unconfigure_lnet {}
[2017-05-10 15:40:15,201: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2442b06b-77f8-4912-b174-314deffcbc72
[2017-05-10 15:40:15,201: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2442b06b-77f8-4912-b174-314deffcbc72
[2017-05-10 15:40:15,277: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:15,277: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:15,277: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:15,277: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:15,277: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2442b06b-77f8-4912-b174-314deffcbc72
[2017-05-10 15:40:15,277: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2442b06b-77f8-4912-b174-314deffcbc72
[2017-05-10 15:40:15,318: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2442b06b-77f8-4912-b174-314deffcbc72
[2017-05-10 15:40:15,318: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2442b06b-77f8-4912-b174-314deffcbc72
[2017-05-10 15:40:15,318: INFO/job] Invalid result None fixed up on called to unconfigure_lnet with args {}
[2017-05-10 15:40:15,318: INFO/job_scheduler] Job 196 finished 1 steps successfully
[2017-05-10 15:40:15,349: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->unconfigured (intentional=True) 140430199249168
[2017-05-10 15:40:15,358: INFO/job] Job 196: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 15:40:15,358: INFO/job] Job 196: not_deleted=True
[2017-05-10 15:40:15,359: INFO/job_scheduler] Job 196: complete_job: Updating cache
[2017-05-10 15:40:15,359: INFO/job_scheduler] Job 196: locked_item 140430199249168 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 15:40:15,361: INFO/job_scheduler] Job 196 completing (errored=False, cancelled=False)
[2017-05-10 15:40:15,375: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-05-10 15:40:16,655: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:16,655: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:16,656: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:16,656: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:16,656: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 79293c9a-cab0-4693-978e-ae0c262f10af
[2017-05-10 15:40:16,656: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 79293c9a-cab0-4693-978e-ae0c262f10af
[2017-05-10 15:40:16,690: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 79293c9a-cab0-4693-978e-ae0c262f10af
[2017-05-10 15:40:16,690: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 79293c9a-cab0-4693-978e-ae0c262f10af
[2017-05-10 15:40:16,690: INFO/job_scheduler] Job 194 finished 1 steps successfully
[2017-05-10 15:40:16,724: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration started->stopped (intentional=True) 140430198505808
[2017-05-10 15:40:16,738: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-10 15:40:16,744: INFO/job] Job 194: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 15:40:16,745: INFO/job] Job 194: not_deleted=True
[2017-05-10 15:40:16,745: INFO/job_scheduler] Job 194: complete_job: Updating cache
[2017-05-10 15:40:16,745: INFO/job_scheduler] Job 194: locked_item 140430198505808 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 15:40:16,763: INFO/job_scheduler] Job 194 completing (errored=False, cancelled=False)
[2017-05-10 15:40:16,783: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 0 tasked)
[2017-05-10 15:40:16,788: INFO/job_scheduler] Job 195: RunJobThread.run
[2017-05-10 15:40:16,789: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unconfigure_corosync2 {'mcast_port': 62535, 'host_fqdn': u'lotus-32vm5.lotus.hpdd.lab.intel.com'}
[2017-05-10 15:40:16,818: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4bae8a13-a00e-4834-abd5-60ff3090bd05
[2017-05-10 15:40:16,818: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4bae8a13-a00e-4834-abd5-60ff3090bd05
[2017-05-10 15:40:23,907: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:23,907: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:23,907: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:23,907: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:23,907: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4bae8a13-a00e-4834-abd5-60ff3090bd05
[2017-05-10 15:40:23,907: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4bae8a13-a00e-4834-abd5-60ff3090bd05
[2017-05-10 15:40:23,934: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4bae8a13-a00e-4834-abd5-60ff3090bd05
[2017-05-10 15:40:23,934: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4bae8a13-a00e-4834-abd5-60ff3090bd05
[2017-05-10 15:40:23,934: INFO/job_scheduler] Job 195 finished 1 steps successfully
[2017-05-10 15:40:23,997: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->unconfigured (intentional=True) 140430198505808
[2017-05-10 15:40:24,003: INFO/job] Job 195: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 15:40:24,003: INFO/job] Job 195: not_deleted=True
[2017-05-10 15:40:24,003: INFO/job_scheduler] Job 195: complete_job: Updating cache
[2017-05-10 15:40:24,004: INFO/job_scheduler] Job 195: locked_item 140430198505808 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 15:40:24,008: INFO/job_scheduler] Job 195 completing (errored=False, cancelled=False)
[2017-05-10 15:40:24,024: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 15:40:24,055: INFO/job_scheduler] Job 197: RunJobThread.run
[2017-05-10 15:40:24,058: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com deregister_server {}
[2017-05-10 15:40:24,086: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2cf2797f-7042-45c5-8373-b9c70e7e8137
[2017-05-10 15:40:24,086: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2cf2797f-7042-45c5-8373-b9c70e7e8137
[2017-05-10 15:40:24,157: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:24,157: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:24,157: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:24,157: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/e98329c9-bb2d-4554-a127-b1e494e55184
[2017-05-10 15:40:24,157: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2cf2797f-7042-45c5-8373-b9c70e7e8137
[2017-05-10 15:40:24,157: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2cf2797f-7042-45c5-8373-b9c70e7e8137
[2017-05-10 15:40:24,203: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2cf2797f-7042-45c5-8373-b9c70e7e8137
[2017-05-10 15:40:24,203: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2cf2797f-7042-45c5-8373-b9c70e7e8137
[2017-05-10 15:40:24,228: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 15:40:24,228: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration unconfigured->removed (intentional=False) 140430199457808
[2017-05-10 15:40:24,241: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> 5
[2017-05-10 15:40:24,267: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 15:40:24,267: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->removed (intentional=False) 140430199295440
[2017-05-10 15:40:24,282: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.corosync2.Corosync2Configuration'> 5
[2017-05-10 15:40:24,306: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 15:40:24,306: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->removed (intentional=False) 140430199455888
[2017-05-10 15:40:24,314: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> 5
[2017-05-10 15:40:24,341: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 15:40:24,341: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration unconfigured->removed (intentional=False) 140430199374416
[2017-05-10 15:40:24,350: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.ntp.NTPConfiguration'> 5
[2017-05-10 15:40:24,368: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 15:40:24,368: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->removed (intentional=False) 140430199458128
[2017-05-10 15:40:24,378: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> 5
[2017-05-10 15:40:24,579: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-05-10 15:40:24,579: INFO/job_scheduler]   Job 197 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 15:40:24,579: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-05-10 15:40:25,044: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.host.ManagedHost'> 5
[2017-05-10 15:40:25,049: INFO/job_scheduler] Job 197 finished 2 steps successfully
[2017-05-10 15:40:25,108: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com managed->removed (intentional=True) 140430198775248
[2017-05-10 15:40:25,114: INFO/job] Job 197: StateChangeJob complete, setting state removed on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-05-10 15:40:25,114: INFO/job] Job 197: not_deleted=None
[2017-05-10 15:40:25,114: INFO/job_scheduler] Job 197: complete_job: Updating cache
[2017-05-10 15:40:25,114: INFO/job_scheduler] Job 197: locked_item 140430198775248 <class 'chroma_core.models.host.ManagedHost'> True None
[2017-05-10 15:40:25,114: INFO/job_scheduler] Job 197 completing (errored=False, cancelled=False)
[2017-05-10 15:40:25,188: WARNING/job_scheduler] _drain_notification_buffer: Dropping buffered notifications for not-found object (u'chroma_core', u'corosync2configuration')/5
[2017-05-10 15:40:25,188: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-05-10 15:40:37,348: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-05-10 15:40:37,352: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-6 transitioning from managed to removed
[2017-05-10 15:40:37,455: INFO/command_plan] Created command 48 (Remove host lotus-32vm6.lotus.hpdd.lab.intel.com from configuration) with 9 jobs
[2017-05-10 15:40:37,509: INFO/job_scheduler] run_next: 4 runnable jobs of (9 pending, 0 tasked)
[2017-05-10 15:40:37,575: INFO/job_scheduler] Job 198: RunJobThread.run
[2017-05-10 15:40:37,575: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-05-10 15:40:37,616: INFO/job_scheduler] Job 199: RunJobThread.run
[2017-05-10 15:40:37,617: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com unconfigure_rsyslog {}
[2017-05-10 15:40:37,619: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8f2b2d9e-debc-408e-927c-4ee9f22aaeec
[2017-05-10 15:40:37,619: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8f2b2d9e-debc-408e-927c-4ee9f22aaeec
[2017-05-10 15:40:37,635: INFO/job_scheduler] Job 200: RunJobThread.run
[2017-05-10 15:40:37,636: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com stop_lnet {}
[2017-05-10 15:40:37,640: INFO/job_scheduler] Job 201: RunJobThread.run
[2017-05-10 15:40:37,641: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com unconfigure_ntp {}
[2017-05-10 15:40:37,657: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 68b780d0-ab12-4565-927c-06a449981825
[2017-05-10 15:40:37,657: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 68b780d0-ab12-4565-927c-06a449981825
[2017-05-10 15:40:37,681: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0f5d1ea5-1c71-4401-822f-b0e7a7e04b5a
[2017-05-10 15:40:37,681: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0f5d1ea5-1c71-4401-822f-b0e7a7e04b5a
[2017-05-10 15:40:37,713: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5fdec1f0-4b02-464e-be16-765e78a52bbb
[2017-05-10 15:40:37,713: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5fdec1f0-4b02-464e-be16-765e78a52bbb
[2017-05-10 15:40:43,836: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:43,836: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:43,837: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:43,837: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:43,837: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5fdec1f0-4b02-464e-be16-765e78a52bbb
[2017-05-10 15:40:43,837: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5fdec1f0-4b02-464e-be16-765e78a52bbb
[2017-05-10 15:40:43,879: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5fdec1f0-4b02-464e-be16-765e78a52bbb
[2017-05-10 15:40:43,879: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5fdec1f0-4b02-464e-be16-765e78a52bbb
[2017-05-10 15:40:43,880: INFO/job_scheduler] Job 201 finished 1 steps successfully
[2017-05-10 15:40:44,056: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration configured->unconfigured (intentional=True) 140430198655184
[2017-05-10 15:40:44,060: INFO/job] Job 201: StateChangeJob complete, setting state unconfigured on lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration
[2017-05-10 15:40:44,060: INFO/job] Job 201: not_deleted=True
[2017-05-10 15:40:44,060: INFO/job_scheduler] Job 201: complete_job: Updating cache
[2017-05-10 15:40:44,060: INFO/job_scheduler] Job 201: locked_item 140430198655184 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-05-10 15:40:44,064: INFO/job_scheduler] Job 201 completing (errored=False, cancelled=False)
[2017-05-10 15:40:44,084: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 3 tasked)
[2017-05-10 15:40:44,583: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 3 tasked)
[2017-05-10 15:40:44,599: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 3 tasked)
[2017-05-10 15:40:44,685: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:44,685: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:44,685: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:44,685: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:44,686: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8f2b2d9e-debc-408e-927c-4ee9f22aaeec
[2017-05-10 15:40:44,686: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8f2b2d9e-debc-408e-927c-4ee9f22aaeec
[2017-05-10 15:40:44,735: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8f2b2d9e-debc-408e-927c-4ee9f22aaeec
[2017-05-10 15:40:44,735: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8f2b2d9e-debc-408e-927c-4ee9f22aaeec
[2017-05-10 15:40:44,736: INFO/job_scheduler] Job 198 finished 1 steps successfully
[2017-05-10 15:40:44,780: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration started->stopped (intentional=True) 140430199286160
[2017-05-10 15:40:44,789: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-05-10 15:40:44,795: INFO/job] Job 198: StateChangeJob complete, setting state stopped on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-05-10 15:40:44,796: INFO/job] Job 198: not_deleted=True
[2017-05-10 15:40:44,797: INFO/job_scheduler] Job 198: complete_job: Updating cache
[2017-05-10 15:40:44,798: INFO/job_scheduler] Job 198: locked_item 140430199286160 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-05-10 15:40:44,807: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:44,807: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:44,810: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:44,810: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:44,811: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 68b780d0-ab12-4565-927c-06a449981825
[2017-05-10 15:40:44,809: INFO/job_scheduler] Job 198 completing (errored=False, cancelled=False)
[2017-05-10 15:40:44,811: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 68b780d0-ab12-4565-927c-06a449981825
[2017-05-10 15:40:44,824: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 68b780d0-ab12-4565-927c-06a449981825
[2017-05-10 15:40:44,824: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 68b780d0-ab12-4565-927c-06a449981825
[2017-05-10 15:40:44,825: INFO/job_scheduler] Job 199 finished 1 steps successfully
[2017-05-10 15:40:44,839: INFO/job_scheduler] run_next: 1 runnable jobs of (5 pending, 2 tasked)
[2017-05-10 15:40:44,846: INFO/job_scheduler] Job 203: RunJobThread.run
[2017-05-10 15:40:44,847: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com stop_corosync2 {}
[2017-05-10 15:40:44,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc af393576-0106-4bc6-a8da-fae1e9c1a06b
[2017-05-10 15:40:44,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc af393576-0106-4bc6-a8da-fae1e9c1a06b
[2017-05-10 15:40:44,914: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration configured->unconfigured (intentional=True) 140430198327760
[2017-05-10 15:40:44,921: INFO/job] Job 199: StateChangeJob complete, setting state unconfigured on lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-05-10 15:40:44,921: INFO/job] Job 199: not_deleted=True
[2017-05-10 15:40:44,921: INFO/job_scheduler] Job 199: complete_job: Updating cache
[2017-05-10 15:40:44,921: INFO/job_scheduler] Job 199: locked_item 140430198327760 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-05-10 15:40:44,924: INFO/job_scheduler] Job 199 completing (errored=False, cancelled=False)
[2017-05-10 15:40:44,943: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-05-10 15:40:49,870: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:49,870: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:49,870: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:49,870: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:49,870: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0f5d1ea5-1c71-4401-822f-b0e7a7e04b5a
[2017-05-10 15:40:49,870: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0f5d1ea5-1c71-4401-822f-b0e7a7e04b5a
[2017-05-10 15:40:49,900: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0f5d1ea5-1c71-4401-822f-b0e7a7e04b5a
[2017-05-10 15:40:49,900: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0f5d1ea5-1c71-4401-822f-b0e7a7e04b5a
[2017-05-10 15:40:49,901: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 15:40:49,925: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8cea277f-24c7-4d6c-a31d-3e1beba2b081
[2017-05-10 15:40:49,925: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8cea277f-24c7-4d6c-a31d-3e1beba2b081
[2017-05-10 15:40:50,041: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:50,041: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:50,041: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:50,041: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:50,041: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8cea277f-24c7-4d6c-a31d-3e1beba2b081
[2017-05-10 15:40:50,041: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8cea277f-24c7-4d6c-a31d-3e1beba2b081
[2017-05-10 15:40:50,091: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8cea277f-24c7-4d6c-a31d-3e1beba2b081
[2017-05-10 15:40:50,091: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8cea277f-24c7-4d6c-a31d-3e1beba2b081
[2017-05-10 15:40:50,241: INFO/job_scheduler] Job 200 finished 2 steps successfully
[2017-05-10 15:40:50,263: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration lnet_up->lnet_down (intentional=True) 140430199251920
[2017-05-10 15:40:50,271: INFO/job] Job 200: StateChangeJob complete, setting state lnet_down on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 15:40:50,271: INFO/job] Job 200: not_deleted=True
[2017-05-10 15:40:50,272: INFO/job_scheduler] Job 200: complete_job: Updating cache
[2017-05-10 15:40:50,274: INFO/job_scheduler] Job 200: locked_item 140430199251920 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 15:40:50,277: INFO/job_scheduler] Job 200 completing (errored=False, cancelled=False)
[2017-05-10 15:40:50,300: INFO/job_scheduler] run_next: 1 runnable jobs of (4 pending, 1 tasked)
[2017-05-10 15:40:50,342: INFO/job_scheduler] Job 202: RunJobThread.run
[2017-05-10 15:40:50,343: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com unload_lnet {}
[2017-05-10 15:40:50,396: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a30bbe2e-de04-462a-bcb2-e6560677540f
[2017-05-10 15:40:50,396: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a30bbe2e-de04-462a-bcb2-e6560677540f
[2017-05-10 15:40:52,482: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:52,482: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:52,482: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:52,482: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:52,483: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a30bbe2e-de04-462a-bcb2-e6560677540f
[2017-05-10 15:40:52,483: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a30bbe2e-de04-462a-bcb2-e6560677540f
[2017-05-10 15:40:52,510: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a30bbe2e-de04-462a-bcb2-e6560677540f
[2017-05-10 15:40:52,510: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a30bbe2e-de04-462a-bcb2-e6560677540f
[2017-05-10 15:40:52,511: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-05-10 15:40:52,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7f90cd2a-3f98-44f7-8127-108cf72fe766
[2017-05-10 15:40:52,528: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7f90cd2a-3f98-44f7-8127-108cf72fe766
[2017-05-10 15:40:52,600: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:52,600: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:52,600: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:52,600: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:52,600: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7f90cd2a-3f98-44f7-8127-108cf72fe766
[2017-05-10 15:40:52,600: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7f90cd2a-3f98-44f7-8127-108cf72fe766
[2017-05-10 15:40:52,643: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7f90cd2a-3f98-44f7-8127-108cf72fe766
[2017-05-10 15:40:52,643: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7f90cd2a-3f98-44f7-8127-108cf72fe766
[2017-05-10 15:40:52,853: INFO/job_scheduler] Job 202 finished 2 steps successfully
[2017-05-10 15:40:52,897: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_unloaded (intentional=True) 140430199251920
[2017-05-10 15:40:52,905: INFO/job] Job 202: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 15:40:52,905: INFO/job] Job 202: not_deleted=True
[2017-05-10 15:40:52,906: INFO/job_scheduler] Job 202: complete_job: Updating cache
[2017-05-10 15:40:52,906: INFO/job_scheduler] Job 202: locked_item 140430199251920 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 15:40:52,910: INFO/job_scheduler] Job 202 completing (errored=False, cancelled=False)
[2017-05-10 15:40:52,931: INFO/job_scheduler] run_next: 1 runnable jobs of (3 pending, 1 tasked)
[2017-05-10 15:40:52,946: INFO/job_scheduler] Job 204: RunJobThread.run
[2017-05-10 15:40:52,947: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com unconfigure_lnet {}
[2017-05-10 15:40:52,964: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c40b0b27-6427-4cad-b49c-6f42ac3881d2
[2017-05-10 15:40:52,964: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c40b0b27-6427-4cad-b49c-6f42ac3881d2
[2017-05-10 15:40:52,980: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:52,980: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:52,980: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:52,980: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:52,980: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc af393576-0106-4bc6-a8da-fae1e9c1a06b
[2017-05-10 15:40:52,980: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc af393576-0106-4bc6-a8da-fae1e9c1a06b
[2017-05-10 15:40:52,992: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc af393576-0106-4bc6-a8da-fae1e9c1a06b
[2017-05-10 15:40:52,992: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc af393576-0106-4bc6-a8da-fae1e9c1a06b
[2017-05-10 15:40:52,993: INFO/job_scheduler] Job 203 finished 1 steps successfully
[2017-05-10 15:40:53,066: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration started->stopped (intentional=True) 140430199375696
[2017-05-10 15:40:53,081: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-05-10 15:40:53,097: INFO/job] Job 203: StateChangeJob complete, setting state stopped on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 15:40:53,097: INFO/job] Job 203: not_deleted=True
[2017-05-10 15:40:53,100: INFO/job_scheduler] Job 203: complete_job: Updating cache
[2017-05-10 15:40:53,100: INFO/job_scheduler] Job 203: locked_item 140430199375696 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 15:40:53,106: INFO/job_scheduler] Job 203 completing (errored=False, cancelled=False)
[2017-05-10 15:40:53,149: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 1 tasked)
[2017-05-10 15:40:53,153: INFO/job_scheduler] Job 205: RunJobThread.run
[2017-05-10 15:40:53,154: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com unconfigure_corosync2 {'mcast_port': 62535, 'host_fqdn': u'lotus-32vm6.lotus.hpdd.lab.intel.com'}
[2017-05-10 15:40:53,188: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc daf22be5-a0f3-49a2-b31c-7ce552425303
[2017-05-10 15:40:53,188: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc daf22be5-a0f3-49a2-b31c-7ce552425303
[2017-05-10 15:40:53,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:53,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:53,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:53,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:53,190: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c40b0b27-6427-4cad-b49c-6f42ac3881d2
[2017-05-10 15:40:53,190: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c40b0b27-6427-4cad-b49c-6f42ac3881d2
[2017-05-10 15:40:53,237: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c40b0b27-6427-4cad-b49c-6f42ac3881d2
[2017-05-10 15:40:53,237: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c40b0b27-6427-4cad-b49c-6f42ac3881d2
[2017-05-10 15:40:53,237: INFO/job] Invalid result None fixed up on called to unconfigure_lnet with args {}
[2017-05-10 15:40:53,237: INFO/job_scheduler] Job 204 finished 1 steps successfully
[2017-05-10 15:40:53,271: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->unconfigured (intentional=True) 140430199251920
[2017-05-10 15:40:53,278: INFO/job] Job 204: StateChangeJob complete, setting state unconfigured on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-05-10 15:40:53,278: INFO/job] Job 204: not_deleted=True
[2017-05-10 15:40:53,278: INFO/job_scheduler] Job 204: complete_job: Updating cache
[2017-05-10 15:40:53,278: INFO/job_scheduler] Job 204: locked_item 140430199251920 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-05-10 15:40:53,281: INFO/job_scheduler] Job 204 completing (errored=False, cancelled=False)
[2017-05-10 15:40:53,326: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 15:40:53,612: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 15:40:53,637: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-05-10 15:40:59,320: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:59,320: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:59,321: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:59,321: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:59,321: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc daf22be5-a0f3-49a2-b31c-7ce552425303
[2017-05-10 15:40:59,321: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc daf22be5-a0f3-49a2-b31c-7ce552425303
[2017-05-10 15:40:59,353: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc daf22be5-a0f3-49a2-b31c-7ce552425303
[2017-05-10 15:40:59,353: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc daf22be5-a0f3-49a2-b31c-7ce552425303
[2017-05-10 15:40:59,353: INFO/job_scheduler] Job 205 finished 1 steps successfully
[2017-05-10 15:40:59,417: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->unconfigured (intentional=True) 140430199375696
[2017-05-10 15:40:59,425: INFO/job] Job 205: StateChangeJob complete, setting state unconfigured on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-05-10 15:40:59,425: INFO/job] Job 205: not_deleted=True
[2017-05-10 15:40:59,425: INFO/job_scheduler] Job 205: complete_job: Updating cache
[2017-05-10 15:40:59,425: INFO/job_scheduler] Job 205: locked_item 140430199375696 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-05-10 15:40:59,429: INFO/job_scheduler] Job 205 completing (errored=False, cancelled=False)
[2017-05-10 15:40:59,448: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-05-10 15:40:59,487: INFO/job_scheduler] Job 206: RunJobThread.run
[2017-05-10 15:40:59,490: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com deregister_server {}
[2017-05-10 15:40:59,513: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 18c83e52-1710-4dfa-871f-d492fe40c870
[2017-05-10 15:40:59,513: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 18c83e52-1710-4dfa-871f-d492fe40c870
[2017-05-10 15:40:59,593: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:59,593: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:59,593: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:59,593: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e7179537-8887-4c9c-88da-9c3bb2764f28
[2017-05-10 15:40:59,594: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 18c83e52-1710-4dfa-871f-d492fe40c870
[2017-05-10 15:40:59,594: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 18c83e52-1710-4dfa-871f-d492fe40c870
[2017-05-10 15:40:59,635: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 18c83e52-1710-4dfa-871f-d492fe40c870
[2017-05-10 15:40:59,635: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 18c83e52-1710-4dfa-871f-d492fe40c870
[2017-05-10 15:40:59,663: INFO/job] Deleting dependent lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration for host lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 15:40:59,664: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->removed (intentional=False) 140430199238032
[2017-05-10 15:40:59,676: INFO/job] Lowered 2 alerts while deleting <class 'chroma_core.models.corosync2.Corosync2Configuration'> 6
[2017-05-10 15:40:59,690: INFO/job] Deleting dependent lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration for host lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 15:40:59,691: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->removed (intentional=False) 140430199264272
[2017-05-10 15:40:59,696: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> 6
[2017-05-10 15:40:59,714: INFO/job] Deleting dependent lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration for host lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 15:40:59,714: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration unconfigured->removed (intentional=False) 140430199349520
[2017-05-10 15:40:59,723: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> 6
[2017-05-10 15:40:59,739: INFO/job] Deleting dependent lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration for host lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 15:40:59,739: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->removed (intentional=False) 140430199280272
[2017-05-10 15:40:59,748: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> 6
[2017-05-10 15:40:59,767: INFO/job] Deleting dependent lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration for host lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 15:40:59,768: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration unconfigured->removed (intentional=False) 140430199283344
[2017-05-10 15:40:59,774: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.ntp.NTPConfiguration'> 6
[2017-05-10 15:41:00,384: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.host.ManagedHost'> 6
[2017-05-10 15:41:00,390: INFO/job_scheduler] Job 206 finished 2 steps successfully
[2017-05-10 15:41:00,415: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com managed->removed (intentional=True) 140430199480272
[2017-05-10 15:41:00,420: INFO/job] Job 206: StateChangeJob complete, setting state removed on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-05-10 15:41:00,420: INFO/job] Job 206: not_deleted=None
[2017-05-10 15:41:00,420: INFO/job_scheduler] Job 206: complete_job: Updating cache
[2017-05-10 15:41:00,420: INFO/job_scheduler] Job 206: locked_item 140430199480272 <class 'chroma_core.models.host.ManagedHost'> True None
[2017-05-10 15:41:00,421: INFO/job_scheduler] Job 206 completing (errored=False, cancelled=False)
[2017-05-10 15:41:00,473: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 15:08:12,870: INFO/chroma_service] Stopping job_scheduler
[2017-06-01 15:08:12,870: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-06-01 15:08:12,870: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-06-01 15:08:12,871: INFO/queue] Stopping ServiceQueue agent_action_runner_rx
[2017-06-01 15:08:13,205: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-06-01 15:08:13,205: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-06-01 15:08:13,206: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-06-01 15:08:13,206: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-06-01 15:08:13,206: INFO/job_scheduler] Stopping...
[2017-06-01 15:08:13,206: INFO/queue] Stopping ServiceQueue job_scheduler_notifications
[2017-06-01 15:08:13,207: INFO/job_scheduler] Joining...
[2017-06-01 15:08:14,081: INFO/job_scheduler] Complete.
[2017-06-01 15:08:14,081: INFO/chroma_service] Joining job_scheduler
[2017-06-01 15:08:14,081: INFO/job_scheduler] Cancelling outstanding jobs...
[2017-06-01 18:32:46,811: INFO/queue] Purged 0 messages from 'job_scheduler_notifications' queue
[2017-06-01 18:32:46,842: INFO/queue] Purged 0 messages from 'agent_action_runner_rx' queue
[2017-06-01 18:37:47,264: WARNING/chroma_core.services.job_scheduler.agent_rpc] Unable to reset action_runner sessions
[2017-06-01 18:37:47,264: WARNING/chroma_core.services.job_scheduler.agent_rpc] Unable to reset action_runner sessions
[2017-06-01 18:37:47,276: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: None/None
[2017-06-01 18:37:47,276: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: None/None
[2017-06-01 18:48:46,839: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:48:46,840: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:48:46,852: INFO/command_plan] add_jobs: created Job 207 (Test for host connectivity)
[2017-06-01 18:48:46,885: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:48:46,910: INFO/job_scheduler] Job 207: RunJobThread.run
[2017-06-01 18:48:46,974: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:48:46,975: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:48:46,991: INFO/command_plan] add_jobs: created Job 208 (Test for host connectivity)
[2017-06-01 18:48:46,999: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:48:47,004: INFO/job_scheduler] Job 208: RunJobThread.run
[2017-06-01 18:48:47,108: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:48:47,116: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:48:47,131: INFO/command_plan] add_jobs: created Job 209 (Test for host connectivity)
[2017-06-01 18:48:47,204: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:48:47,205: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:48:47,248: INFO/command_plan] add_jobs: created Job 210 (Test for host connectivity)
[2017-06-01 18:48:47,260: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 18:48:47,273: INFO/job_scheduler] Job 209: RunJobThread.run
[2017-06-01 18:48:47,296: INFO/job_scheduler] Job 210: RunJobThread.run
[2017-06-01 18:48:47,308: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 18:48:51,357: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:48:51,463: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:48:51,711: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:48:51,920: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:48:52,682: INFO/job_scheduler] Job 207 finished 1 steps successfully
[2017-06-01 18:48:52,723: INFO/job_scheduler] Job 207: complete_job: Updating cache
[2017-06-01 18:48:52,740: INFO/job_scheduler] Job 207 completing (errored=False, cancelled=False)
[2017-06-01 18:48:52,789: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:48:52,790: INFO/job_scheduler] Job 208 finished 1 steps successfully
[2017-06-01 18:48:52,809: INFO/job_scheduler] Job 208: complete_job: Updating cache
[2017-06-01 18:48:52,827: INFO/job_scheduler] Job 208 completing (errored=False, cancelled=False)
[2017-06-01 18:48:52,885: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:48:53,051: INFO/job_scheduler] Job 210 finished 1 steps successfully
[2017-06-01 18:48:53,081: INFO/job_scheduler] Job 210: complete_job: Updating cache
[2017-06-01 18:48:53,081: INFO/job_scheduler] Job 210 completing (errored=False, cancelled=False)
[2017-06-01 18:48:53,124: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:48:53,276: INFO/job_scheduler] Job 209 finished 1 steps successfully
[2017-06-01 18:48:53,306: INFO/job_scheduler] Job 209: complete_job: Updating cache
[2017-06-01 18:48:53,306: INFO/job_scheduler] Job 209 completing (errored=False, cancelled=False)
[2017-06-01 18:48:53,345: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:48:55,353: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:48:55,356: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:48:55,372: INFO/command_plan] add_jobs: created Job 211 (Test for host connectivity)
[2017-06-01 18:48:55,390: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:48:55,399: INFO/job_scheduler] Job 211: RunJobThread.run
[2017-06-01 18:48:55,427: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:48:55,428: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:48:55,454: INFO/command_plan] add_jobs: created Job 212 (Test for host connectivity)
[2017-06-01 18:48:55,506: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:48:55,541: INFO/job_scheduler] Job 212: RunJobThread.run
[2017-06-01 18:48:55,561: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:48:55,562: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:48:55,606: INFO/command_plan] add_jobs: created Job 213 (Test for host connectivity)
[2017-06-01 18:48:55,629: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:48:55,647: INFO/job_scheduler] Job 213: RunJobThread.run
[2017-06-01 18:48:55,722: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:48:55,722: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:48:55,748: INFO/command_plan] add_jobs: created Job 214 (Test for host connectivity)
[2017-06-01 18:48:55,799: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:48:55,806: INFO/job_scheduler] Job 214: RunJobThread.run
[2017-06-01 18:48:57,672: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:48:57,980: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:48:58,069: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:48:58,071: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:48:59,000: INFO/job_scheduler] Job 211 finished 1 steps successfully
[2017-06-01 18:48:59,198: INFO/job_scheduler] Job 211: complete_job: Updating cache
[2017-06-01 18:48:59,198: INFO/job_scheduler] Job 211 completing (errored=False, cancelled=False)
[2017-06-01 18:48:59,289: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:48:59,403: INFO/job_scheduler] Job 212 finished 1 steps successfully
[2017-06-01 18:48:59,432: INFO/job_scheduler] Job 213 finished 1 steps successfully
[2017-06-01 18:48:59,489: INFO/job_scheduler] Job 212: complete_job: Updating cache
[2017-06-01 18:48:59,489: INFO/job_scheduler] Job 212 completing (errored=False, cancelled=False)
[2017-06-01 18:48:59,521: INFO/job_scheduler] Job 214 finished 1 steps successfully
[2017-06-01 18:48:59,555: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:48:59,564: INFO/job_scheduler] Job 213: complete_job: Updating cache
[2017-06-01 18:48:59,564: INFO/job_scheduler] Job 213 completing (errored=False, cancelled=False)
[2017-06-01 18:48:59,623: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:48:59,648: INFO/job_scheduler] Job 214: complete_job: Updating cache
[2017-06-01 18:48:59,648: INFO/job_scheduler] Job 214 completing (errored=False, cancelled=False)
[2017-06-01 18:48:59,731: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:49:01,398: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:01,407: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:01,430: INFO/command_plan] add_jobs: created Job 215 (Test for host connectivity)
[2017-06-01 18:49:01,450: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:49:01,454: INFO/job_scheduler] Job 215: RunJobThread.run
[2017-06-01 18:49:01,473: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:01,474: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:01,500: INFO/command_plan] add_jobs: created Job 216 (Test for host connectivity)
[2017-06-01 18:49:01,520: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:49:01,547: INFO/job_scheduler] Job 216: RunJobThread.run
[2017-06-01 18:49:01,617: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:01,617: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:01,638: INFO/command_plan] add_jobs: created Job 217 (Test for host connectivity)
[2017-06-01 18:49:01,663: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:49:01,675: INFO/job_scheduler] Job 217: RunJobThread.run
[2017-06-01 18:49:01,726: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:01,733: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:01,764: INFO/command_plan] add_jobs: created Job 218 (Test for host connectivity)
[2017-06-01 18:49:01,788: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:49:01,828: INFO/job_scheduler] Job 218: RunJobThread.run
[2017-06-01 18:49:03,666: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:49:03,778: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:49:03,980: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:49:04,146: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:49:05,050: INFO/job_scheduler] Job 215 finished 1 steps successfully
[2017-06-01 18:49:05,123: INFO/job_scheduler] Job 215: complete_job: Updating cache
[2017-06-01 18:49:05,123: INFO/job_scheduler] Job 215 completing (errored=False, cancelled=False)
[2017-06-01 18:49:05,136: INFO/job_scheduler] Job 216 finished 1 steps successfully
[2017-06-01 18:49:05,193: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:49:05,313: INFO/job_scheduler] Job 216: complete_job: Updating cache
[2017-06-01 18:49:05,314: INFO/job_scheduler] Job 216 completing (errored=False, cancelled=False)
[2017-06-01 18:49:05,357: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:49:05,359: INFO/job_scheduler] Job 217 finished 1 steps successfully
[2017-06-01 18:49:05,373: INFO/job_scheduler] Job 217: complete_job: Updating cache
[2017-06-01 18:49:05,373: INFO/job_scheduler] Job 217 completing (errored=False, cancelled=False)
[2017-06-01 18:49:05,426: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:49:05,519: INFO/job_scheduler] Job 218 finished 1 steps successfully
[2017-06-01 18:49:05,563: INFO/job_scheduler] Job 218: complete_job: Updating cache
[2017-06-01 18:49:05,563: INFO/job_scheduler] Job 218 completing (errored=False, cancelled=False)
[2017-06-01 18:49:05,605: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:49:07,393: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:07,394: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:07,405: INFO/command_plan] add_jobs: created Job 219 (Test for host connectivity)
[2017-06-01 18:49:07,423: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:49:07,427: INFO/job_scheduler] Job 219: RunJobThread.run
[2017-06-01 18:49:07,479: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:07,479: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:07,564: INFO/command_plan] add_jobs: created Job 220 (Test for host connectivity)
[2017-06-01 18:49:07,605: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:49:07,607: INFO/job_scheduler] Job 220: RunJobThread.run
[2017-06-01 18:49:07,673: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:07,673: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:07,688: INFO/command_plan] add_jobs: created Job 221 (Test for host connectivity)
[2017-06-01 18:49:07,743: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:07,743: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:07,780: INFO/command_plan] add_jobs: created Job 222 (Test for host connectivity)
[2017-06-01 18:49:07,797: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 18:49:07,823: INFO/job_scheduler] Job 221: RunJobThread.run
[2017-06-01 18:49:07,823: INFO/job_scheduler] Job 222: RunJobThread.run
[2017-06-01 18:49:07,841: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 18:49:09,661: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:49:09,777: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:49:10,095: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:49:10,144: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:49:11,035: INFO/job_scheduler] Job 219 finished 1 steps successfully
[2017-06-01 18:49:11,054: INFO/job_scheduler] Job 219: complete_job: Updating cache
[2017-06-01 18:49:11,055: INFO/job_scheduler] Job 219 completing (errored=False, cancelled=False)
[2017-06-01 18:49:11,114: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:49:11,115: INFO/job_scheduler] Job 220 finished 1 steps successfully
[2017-06-01 18:49:11,142: INFO/job_scheduler] Job 220: complete_job: Updating cache
[2017-06-01 18:49:11,142: INFO/job_scheduler] Job 220 completing (errored=False, cancelled=False)
[2017-06-01 18:49:11,239: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:49:11,500: INFO/job_scheduler] Job 222 finished 1 steps successfully
[2017-06-01 18:49:11,538: INFO/job_scheduler] Job 222: complete_job: Updating cache
[2017-06-01 18:49:11,538: INFO/job_scheduler] Job 222 completing (errored=False, cancelled=False)
[2017-06-01 18:49:11,553: INFO/job_scheduler] Job 221 finished 1 steps successfully
[2017-06-01 18:49:11,598: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:49:11,606: INFO/job_scheduler] Job 221: complete_job: Updating cache
[2017-06-01 18:49:11,606: INFO/job_scheduler] Job 221 completing (errored=False, cancelled=False)
[2017-06-01 18:49:11,666: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:49:13,570: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:13,570: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:13,589: INFO/command_plan] add_jobs: created Job 223 (Test for host connectivity)
[2017-06-01 18:49:13,622: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:13,623: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:13,647: INFO/command_plan] add_jobs: created Job 224 (Test for host connectivity)
[2017-06-01 18:49:13,658: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:49:13,672: INFO/job_scheduler] Job 224: RunJobThread.run
[2017-06-01 18:49:13,699: INFO/job_scheduler] Job 223: RunJobThread.run
[2017-06-01 18:49:13,706: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:49:13,770: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:13,771: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:13,855: INFO/command_plan] add_jobs: created Job 225 (Test for host connectivity)
[2017-06-01 18:49:13,871: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:49:13,902: INFO/job_scheduler] Job 225: RunJobThread.run
[2017-06-01 18:49:13,931: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:13,932: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:13,955: INFO/command_plan] add_jobs: created Job 226 (Test for host connectivity)
[2017-06-01 18:49:13,971: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:49:14,006: INFO/job_scheduler] Job 226: RunJobThread.run
[2017-06-01 18:49:15,901: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:49:15,912: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:49:16,128: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:49:16,212: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:49:17,256: INFO/job_scheduler] Job 223 finished 1 steps successfully
[2017-06-01 18:49:17,329: INFO/job_scheduler] Job 224 finished 1 steps successfully
[2017-06-01 18:49:17,363: INFO/job_scheduler] Job 223: complete_job: Updating cache
[2017-06-01 18:49:17,363: INFO/job_scheduler] Job 223 completing (errored=False, cancelled=False)
[2017-06-01 18:49:17,413: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:49:17,435: INFO/job_scheduler] Job 224: complete_job: Updating cache
[2017-06-01 18:49:17,435: INFO/job_scheduler] Job 224 completing (errored=False, cancelled=False)
[2017-06-01 18:49:17,468: INFO/job_scheduler] Job 225 finished 1 steps successfully
[2017-06-01 18:49:17,496: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:49:17,514: INFO/job_scheduler] Job 225: complete_job: Updating cache
[2017-06-01 18:49:17,514: INFO/job_scheduler] Job 225 completing (errored=False, cancelled=False)
[2017-06-01 18:49:17,572: INFO/job_scheduler] Job 226 finished 1 steps successfully
[2017-06-01 18:49:17,572: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:49:17,581: INFO/job_scheduler] Job 226: complete_job: Updating cache
[2017-06-01 18:49:17,582: INFO/job_scheduler] Job 226 completing (errored=False, cancelled=False)
[2017-06-01 18:49:17,642: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:49:19,592: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:19,592: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:19,613: INFO/command_plan] add_jobs: created Job 227 (Test for host connectivity)
[2017-06-01 18:49:19,643: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:49:19,649: INFO/job_scheduler] Job 227: RunJobThread.run
[2017-06-01 18:49:19,701: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:19,703: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:19,760: INFO/command_plan] add_jobs: created Job 228 (Test for host connectivity)
[2017-06-01 18:49:19,781: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:49:19,796: INFO/job_scheduler] Job 228: RunJobThread.run
[2017-06-01 18:49:19,879: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:19,880: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:19,900: INFO/command_plan] add_jobs: created Job 229 (Test for host connectivity)
[2017-06-01 18:49:19,982: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:19,982: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:20,004: INFO/command_plan] add_jobs: created Job 230 (Test for host connectivity)
[2017-06-01 18:49:20,013: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 18:49:20,019: INFO/job_scheduler] Job 229: RunJobThread.run
[2017-06-01 18:49:20,032: INFO/job_scheduler] Job 230: RunJobThread.run
[2017-06-01 18:49:20,040: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 18:49:21,871: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:49:22,072: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:49:22,333: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:49:22,431: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:49:23,229: INFO/job_scheduler] Job 227 finished 1 steps successfully
[2017-06-01 18:49:23,245: INFO/job_scheduler] Job 227: complete_job: Updating cache
[2017-06-01 18:49:23,246: INFO/job_scheduler] Job 227 completing (errored=False, cancelled=False)
[2017-06-01 18:49:23,299: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:49:23,409: INFO/job_scheduler] Job 228 finished 1 steps successfully
[2017-06-01 18:49:23,454: INFO/job_scheduler] Job 228: complete_job: Updating cache
[2017-06-01 18:49:23,454: INFO/job_scheduler] Job 228 completing (errored=False, cancelled=False)
[2017-06-01 18:49:23,520: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:49:23,710: INFO/job_scheduler] Job 229 finished 1 steps successfully
[2017-06-01 18:49:23,762: INFO/job_scheduler] Job 229: complete_job: Updating cache
[2017-06-01 18:49:23,762: INFO/job_scheduler] Job 229 completing (errored=False, cancelled=False)
[2017-06-01 18:49:23,798: INFO/job_scheduler] Job 230 finished 1 steps successfully
[2017-06-01 18:49:23,820: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:49:23,828: INFO/job_scheduler] Job 230: complete_job: Updating cache
[2017-06-01 18:49:23,829: INFO/job_scheduler] Job 230 completing (errored=False, cancelled=False)
[2017-06-01 18:49:23,866: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:49:25,629: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:25,631: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:25,653: INFO/command_plan] add_jobs: created Job 231 (Test for host connectivity)
[2017-06-01 18:49:25,684: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:49:25,699: INFO/job_scheduler] Job 231: RunJobThread.run
[2017-06-01 18:49:25,728: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:25,728: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:25,749: INFO/command_plan] add_jobs: created Job 232 (Test for host connectivity)
[2017-06-01 18:49:25,798: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:25,799: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:25,820: INFO/command_plan] add_jobs: created Job 233 (Test for host connectivity)
[2017-06-01 18:49:25,945: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 18:49:25,948: INFO/job_scheduler] Job 232: RunJobThread.run
[2017-06-01 18:49:25,949: INFO/job_scheduler] Job 233: RunJobThread.run
[2017-06-01 18:49:25,993: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:25,995: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:26,011: INFO/command_plan] add_jobs: created Job 234 (Test for host connectivity)
[2017-06-01 18:49:26,021: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:49:26,023: INFO/job_scheduler] Job 234: RunJobThread.run
[2017-06-01 18:49:26,112: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 18:49:27,901: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:49:28,128: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:49:28,226: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:49:28,352: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:49:29,228: INFO/job_scheduler] Job 231 finished 1 steps successfully
[2017-06-01 18:49:29,289: INFO/job_scheduler] Job 231: complete_job: Updating cache
[2017-06-01 18:49:29,289: INFO/job_scheduler] Job 231 completing (errored=False, cancelled=False)
[2017-06-01 18:49:29,345: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:49:29,453: INFO/job_scheduler] Job 232 finished 1 steps successfully
[2017-06-01 18:49:29,477: INFO/job_scheduler] Job 232: complete_job: Updating cache
[2017-06-01 18:49:29,478: INFO/job_scheduler] Job 232 completing (errored=False, cancelled=False)
[2017-06-01 18:49:29,519: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:49:29,573: INFO/job_scheduler] Job 234 finished 1 steps successfully
[2017-06-01 18:49:29,686: INFO/job_scheduler] Job 234: complete_job: Updating cache
[2017-06-01 18:49:29,686: INFO/job_scheduler] Job 234 completing (errored=False, cancelled=False)
[2017-06-01 18:49:29,729: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:49:29,742: INFO/job_scheduler] Job 233 finished 1 steps successfully
[2017-06-01 18:49:29,762: INFO/job_scheduler] Job 233: complete_job: Updating cache
[2017-06-01 18:49:29,762: INFO/job_scheduler] Job 233 completing (errored=False, cancelled=False)
[2017-06-01 18:49:29,819: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:49:31,710: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:31,711: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:31,719: INFO/command_plan] add_jobs: created Job 235 (Test for host connectivity)
[2017-06-01 18:49:31,740: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:49:31,752: INFO/job_scheduler] Job 235: RunJobThread.run
[2017-06-01 18:49:31,796: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:31,797: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:31,818: INFO/command_plan] add_jobs: created Job 236 (Test for host connectivity)
[2017-06-01 18:49:31,850: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:49:31,881: INFO/job_scheduler] Job 236: RunJobThread.run
[2017-06-01 18:49:31,929: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:31,938: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:31,968: INFO/command_plan] add_jobs: created Job 237 (Test for host connectivity)
[2017-06-01 18:49:31,995: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:49:31,997: INFO/job_scheduler] Job 237: RunJobThread.run
[2017-06-01 18:49:32,111: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:32,111: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:32,131: INFO/command_plan] add_jobs: created Job 238 (Test for host connectivity)
[2017-06-01 18:49:32,233: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:49:32,236: INFO/job_scheduler] Job 238: RunJobThread.run
[2017-06-01 18:49:34,044: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:49:34,171: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:49:34,338: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:49:34,632: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:49:35,425: INFO/job_scheduler] Job 235 finished 1 steps successfully
[2017-06-01 18:49:35,486: INFO/job_scheduler] Job 235: complete_job: Updating cache
[2017-06-01 18:49:35,486: INFO/job_scheduler] Job 235 completing (errored=False, cancelled=False)
[2017-06-01 18:49:35,527: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:49:35,613: INFO/job_scheduler] Job 236 finished 1 steps successfully
[2017-06-01 18:49:35,668: INFO/job_scheduler] Job 236: complete_job: Updating cache
[2017-06-01 18:49:35,668: INFO/job_scheduler] Job 236 completing (errored=False, cancelled=False)
[2017-06-01 18:49:35,715: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:49:35,807: INFO/job_scheduler] Job 237 finished 1 steps successfully
[2017-06-01 18:49:35,851: INFO/job_scheduler] Job 237: complete_job: Updating cache
[2017-06-01 18:49:35,851: INFO/job_scheduler] Job 237 completing (errored=False, cancelled=False)
[2017-06-01 18:49:35,893: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:49:36,096: INFO/job_scheduler] Job 238 finished 1 steps successfully
[2017-06-01 18:49:36,126: INFO/job_scheduler] Job 238: complete_job: Updating cache
[2017-06-01 18:49:36,126: INFO/job_scheduler] Job 238 completing (errored=False, cancelled=False)
[2017-06-01 18:49:36,161: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:49:37,835: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:37,835: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:37,851: INFO/command_plan] add_jobs: created Job 239 (Test for host connectivity)
[2017-06-01 18:49:37,891: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:37,892: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:37,918: INFO/command_plan] add_jobs: created Job 240 (Test for host connectivity)
[2017-06-01 18:49:37,929: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:49:37,939: INFO/job_scheduler] Job 240: RunJobThread.run
[2017-06-01 18:49:37,939: INFO/job_scheduler] Job 239: RunJobThread.run
[2017-06-01 18:49:37,944: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:49:38,049: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:38,051: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:38,083: INFO/command_plan] add_jobs: created Job 241 (Test for host connectivity)
[2017-06-01 18:49:38,103: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:49:38,105: INFO/job_scheduler] Job 241: RunJobThread.run
[2017-06-01 18:49:38,168: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:38,168: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:38,284: INFO/command_plan] add_jobs: created Job 242 (Test for host connectivity)
[2017-06-01 18:49:38,331: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:49:38,345: INFO/job_scheduler] Job 242: RunJobThread.run
[2017-06-01 18:49:40,164: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:49:40,341: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:49:40,430: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:49:40,593: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:49:41,474: INFO/job_scheduler] Job 239 finished 1 steps successfully
[2017-06-01 18:49:41,525: INFO/job_scheduler] Job 239: complete_job: Updating cache
[2017-06-01 18:49:41,525: INFO/job_scheduler] Job 239 completing (errored=False, cancelled=False)
[2017-06-01 18:49:41,588: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:49:41,663: INFO/job_scheduler] Job 240 finished 1 steps successfully
[2017-06-01 18:49:41,792: INFO/job_scheduler] Job 240: complete_job: Updating cache
[2017-06-01 18:49:41,792: INFO/job_scheduler] Job 240 completing (errored=False, cancelled=False)
[2017-06-01 18:49:41,831: INFO/job_scheduler] Job 241 finished 1 steps successfully
[2017-06-01 18:49:41,858: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:49:41,963: INFO/job_scheduler] Job 242 finished 1 steps successfully
[2017-06-01 18:49:41,983: INFO/job_scheduler] Job 241: complete_job: Updating cache
[2017-06-01 18:49:41,983: INFO/job_scheduler] Job 241 completing (errored=False, cancelled=False)
[2017-06-01 18:49:42,025: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:49:42,034: INFO/job_scheduler] Job 242: complete_job: Updating cache
[2017-06-01 18:49:42,034: INFO/job_scheduler] Job 242 completing (errored=False, cancelled=False)
[2017-06-01 18:49:42,088: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:49:43,911: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:43,912: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:43,924: INFO/command_plan] add_jobs: created Job 243 (Test for host connectivity)
[2017-06-01 18:49:43,961: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:43,961: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:43,984: INFO/command_plan] add_jobs: created Job 244 (Test for host connectivity)
[2017-06-01 18:49:43,992: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:49:44,008: INFO/job_scheduler] Job 243: RunJobThread.run
[2017-06-01 18:49:44,020: INFO/job_scheduler] Job 244: RunJobThread.run
[2017-06-01 18:49:44,028: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:49:44,116: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:44,116: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:44,168: INFO/command_plan] add_jobs: created Job 245 (Test for host connectivity)
[2017-06-01 18:49:44,201: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:49:44,211: INFO/job_scheduler] Job 245: RunJobThread.run
[2017-06-01 18:49:44,259: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:44,260: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:44,283: INFO/command_plan] add_jobs: created Job 246 (Test for host connectivity)
[2017-06-01 18:49:44,371: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:49:44,389: INFO/job_scheduler] Job 246: RunJobThread.run
[2017-06-01 18:49:46,199: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:49:46,295: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:49:46,410: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:49:46,517: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:49:47,509: INFO/job_scheduler] Job 243 finished 1 steps successfully
[2017-06-01 18:49:47,576: INFO/job_scheduler] Job 243: complete_job: Updating cache
[2017-06-01 18:49:47,576: INFO/job_scheduler] Job 243 completing (errored=False, cancelled=False)
[2017-06-01 18:49:47,626: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:49:47,672: INFO/job_scheduler] Job 244 finished 1 steps successfully
[2017-06-01 18:49:47,683: INFO/job_scheduler] Job 244: complete_job: Updating cache
[2017-06-01 18:49:47,683: INFO/job_scheduler] Job 244 completing (errored=False, cancelled=False)
[2017-06-01 18:49:47,733: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:49:47,798: INFO/job_scheduler] Job 245 finished 1 steps successfully
[2017-06-01 18:49:47,815: INFO/job_scheduler] Job 245: complete_job: Updating cache
[2017-06-01 18:49:47,815: INFO/job_scheduler] Job 245 completing (errored=False, cancelled=False)
[2017-06-01 18:49:47,857: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:49:47,865: INFO/job_scheduler] Job 246 finished 1 steps successfully
[2017-06-01 18:49:47,891: INFO/job_scheduler] Job 246: complete_job: Updating cache
[2017-06-01 18:49:47,891: INFO/job_scheduler] Job 246 completing (errored=False, cancelled=False)
[2017-06-01 18:49:47,941: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:49:49,935: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:49,935: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:49,956: INFO/command_plan] add_jobs: created Job 247 (Test for host connectivity)
[2017-06-01 18:49:49,977: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:49:49,997: INFO/job_scheduler] Job 247: RunJobThread.run
[2017-06-01 18:49:50,161: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:50,164: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:50,190: INFO/command_plan] add_jobs: created Job 248 (Test for host connectivity)
[2017-06-01 18:49:50,215: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:49:50,244: INFO/job_scheduler] Job 248: RunJobThread.run
[2017-06-01 18:49:50,301: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:50,301: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:50,323: INFO/command_plan] add_jobs: created Job 249 (Test for host connectivity)
[2017-06-01 18:49:50,340: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:49:50,342: INFO/job_scheduler] Job 249: RunJobThread.run
[2017-06-01 18:49:50,408: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:50,409: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:50,426: INFO/command_plan] add_jobs: created Job 250 (Test for host connectivity)
[2017-06-01 18:49:50,461: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:49:50,471: INFO/job_scheduler] Job 250: RunJobThread.run
[2017-06-01 18:49:52,257: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:49:52,341: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:49:52,697: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:49:52,787: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:49:53,562: INFO/job_scheduler] Job 247 finished 1 steps successfully
[2017-06-01 18:49:53,581: INFO/job_scheduler] Job 247: complete_job: Updating cache
[2017-06-01 18:49:53,581: INFO/job_scheduler] Job 247 completing (errored=False, cancelled=False)
[2017-06-01 18:49:53,655: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:49:53,724: INFO/job_scheduler] Job 248 finished 1 steps successfully
[2017-06-01 18:49:53,750: INFO/job_scheduler] Job 248: complete_job: Updating cache
[2017-06-01 18:49:53,750: INFO/job_scheduler] Job 248 completing (errored=False, cancelled=False)
[2017-06-01 18:49:53,799: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:49:54,104: INFO/job_scheduler] Job 249 finished 1 steps successfully
[2017-06-01 18:49:54,158: INFO/job_scheduler] Job 249: complete_job: Updating cache
[2017-06-01 18:49:54,158: INFO/job_scheduler] Job 249 completing (errored=False, cancelled=False)
[2017-06-01 18:49:54,198: INFO/job_scheduler] Job 250 finished 1 steps successfully
[2017-06-01 18:49:54,208: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:49:54,225: INFO/job_scheduler] Job 250: complete_job: Updating cache
[2017-06-01 18:49:54,225: INFO/job_scheduler] Job 250 completing (errored=False, cancelled=False)
[2017-06-01 18:49:54,275: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:49:56,082: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:56,082: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:56,099: INFO/command_plan] add_jobs: created Job 251 (Test for host connectivity)
[2017-06-01 18:49:56,151: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:56,152: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:56,167: INFO/command_plan] add_jobs: created Job 252 (Test for host connectivity)
[2017-06-01 18:49:56,176: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:49:56,225: INFO/job_scheduler] Job 251: RunJobThread.run
[2017-06-01 18:49:56,235: INFO/job_scheduler] Job 252: RunJobThread.run
[2017-06-01 18:49:56,265: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:56,265: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:56,284: INFO/command_plan] add_jobs: created Job 253 (Test for host connectivity)
[2017-06-01 18:49:56,321: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:49:56,323: INFO/job_scheduler] Job 253: RunJobThread.run
[2017-06-01 18:49:56,457: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:49:56,464: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:49:56,491: INFO/command_plan] add_jobs: created Job 254 (Test for host connectivity)
[2017-06-01 18:49:56,528: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:49:56,542: INFO/job_scheduler] Job 254: RunJobThread.run
[2017-06-01 18:49:56,675: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 18:49:58,455: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:49:58,571: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:49:58,605: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:49:58,705: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:49:59,836: INFO/job_scheduler] Job 251 finished 1 steps successfully
[2017-06-01 18:49:59,883: INFO/job_scheduler] Job 251: complete_job: Updating cache
[2017-06-01 18:49:59,883: INFO/job_scheduler] Job 251 completing (errored=False, cancelled=False)
[2017-06-01 18:49:59,935: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:49:59,950: INFO/job_scheduler] Job 253 finished 1 steps successfully
[2017-06-01 18:49:59,966: INFO/job_scheduler] Job 253: complete_job: Updating cache
[2017-06-01 18:49:59,966: INFO/job_scheduler] Job 253 completing (errored=False, cancelled=False)
[2017-06-01 18:50:00,016: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:50:00,060: INFO/job_scheduler] Job 254 finished 1 steps successfully
[2017-06-01 18:50:00,077: INFO/job_scheduler] Job 252 finished 1 steps successfully
[2017-06-01 18:50:00,108: INFO/job_scheduler] Job 254: complete_job: Updating cache
[2017-06-01 18:50:00,108: INFO/job_scheduler] Job 254 completing (errored=False, cancelled=False)
[2017-06-01 18:50:00,169: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:50:00,183: INFO/job_scheduler] Job 252: complete_job: Updating cache
[2017-06-01 18:50:00,183: INFO/job_scheduler] Job 252 completing (errored=False, cancelled=False)
[2017-06-01 18:50:00,227: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:50:02,181: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:02,187: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:02,204: INFO/command_plan] add_jobs: created Job 255 (Test for host connectivity)
[2017-06-01 18:50:02,254: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:50:02,261: INFO/job_scheduler] Job 255: RunJobThread.run
[2017-06-01 18:50:02,307: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:02,308: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:02,332: INFO/command_plan] add_jobs: created Job 256 (Test for host connectivity)
[2017-06-01 18:50:02,370: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:50:02,391: INFO/job_scheduler] Job 256: RunJobThread.run
[2017-06-01 18:50:02,455: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:02,455: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:02,480: INFO/command_plan] add_jobs: created Job 257 (Test for host connectivity)
[2017-06-01 18:50:02,658: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:02,659: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:02,697: INFO/command_plan] add_jobs: created Job 258 (Test for host connectivity)
[2017-06-01 18:50:02,713: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 18:50:02,724: INFO/job_scheduler] Job 257: RunJobThread.run
[2017-06-01 18:50:02,725: INFO/job_scheduler] Job 258: RunJobThread.run
[2017-06-01 18:50:02,739: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 18:50:04,535: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:50:04,715: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:50:04,981: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:50:05,185: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:50:05,921: INFO/job_scheduler] Job 255 finished 1 steps successfully
[2017-06-01 18:50:05,950: INFO/job_scheduler] Job 255: complete_job: Updating cache
[2017-06-01 18:50:05,951: INFO/job_scheduler] Job 255 completing (errored=False, cancelled=False)
[2017-06-01 18:50:06,017: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:50:06,035: INFO/job_scheduler] Job 256 finished 1 steps successfully
[2017-06-01 18:50:06,063: INFO/job_scheduler] Job 256: complete_job: Updating cache
[2017-06-01 18:50:06,063: INFO/job_scheduler] Job 256 completing (errored=False, cancelled=False)
[2017-06-01 18:50:06,129: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:50:06,353: INFO/job_scheduler] Job 257 finished 1 steps successfully
[2017-06-01 18:50:06,379: INFO/job_scheduler] Job 257: complete_job: Updating cache
[2017-06-01 18:50:06,379: INFO/job_scheduler] Job 257 completing (errored=False, cancelled=False)
[2017-06-01 18:50:06,420: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:50:06,600: INFO/job_scheduler] Job 258 finished 1 steps successfully
[2017-06-01 18:50:06,664: INFO/job_scheduler] Job 258: complete_job: Updating cache
[2017-06-01 18:50:06,665: INFO/job_scheduler] Job 258 completing (errored=False, cancelled=False)
[2017-06-01 18:50:06,707: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:50:08,372: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:08,372: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:08,397: INFO/command_plan] add_jobs: created Job 259 (Test for host connectivity)
[2017-06-01 18:50:08,426: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:50:08,430: INFO/job_scheduler] Job 259: RunJobThread.run
[2017-06-01 18:50:08,458: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:08,458: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:08,514: INFO/command_plan] add_jobs: created Job 260 (Test for host connectivity)
[2017-06-01 18:50:08,569: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:08,569: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:08,623: INFO/command_plan] add_jobs: created Job 261 (Test for host connectivity)
[2017-06-01 18:50:08,643: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 18:50:08,657: INFO/job_scheduler] Job 260: RunJobThread.run
[2017-06-01 18:50:08,658: INFO/job_scheduler] Job 261: RunJobThread.run
[2017-06-01 18:50:08,675: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:50:08,707: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:08,708: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:08,732: INFO/command_plan] add_jobs: created Job 262 (Test for host connectivity)
[2017-06-01 18:50:08,758: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:50:08,769: INFO/job_scheduler] Job 262: RunJobThread.run
[2017-06-01 18:50:10,737: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:50:10,976: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:50:11,035: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:50:11,076: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:50:12,092: INFO/job_scheduler] Job 259 finished 1 steps successfully
[2017-06-01 18:50:12,184: INFO/job_scheduler] Job 259: complete_job: Updating cache
[2017-06-01 18:50:12,185: INFO/job_scheduler] Job 259 completing (errored=False, cancelled=False)
[2017-06-01 18:50:12,256: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:50:12,450: INFO/job_scheduler] Job 261 finished 1 steps successfully
[2017-06-01 18:50:12,465: INFO/job_scheduler] Job 262 finished 1 steps successfully
[2017-06-01 18:50:12,497: INFO/job_scheduler] Job 261: complete_job: Updating cache
[2017-06-01 18:50:12,497: INFO/job_scheduler] Job 261 completing (errored=False, cancelled=False)
[2017-06-01 18:50:12,540: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:50:12,549: INFO/job_scheduler] Job 260 finished 1 steps successfully
[2017-06-01 18:50:12,572: INFO/job_scheduler] Job 262: complete_job: Updating cache
[2017-06-01 18:50:12,572: INFO/job_scheduler] Job 262 completing (errored=False, cancelled=False)
[2017-06-01 18:50:12,632: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:50:12,648: INFO/job_scheduler] Job 260: complete_job: Updating cache
[2017-06-01 18:50:12,649: INFO/job_scheduler] Job 260 completing (errored=False, cancelled=False)
[2017-06-01 18:50:12,710: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:50:13,904: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:13,905: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:13,930: INFO/command_plan] add_jobs: created Job 263 (Test for host connectivity)
[2017-06-01 18:50:13,976: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:13,976: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:13,998: INFO/command_plan] add_jobs: created Job 264 (Test for host connectivity)
[2017-06-01 18:50:14,010: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:50:14,027: INFO/job_scheduler] Job 264: RunJobThread.run
[2017-06-01 18:50:14,028: INFO/job_scheduler] Job 263: RunJobThread.run
[2017-06-01 18:50:14,046: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:50:14,102: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:14,102: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:14,167: INFO/command_plan] add_jobs: created Job 265 (Test for host connectivity)
[2017-06-01 18:50:14,224: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:50:14,259: INFO/job_scheduler] Job 265: RunJobThread.run
[2017-06-01 18:50:14,318: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:14,320: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:14,347: INFO/command_plan] add_jobs: created Job 266 (Test for host connectivity)
[2017-06-01 18:50:14,372: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:50:14,393: INFO/job_scheduler] Job 266: RunJobThread.run
[2017-06-01 18:50:16,286: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:50:16,431: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:50:16,532: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:50:16,594: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:50:17,627: INFO/job_scheduler] Job 264 finished 1 steps successfully
[2017-06-01 18:50:17,696: INFO/job_scheduler] Job 264: complete_job: Updating cache
[2017-06-01 18:50:17,696: INFO/job_scheduler] Job 264 completing (errored=False, cancelled=False)
[2017-06-01 18:50:17,757: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:50:17,855: INFO/job_scheduler] Job 265 finished 1 steps successfully
[2017-06-01 18:50:17,870: INFO/job_scheduler] Job 263 finished 1 steps successfully
[2017-06-01 18:50:17,888: INFO/job_scheduler] Job 265: complete_job: Updating cache
[2017-06-01 18:50:17,888: INFO/job_scheduler] Job 265 completing (errored=False, cancelled=False)
[2017-06-01 18:50:17,930: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:50:17,947: INFO/job_scheduler] Job 263: complete_job: Updating cache
[2017-06-01 18:50:17,948: INFO/job_scheduler] Job 263 completing (errored=False, cancelled=False)
[2017-06-01 18:50:17,981: INFO/job_scheduler] Job 266 finished 1 steps successfully
[2017-06-01 18:50:18,015: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:50:18,030: INFO/job_scheduler] Job 266: complete_job: Updating cache
[2017-06-01 18:50:18,030: INFO/job_scheduler] Job 266 completing (errored=False, cancelled=False)
[2017-06-01 18:50:18,088: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:50:19,519: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:19,521: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:19,537: INFO/command_plan] add_jobs: created Job 267 (Test for host connectivity)
[2017-06-01 18:50:19,558: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:50:19,560: INFO/job_scheduler] Job 267: RunJobThread.run
[2017-06-01 18:50:19,604: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:19,604: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:19,621: INFO/command_plan] add_jobs: created Job 268 (Test for host connectivity)
[2017-06-01 18:50:19,644: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:50:19,653: INFO/job_scheduler] Job 268: RunJobThread.run
[2017-06-01 18:50:19,760: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:19,760: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:19,779: INFO/command_plan] add_jobs: created Job 269 (Test for host connectivity)
[2017-06-01 18:50:19,852: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:50:19,866: INFO/job_scheduler] Job 269: RunJobThread.run
[2017-06-01 18:50:19,917: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:19,918: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:19,937: INFO/command_plan] add_jobs: created Job 270 (Test for host connectivity)
[2017-06-01 18:50:19,949: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:50:19,973: INFO/job_scheduler] Job 270: RunJobThread.run
[2017-06-01 18:50:21,761: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:50:21,912: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:50:22,064: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:50:22,281: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:50:23,157: INFO/job_scheduler] Job 267 finished 1 steps successfully
[2017-06-01 18:50:23,195: INFO/job_scheduler] Job 267: complete_job: Updating cache
[2017-06-01 18:50:23,195: INFO/job_scheduler] Job 267 completing (errored=False, cancelled=False)
[2017-06-01 18:50:23,237: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:50:23,262: INFO/job_scheduler] Job 268 finished 1 steps successfully
[2017-06-01 18:50:23,281: INFO/job_scheduler] Job 268: complete_job: Updating cache
[2017-06-01 18:50:23,281: INFO/job_scheduler] Job 268 completing (errored=False, cancelled=False)
[2017-06-01 18:50:23,439: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:50:23,444: INFO/job_scheduler] Job 269 finished 1 steps successfully
[2017-06-01 18:50:23,454: INFO/job_scheduler] Job 269: complete_job: Updating cache
[2017-06-01 18:50:23,455: INFO/job_scheduler] Job 269 completing (errored=False, cancelled=False)
[2017-06-01 18:50:23,502: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:50:23,732: INFO/job_scheduler] Job 270 finished 1 steps successfully
[2017-06-01 18:50:23,795: INFO/job_scheduler] Job 270: complete_job: Updating cache
[2017-06-01 18:50:23,795: INFO/job_scheduler] Job 270 completing (errored=False, cancelled=False)
[2017-06-01 18:50:23,838: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:50:25,633: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:25,634: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:25,655: INFO/command_plan] add_jobs: created Job 271 (Test for host connectivity)
[2017-06-01 18:50:25,697: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:25,697: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:25,711: INFO/command_plan] add_jobs: created Job 272 (Test for host connectivity)
[2017-06-01 18:50:25,720: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:50:25,735: INFO/job_scheduler] Job 272: RunJobThread.run
[2017-06-01 18:50:25,764: INFO/job_scheduler] Job 271: RunJobThread.run
[2017-06-01 18:50:25,771: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:50:25,807: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:25,808: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:25,828: INFO/command_plan] add_jobs: created Job 273 (Test for host connectivity)
[2017-06-01 18:50:25,877: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:50:25,888: INFO/job_scheduler] Job 273: RunJobThread.run
[2017-06-01 18:50:25,915: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:25,916: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:25,979: INFO/command_plan] add_jobs: created Job 274 (Test for host connectivity)
[2017-06-01 18:50:26,030: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:50:26,032: INFO/job_scheduler] Job 274: RunJobThread.run
[2017-06-01 18:50:27,978: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:50:28,114: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:50:28,205: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:50:28,214: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:50:29,336: INFO/job_scheduler] Job 271 finished 1 steps successfully
[2017-06-01 18:50:29,394: INFO/job_scheduler] Job 271: complete_job: Updating cache
[2017-06-01 18:50:29,394: INFO/job_scheduler] Job 271 completing (errored=False, cancelled=False)
[2017-06-01 18:50:29,460: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:50:29,502: INFO/job_scheduler] Job 272 finished 1 steps successfully
[2017-06-01 18:50:29,536: INFO/job_scheduler] Job 272: complete_job: Updating cache
[2017-06-01 18:50:29,536: INFO/job_scheduler] Job 272 completing (errored=False, cancelled=False)
[2017-06-01 18:50:29,578: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:50:29,592: INFO/job_scheduler] Job 274 finished 1 steps successfully
[2017-06-01 18:50:29,611: INFO/job_scheduler] Job 274: complete_job: Updating cache
[2017-06-01 18:50:29,611: INFO/job_scheduler] Job 274 completing (errored=False, cancelled=False)
[2017-06-01 18:50:29,650: INFO/job_scheduler] Job 273 finished 1 steps successfully
[2017-06-01 18:50:29,674: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:50:29,686: INFO/job_scheduler] Job 273: complete_job: Updating cache
[2017-06-01 18:50:29,686: INFO/job_scheduler] Job 273 completing (errored=False, cancelled=False)
[2017-06-01 18:50:29,761: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:50:31,721: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:31,722: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:31,743: INFO/command_plan] add_jobs: created Job 275 (Test for host connectivity)
[2017-06-01 18:50:31,763: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:50:31,769: INFO/job_scheduler] Job 275: RunJobThread.run
[2017-06-01 18:50:31,800: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:31,800: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:31,817: INFO/command_plan] add_jobs: created Job 276 (Test for host connectivity)
[2017-06-01 18:50:31,839: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:50:31,866: INFO/job_scheduler] Job 276: RunJobThread.run
[2017-06-01 18:50:31,964: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:31,964: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:31,985: INFO/command_plan] add_jobs: created Job 277 (Test for host connectivity)
[2017-06-01 18:50:32,040: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:50:32,051: INFO/job_scheduler] Job 277: RunJobThread.run
[2017-06-01 18:50:32,108: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:32,108: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:32,135: INFO/command_plan] add_jobs: created Job 278 (Test for host connectivity)
[2017-06-01 18:50:32,155: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:50:32,165: INFO/job_scheduler] Job 278: RunJobThread.run
[2017-06-01 18:50:34,055: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:50:34,175: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:50:34,367: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:50:34,497: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:50:35,414: INFO/job_scheduler] Job 275 finished 1 steps successfully
[2017-06-01 18:50:35,479: INFO/job_scheduler] Job 275: complete_job: Updating cache
[2017-06-01 18:50:35,480: INFO/job_scheduler] Job 275 completing (errored=False, cancelled=False)
[2017-06-01 18:50:35,546: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:50:35,585: INFO/job_scheduler] Job 276 finished 1 steps successfully
[2017-06-01 18:50:35,609: INFO/job_scheduler] Job 276: complete_job: Updating cache
[2017-06-01 18:50:35,609: INFO/job_scheduler] Job 276 completing (errored=False, cancelled=False)
[2017-06-01 18:50:35,656: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:50:35,754: INFO/job_scheduler] Job 277 finished 1 steps successfully
[2017-06-01 18:50:35,840: INFO/job_scheduler] Job 278 finished 1 steps successfully
[2017-06-01 18:50:35,867: INFO/job_scheduler] Job 277: complete_job: Updating cache
[2017-06-01 18:50:35,868: INFO/job_scheduler] Job 277 completing (errored=False, cancelled=False)
[2017-06-01 18:50:35,926: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:50:35,935: INFO/job_scheduler] Job 278: complete_job: Updating cache
[2017-06-01 18:50:35,935: INFO/job_scheduler] Job 278 completing (errored=False, cancelled=False)
[2017-06-01 18:50:35,985: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:50:37,810: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:37,812: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:37,834: INFO/command_plan] add_jobs: created Job 279 (Test for host connectivity)
[2017-06-01 18:50:37,856: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:50:37,867: INFO/job_scheduler] Job 279: RunJobThread.run
[2017-06-01 18:50:37,895: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:37,896: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:37,917: INFO/command_plan] add_jobs: created Job 280 (Test for host connectivity)
[2017-06-01 18:50:37,962: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:50:37,974: INFO/job_scheduler] Job 280: RunJobThread.run
[2017-06-01 18:50:38,040: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:38,042: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:38,067: INFO/command_plan] add_jobs: created Job 281 (Test for host connectivity)
[2017-06-01 18:50:38,149: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:38,150: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:38,160: INFO/command_plan] add_jobs: created Job 282 (Test for host connectivity)
[2017-06-01 18:50:38,177: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 18:50:38,213: INFO/job_scheduler] Job 281: RunJobThread.run
[2017-06-01 18:50:38,219: INFO/job_scheduler] Job 282: RunJobThread.run
[2017-06-01 18:50:38,230: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 18:50:40,125: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:50:40,271: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:50:40,635: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:50:40,640: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:50:41,480: INFO/job_scheduler] Job 279 finished 1 steps successfully
[2017-06-01 18:50:41,516: INFO/job_scheduler] Job 279: complete_job: Updating cache
[2017-06-01 18:50:41,517: INFO/job_scheduler] Job 279 completing (errored=False, cancelled=False)
[2017-06-01 18:50:41,558: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:50:41,619: INFO/job_scheduler] Job 280 finished 1 steps successfully
[2017-06-01 18:50:41,641: INFO/job_scheduler] Job 280: complete_job: Updating cache
[2017-06-01 18:50:41,641: INFO/job_scheduler] Job 280 completing (errored=False, cancelled=False)
[2017-06-01 18:50:41,693: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:50:42,077: INFO/job_scheduler] Job 282 finished 1 steps successfully
[2017-06-01 18:50:42,079: INFO/job_scheduler] Job 281 finished 1 steps successfully
[2017-06-01 18:50:42,133: INFO/job_scheduler] Job 282: complete_job: Updating cache
[2017-06-01 18:50:42,133: INFO/job_scheduler] Job 282 completing (errored=False, cancelled=False)
[2017-06-01 18:50:42,184: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:50:42,199: INFO/job_scheduler] Job 281: complete_job: Updating cache
[2017-06-01 18:50:42,200: INFO/job_scheduler] Job 281 completing (errored=False, cancelled=False)
[2017-06-01 18:50:42,251: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:50:43,851: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:43,851: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:43,874: INFO/command_plan] add_jobs: created Job 283 (Test for host connectivity)
[2017-06-01 18:50:43,921: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:50:43,939: INFO/job_scheduler] Job 283: RunJobThread.run
[2017-06-01 18:50:43,965: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:43,965: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:43,999: INFO/command_plan] add_jobs: created Job 284 (Test for host connectivity)
[2017-06-01 18:50:44,045: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:50:44,048: INFO/job_scheduler] Job 284: RunJobThread.run
[2017-06-01 18:50:44,096: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:44,096: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:44,124: INFO/command_plan] add_jobs: created Job 285 (Test for host connectivity)
[2017-06-01 18:50:44,167: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:50:44,182: INFO/job_scheduler] Job 285: RunJobThread.run
[2017-06-01 18:50:44,266: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:44,266: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:44,289: INFO/command_plan] add_jobs: created Job 286 (Test for host connectivity)
[2017-06-01 18:50:44,359: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:50:44,364: INFO/job_scheduler] Job 286: RunJobThread.run
[2017-06-01 18:50:46,236: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:50:46,351: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:50:46,525: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:50:46,608: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:50:47,556: INFO/job_scheduler] Job 283 finished 1 steps successfully
[2017-06-01 18:50:47,621: INFO/job_scheduler] Job 283: complete_job: Updating cache
[2017-06-01 18:50:47,622: INFO/job_scheduler] Job 283 completing (errored=False, cancelled=False)
[2017-06-01 18:50:47,757: INFO/job_scheduler] Job 284 finished 1 steps successfully
[2017-06-01 18:50:47,800: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:50:47,832: INFO/job_scheduler] Job 284: complete_job: Updating cache
[2017-06-01 18:50:47,833: INFO/job_scheduler] Job 284 completing (errored=False, cancelled=False)
[2017-06-01 18:50:47,907: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:50:47,947: INFO/job_scheduler] Job 285 finished 1 steps successfully
[2017-06-01 18:50:47,975: INFO/job_scheduler] Job 285: complete_job: Updating cache
[2017-06-01 18:50:47,975: INFO/job_scheduler] Job 285 completing (errored=False, cancelled=False)
[2017-06-01 18:50:48,009: INFO/job_scheduler] Job 286 finished 1 steps successfully
[2017-06-01 18:50:48,043: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:50:48,086: INFO/job_scheduler] Job 286: complete_job: Updating cache
[2017-06-01 18:50:48,086: INFO/job_scheduler] Job 286 completing (errored=False, cancelled=False)
[2017-06-01 18:50:48,149: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:50:49,989: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:49,990: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:50,006: INFO/command_plan] add_jobs: created Job 287 (Test for host connectivity)
[2017-06-01 18:50:50,020: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:50:50,040: INFO/job_scheduler] Job 287: RunJobThread.run
[2017-06-01 18:50:50,090: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:50,092: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:50,110: INFO/command_plan] add_jobs: created Job 288 (Test for host connectivity)
[2017-06-01 18:50:50,135: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:50:50,146: INFO/job_scheduler] Job 288: RunJobThread.run
[2017-06-01 18:50:50,244: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:50,244: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:50,273: INFO/command_plan] add_jobs: created Job 289 (Test for host connectivity)
[2017-06-01 18:50:50,320: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:50:50,322: INFO/job_scheduler] Job 289: RunJobThread.run
[2017-06-01 18:50:50,373: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:50,373: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:50,400: INFO/command_plan] add_jobs: created Job 290 (Test for host connectivity)
[2017-06-01 18:50:50,430: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:50:50,443: INFO/job_scheduler] Job 290: RunJobThread.run
[2017-06-01 18:50:52,293: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:50:52,392: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:50:52,750: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:50:52,758: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:50:53,633: INFO/job_scheduler] Job 287 finished 1 steps successfully
[2017-06-01 18:50:53,711: INFO/job_scheduler] Job 287: complete_job: Updating cache
[2017-06-01 18:50:53,711: INFO/job_scheduler] Job 287 completing (errored=False, cancelled=False)
[2017-06-01 18:50:53,773: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:50:53,805: INFO/job_scheduler] Job 288 finished 1 steps successfully
[2017-06-01 18:50:53,833: INFO/job_scheduler] Job 288: complete_job: Updating cache
[2017-06-01 18:50:53,833: INFO/job_scheduler] Job 288 completing (errored=False, cancelled=False)
[2017-06-01 18:50:53,925: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:50:54,078: INFO/job_scheduler] Job 290 finished 1 steps successfully
[2017-06-01 18:50:54,106: INFO/job_scheduler] Job 290: complete_job: Updating cache
[2017-06-01 18:50:54,106: INFO/job_scheduler] Job 290 completing (errored=False, cancelled=False)
[2017-06-01 18:50:54,157: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:50:54,195: INFO/job_scheduler] Job 289 finished 1 steps successfully
[2017-06-01 18:50:54,214: INFO/job_scheduler] Job 289: complete_job: Updating cache
[2017-06-01 18:50:54,214: INFO/job_scheduler] Job 289 completing (errored=False, cancelled=False)
[2017-06-01 18:50:54,284: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:50:56,180: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:56,182: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:56,265: INFO/command_plan] add_jobs: created Job 291 (Test for host connectivity)
[2017-06-01 18:50:56,288: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:50:56,306: INFO/job_scheduler] Job 291: RunJobThread.run
[2017-06-01 18:50:56,334: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:56,335: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:56,357: INFO/command_plan] add_jobs: created Job 292 (Test for host connectivity)
[2017-06-01 18:50:56,401: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:50:56,406: INFO/job_scheduler] Job 292: RunJobThread.run
[2017-06-01 18:50:56,482: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:56,485: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:56,533: INFO/command_plan] add_jobs: created Job 293 (Test for host connectivity)
[2017-06-01 18:50:56,607: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:50:56,617: INFO/job_scheduler] Job 293: RunJobThread.run
[2017-06-01 18:50:56,658: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:50:56,658: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:50:56,683: INFO/command_plan] add_jobs: created Job 294 (Test for host connectivity)
[2017-06-01 18:50:56,707: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:50:56,738: INFO/job_scheduler] Job 294: RunJobThread.run
[2017-06-01 18:50:58,659: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:50:58,739: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:50:58,860: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:50:59,060: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:51:00,037: INFO/job_scheduler] Job 292 finished 1 steps successfully
[2017-06-01 18:51:00,065: INFO/job_scheduler] Job 292: complete_job: Updating cache
[2017-06-01 18:51:00,065: INFO/job_scheduler] Job 292 completing (errored=False, cancelled=False)
[2017-06-01 18:51:00,110: INFO/job_scheduler] Job 291 finished 1 steps successfully
[2017-06-01 18:51:00,144: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:51:00,162: INFO/job_scheduler] Job 291: complete_job: Updating cache
[2017-06-01 18:51:00,162: INFO/job_scheduler] Job 291 completing (errored=False, cancelled=False)
[2017-06-01 18:51:00,223: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:51:00,325: INFO/job_scheduler] Job 293 finished 1 steps successfully
[2017-06-01 18:51:00,407: INFO/job_scheduler] Job 293: complete_job: Updating cache
[2017-06-01 18:51:00,407: INFO/job_scheduler] Job 293 completing (errored=False, cancelled=False)
[2017-06-01 18:51:00,444: INFO/job_scheduler] Job 294 finished 1 steps successfully
[2017-06-01 18:51:00,457: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:51:00,465: INFO/job_scheduler] Job 294: complete_job: Updating cache
[2017-06-01 18:51:00,465: INFO/job_scheduler] Job 294 completing (errored=False, cancelled=False)
[2017-06-01 18:51:00,492: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:51:02,416: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:02,417: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:02,432: INFO/command_plan] add_jobs: created Job 295 (Test for host connectivity)
[2017-06-01 18:51:02,481: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:02,481: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:02,507: INFO/command_plan] add_jobs: created Job 296 (Test for host connectivity)
[2017-06-01 18:51:02,525: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:51:02,543: INFO/job_scheduler] Job 296: RunJobThread.run
[2017-06-01 18:51:02,549: INFO/job_scheduler] Job 295: RunJobThread.run
[2017-06-01 18:51:02,557: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:51:02,607: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:02,608: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:02,648: INFO/command_plan] add_jobs: created Job 297 (Test for host connectivity)
[2017-06-01 18:51:02,738: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:51:02,744: INFO/job_scheduler] Job 297: RunJobThread.run
[2017-06-01 18:51:02,800: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:02,803: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:02,823: INFO/command_plan] add_jobs: created Job 298 (Test for host connectivity)
[2017-06-01 18:51:02,856: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:51:02,862: INFO/job_scheduler] Job 298: RunJobThread.run
[2017-06-01 18:51:04,814: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:51:04,869: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:51:05,008: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:51:05,069: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:51:06,161: INFO/job_scheduler] Job 295 finished 1 steps successfully
[2017-06-01 18:51:06,189: INFO/job_scheduler] Job 295: complete_job: Updating cache
[2017-06-01 18:51:06,190: INFO/job_scheduler] Job 295 completing (errored=False, cancelled=False)
[2017-06-01 18:51:06,253: INFO/job_scheduler] Job 296 finished 1 steps successfully
[2017-06-01 18:51:06,266: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:51:06,285: INFO/job_scheduler] Job 296: complete_job: Updating cache
[2017-06-01 18:51:06,285: INFO/job_scheduler] Job 296 completing (errored=False, cancelled=False)
[2017-06-01 18:51:06,319: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:51:06,398: INFO/job_scheduler] Job 297 finished 1 steps successfully
[2017-06-01 18:51:06,422: INFO/job_scheduler] Job 297: complete_job: Updating cache
[2017-06-01 18:51:06,422: INFO/job_scheduler] Job 297 completing (errored=False, cancelled=False)
[2017-06-01 18:51:06,471: INFO/job_scheduler] Job 298 finished 1 steps successfully
[2017-06-01 18:51:06,481: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:51:06,497: INFO/job_scheduler] Job 298: complete_job: Updating cache
[2017-06-01 18:51:06,497: INFO/job_scheduler] Job 298 completing (errored=False, cancelled=False)
[2017-06-01 18:51:06,541: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:51:08,507: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:08,509: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:08,523: INFO/command_plan] add_jobs: created Job 299 (Test for host connectivity)
[2017-06-01 18:51:08,543: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:51:08,574: INFO/job_scheduler] Job 299: RunJobThread.run
[2017-06-01 18:51:08,605: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:08,607: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:08,638: INFO/command_plan] add_jobs: created Job 300 (Test for host connectivity)
[2017-06-01 18:51:08,663: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:51:08,671: INFO/job_scheduler] Job 300: RunJobThread.run
[2017-06-01 18:51:08,734: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:08,735: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:08,762: INFO/command_plan] add_jobs: created Job 301 (Test for host connectivity)
[2017-06-01 18:51:08,807: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:51:08,821: INFO/job_scheduler] Job 301: RunJobThread.run
[2017-06-01 18:51:08,887: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:08,887: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:08,899: INFO/command_plan] add_jobs: created Job 302 (Test for host connectivity)
[2017-06-01 18:51:08,923: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:51:08,936: INFO/job_scheduler] Job 302: RunJobThread.run
[2017-06-01 18:51:10,796: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:51:10,951: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:51:11,101: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:51:11,310: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:51:12,163: INFO/job_scheduler] Job 299 finished 1 steps successfully
[2017-06-01 18:51:12,229: INFO/job_scheduler] Job 299: complete_job: Updating cache
[2017-06-01 18:51:12,230: INFO/job_scheduler] Job 299 completing (errored=False, cancelled=False)
[2017-06-01 18:51:12,290: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:51:12,304: INFO/job_scheduler] Job 300 finished 1 steps successfully
[2017-06-01 18:51:12,325: INFO/job_scheduler] Job 300: complete_job: Updating cache
[2017-06-01 18:51:12,326: INFO/job_scheduler] Job 300 completing (errored=False, cancelled=False)
[2017-06-01 18:51:12,407: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:51:12,496: INFO/job_scheduler] Job 301 finished 1 steps successfully
[2017-06-01 18:51:12,538: INFO/job_scheduler] Job 301: complete_job: Updating cache
[2017-06-01 18:51:12,538: INFO/job_scheduler] Job 301 completing (errored=False, cancelled=False)
[2017-06-01 18:51:12,588: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:51:12,700: INFO/job_scheduler] Job 302 finished 1 steps successfully
[2017-06-01 18:51:12,729: INFO/job_scheduler] Job 302: complete_job: Updating cache
[2017-06-01 18:51:12,730: INFO/job_scheduler] Job 302 completing (errored=False, cancelled=False)
[2017-06-01 18:51:12,784: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:51:14,616: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:14,617: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:14,630: INFO/command_plan] add_jobs: created Job 303 (Test for host connectivity)
[2017-06-01 18:51:14,664: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:14,665: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:14,685: INFO/command_plan] add_jobs: created Job 304 (Test for host connectivity)
[2017-06-01 18:51:14,804: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:51:14,807: INFO/job_scheduler] Job 304: RunJobThread.run
[2017-06-01 18:51:14,807: INFO/job_scheduler] Job 303: RunJobThread.run
[2017-06-01 18:51:14,835: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:14,836: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:14,862: INFO/command_plan] add_jobs: created Job 305 (Test for host connectivity)
[2017-06-01 18:51:14,871: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:51:14,874: INFO/job_scheduler] Job 305: RunJobThread.run
[2017-06-01 18:51:14,921: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:51:15,009: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:15,009: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:15,031: INFO/command_plan] add_jobs: created Job 306 (Test for host connectivity)
[2017-06-01 18:51:15,114: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:51:15,117: INFO/job_scheduler] Job 306: RunJobThread.run
[2017-06-01 18:51:17,026: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:51:17,067: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:51:17,237: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:51:17,388: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:51:18,365: INFO/job_scheduler] Job 304 finished 1 steps successfully
[2017-06-01 18:51:18,417: INFO/job_scheduler] Job 303 finished 1 steps successfully
[2017-06-01 18:51:18,428: INFO/job_scheduler] Job 304: complete_job: Updating cache
[2017-06-01 18:51:18,428: INFO/job_scheduler] Job 304 completing (errored=False, cancelled=False)
[2017-06-01 18:51:18,527: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:51:18,545: INFO/job_scheduler] Job 303: complete_job: Updating cache
[2017-06-01 18:51:18,545: INFO/job_scheduler] Job 303 completing (errored=False, cancelled=False)
[2017-06-01 18:51:18,597: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:51:18,705: INFO/job_scheduler] Job 305 finished 1 steps successfully
[2017-06-01 18:51:18,737: INFO/job_scheduler] Job 305: complete_job: Updating cache
[2017-06-01 18:51:18,737: INFO/job_scheduler] Job 305 completing (errored=False, cancelled=False)
[2017-06-01 18:51:18,796: INFO/job_scheduler] Job 306 finished 1 steps successfully
[2017-06-01 18:51:18,812: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:51:18,845: INFO/job_scheduler] Job 306: complete_job: Updating cache
[2017-06-01 18:51:18,845: INFO/job_scheduler] Job 306 completing (errored=False, cancelled=False)
[2017-06-01 18:51:18,878: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:51:20,781: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:20,781: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:20,803: INFO/command_plan] add_jobs: created Job 307 (Test for host connectivity)
[2017-06-01 18:51:20,864: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:20,864: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:20,878: INFO/command_plan] add_jobs: created Job 308 (Test for host connectivity)
[2017-06-01 18:51:20,891: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:51:20,916: INFO/job_scheduler] Job 307: RunJobThread.run
[2017-06-01 18:51:20,932: INFO/job_scheduler] Job 308: RunJobThread.run
[2017-06-01 18:51:20,958: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:20,958: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:20,981: INFO/command_plan] add_jobs: created Job 309 (Test for host connectivity)
[2017-06-01 18:51:20,997: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:51:20,999: INFO/job_scheduler] Job 309: RunJobThread.run
[2017-06-01 18:51:21,038: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:51:21,133: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:21,134: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:21,169: INFO/command_plan] add_jobs: created Job 310 (Test for host connectivity)
[2017-06-01 18:51:21,227: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:51:21,245: INFO/job_scheduler] Job 310: RunJobThread.run
[2017-06-01 18:51:23,247: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:51:23,274: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:51:23,382: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:51:23,441: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:51:24,605: INFO/job_scheduler] Job 307 finished 1 steps successfully
[2017-06-01 18:51:24,607: INFO/job_scheduler] Job 308 finished 1 steps successfully
[2017-06-01 18:51:24,635: INFO/job_scheduler] Job 307: complete_job: Updating cache
[2017-06-01 18:51:24,635: INFO/job_scheduler] Job 307 completing (errored=False, cancelled=False)
[2017-06-01 18:51:24,722: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:51:24,743: INFO/job_scheduler] Job 308: complete_job: Updating cache
[2017-06-01 18:51:24,744: INFO/job_scheduler] Job 308 completing (errored=False, cancelled=False)
[2017-06-01 18:51:24,789: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:51:24,817: INFO/job_scheduler] Job 309 finished 1 steps successfully
[2017-06-01 18:51:24,835: INFO/job_scheduler] Job 309: complete_job: Updating cache
[2017-06-01 18:51:24,835: INFO/job_scheduler] Job 309 completing (errored=False, cancelled=False)
[2017-06-01 18:51:24,899: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:51:24,936: INFO/job_scheduler] Job 310 finished 1 steps successfully
[2017-06-01 18:51:25,019: INFO/job_scheduler] Job 310: complete_job: Updating cache
[2017-06-01 18:51:25,019: INFO/job_scheduler] Job 310 completing (errored=False, cancelled=False)
[2017-06-01 18:51:25,061: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:51:26,837: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:26,838: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:26,860: INFO/command_plan] add_jobs: created Job 311 (Test for host connectivity)
[2017-06-01 18:51:26,879: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:51:26,898: INFO/job_scheduler] Job 311: RunJobThread.run
[2017-06-01 18:51:26,934: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:26,934: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:26,961: INFO/command_plan] add_jobs: created Job 312 (Test for host connectivity)
[2017-06-01 18:51:27,016: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:51:27,046: INFO/job_scheduler] Job 312: RunJobThread.run
[2017-06-01 18:51:27,092: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:27,093: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:27,143: INFO/command_plan] add_jobs: created Job 313 (Test for host connectivity)
[2017-06-01 18:51:27,193: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:51:27,224: INFO/job_scheduler] Job 313: RunJobThread.run
[2017-06-01 18:51:27,266: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:27,267: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:27,277: INFO/command_plan] add_jobs: created Job 314 (Test for host connectivity)
[2017-06-01 18:51:27,331: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:51:27,369: INFO/job_scheduler] Job 314: RunJobThread.run
[2017-06-01 18:51:29,171: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:51:29,353: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:51:29,437: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:51:29,598: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:51:30,556: INFO/job_scheduler] Job 311 finished 1 steps successfully
[2017-06-01 18:51:30,576: INFO/job_scheduler] Job 311: complete_job: Updating cache
[2017-06-01 18:51:30,576: INFO/job_scheduler] Job 311 completing (errored=False, cancelled=False)
[2017-06-01 18:51:30,654: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:51:30,686: INFO/job_scheduler] Job 312 finished 1 steps successfully
[2017-06-01 18:51:30,736: INFO/job_scheduler] Job 312: complete_job: Updating cache
[2017-06-01 18:51:30,736: INFO/job_scheduler] Job 312 completing (errored=False, cancelled=False)
[2017-06-01 18:51:30,790: INFO/job_scheduler] Job 313 finished 1 steps successfully
[2017-06-01 18:51:30,812: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:51:30,837: INFO/job_scheduler] Job 313: complete_job: Updating cache
[2017-06-01 18:51:30,837: INFO/job_scheduler] Job 313 completing (errored=False, cancelled=False)
[2017-06-01 18:51:30,878: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:51:31,091: INFO/job_scheduler] Job 314 finished 1 steps successfully
[2017-06-01 18:51:31,109: INFO/job_scheduler] Job 314: complete_job: Updating cache
[2017-06-01 18:51:31,109: INFO/job_scheduler] Job 314 completing (errored=False, cancelled=False)
[2017-06-01 18:51:31,151: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:51:32,985: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:32,985: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:33,009: INFO/command_plan] add_jobs: created Job 315 (Test for host connectivity)
[2017-06-01 18:51:33,071: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:51:33,084: INFO/job_scheduler] Job 315: RunJobThread.run
[2017-06-01 18:51:33,106: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:33,107: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:33,118: INFO/command_plan] add_jobs: created Job 316 (Test for host connectivity)
[2017-06-01 18:51:33,173: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:51:33,176: INFO/job_scheduler] Job 316: RunJobThread.run
[2017-06-01 18:51:33,246: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:33,247: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:33,275: INFO/command_plan] add_jobs: created Job 317 (Test for host connectivity)
[2017-06-01 18:51:33,360: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:33,360: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:33,381: INFO/command_plan] add_jobs: created Job 318 (Test for host connectivity)
[2017-06-01 18:51:33,420: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 18:51:33,463: INFO/job_scheduler] Job 317: RunJobThread.run
[2017-06-01 18:51:33,481: INFO/job_scheduler] Job 318: RunJobThread.run
[2017-06-01 18:51:33,510: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 18:51:35,364: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:51:35,485: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:51:35,786: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:51:35,865: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:51:36,660: INFO/job_scheduler] Job 315 finished 1 steps successfully
[2017-06-01 18:51:36,708: INFO/job_scheduler] Job 315: complete_job: Updating cache
[2017-06-01 18:51:36,708: INFO/job_scheduler] Job 315 completing (errored=False, cancelled=False)
[2017-06-01 18:51:36,808: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:51:36,914: INFO/job_scheduler] Job 316 finished 1 steps successfully
[2017-06-01 18:51:36,958: INFO/job_scheduler] Job 316: complete_job: Updating cache
[2017-06-01 18:51:36,958: INFO/job_scheduler] Job 316 completing (errored=False, cancelled=False)
[2017-06-01 18:51:37,000: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:51:37,036: INFO/job_scheduler] Job 317 finished 1 steps successfully
[2017-06-01 18:51:37,066: INFO/job_scheduler] Job 317: complete_job: Updating cache
[2017-06-01 18:51:37,067: INFO/job_scheduler] Job 317 completing (errored=False, cancelled=False)
[2017-06-01 18:51:37,133: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:51:37,193: INFO/job_scheduler] Job 318 finished 1 steps successfully
[2017-06-01 18:51:37,216: INFO/job_scheduler] Job 318: complete_job: Updating cache
[2017-06-01 18:51:37,216: INFO/job_scheduler] Job 318 completing (errored=False, cancelled=False)
[2017-06-01 18:51:37,258: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:51:39,834: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:39,834: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:39,853: INFO/command_plan] add_jobs: created Job 319 (Test for host connectivity)
[2017-06-01 18:51:39,890: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:39,890: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:39,912: INFO/command_plan] add_jobs: created Job 320 (Test for host connectivity)
[2017-06-01 18:51:39,921: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:51:39,946: INFO/job_scheduler] Job 320: RunJobThread.run
[2017-06-01 18:51:39,957: INFO/job_scheduler] Job 319: RunJobThread.run
[2017-06-01 18:51:39,993: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:39,993: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:40,024: INFO/command_plan] add_jobs: created Job 321 (Test for host connectivity)
[2017-06-01 18:51:40,037: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:51:40,051: INFO/job_scheduler] Job 321: RunJobThread.run
[2017-06-01 18:51:40,104: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:51:40,184: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:40,186: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:40,209: INFO/command_plan] add_jobs: created Job 322 (Test for host connectivity)
[2017-06-01 18:51:40,276: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:51:40,290: INFO/job_scheduler] Job 322: RunJobThread.run
[2017-06-01 18:51:42,173: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:51:42,218: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:51:42,344: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:51:42,460: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:51:43,537: INFO/job_scheduler] Job 320 finished 1 steps successfully
[2017-06-01 18:51:43,605: INFO/job_scheduler] Job 319 finished 1 steps successfully
[2017-06-01 18:51:43,646: INFO/job_scheduler] Job 320: complete_job: Updating cache
[2017-06-01 18:51:43,754: INFO/job_scheduler] Job 320 completing (errored=False, cancelled=False)
[2017-06-01 18:51:43,767: INFO/job_scheduler] Job 321 finished 1 steps successfully
[2017-06-01 18:51:43,820: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:51:43,845: INFO/job_scheduler] Job 319: complete_job: Updating cache
[2017-06-01 18:51:43,846: INFO/job_scheduler] Job 319 completing (errored=False, cancelled=False)
[2017-06-01 18:51:43,849: INFO/job_scheduler] Job 322 finished 1 steps successfully
[2017-06-01 18:51:43,896: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:51:43,920: INFO/job_scheduler] Job 321: complete_job: Updating cache
[2017-06-01 18:51:43,920: INFO/job_scheduler] Job 321 completing (errored=False, cancelled=False)
[2017-06-01 18:51:43,984: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:51:44,103: INFO/job_scheduler] Job 322: complete_job: Updating cache
[2017-06-01 18:51:44,103: INFO/job_scheduler] Job 322 completing (errored=False, cancelled=False)
[2017-06-01 18:51:44,147: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:51:45,900: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:45,901: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:45,921: INFO/command_plan] add_jobs: created Job 323 (Test for host connectivity)
[2017-06-01 18:51:45,972: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:51:45,974: INFO/job_scheduler] Job 323: RunJobThread.run
[2017-06-01 18:51:45,995: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:45,995: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:46,070: INFO/command_plan] add_jobs: created Job 324 (Test for host connectivity)
[2017-06-01 18:51:46,088: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:51:46,121: INFO/job_scheduler] Job 324: RunJobThread.run
[2017-06-01 18:51:46,185: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:46,186: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:46,203: INFO/command_plan] add_jobs: created Job 325 (Test for host connectivity)
[2017-06-01 18:51:46,215: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:51:46,226: INFO/job_scheduler] Job 325: RunJobThread.run
[2017-06-01 18:51:46,309: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:46,309: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:46,348: INFO/command_plan] add_jobs: created Job 326 (Test for host connectivity)
[2017-06-01 18:51:46,393: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:51:46,400: INFO/job_scheduler] Job 326: RunJobThread.run
[2017-06-01 18:51:48,284: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:51:48,462: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:51:48,552: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:51:48,614: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:51:49,622: INFO/job_scheduler] Job 323 finished 1 steps successfully
[2017-06-01 18:51:49,677: INFO/job_scheduler] Job 323: complete_job: Updating cache
[2017-06-01 18:51:49,677: INFO/job_scheduler] Job 323 completing (errored=False, cancelled=False)
[2017-06-01 18:51:49,767: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:51:49,810: INFO/job_scheduler] Job 324 finished 1 steps successfully
[2017-06-01 18:51:49,877: INFO/job_scheduler] Job 324: complete_job: Updating cache
[2017-06-01 18:51:49,877: INFO/job_scheduler] Job 324 completing (errored=False, cancelled=False)
[2017-06-01 18:51:49,919: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:51:50,032: INFO/job_scheduler] Job 325 finished 1 steps successfully
[2017-06-01 18:51:50,060: INFO/job_scheduler] Job 325: complete_job: Updating cache
[2017-06-01 18:51:50,060: INFO/job_scheduler] Job 325 completing (errored=False, cancelled=False)
[2017-06-01 18:51:50,061: INFO/job_scheduler] Job 326 finished 1 steps successfully
[2017-06-01 18:51:50,102: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:51:50,119: INFO/job_scheduler] Job 326: complete_job: Updating cache
[2017-06-01 18:51:50,120: INFO/job_scheduler] Job 326 completing (errored=False, cancelled=False)
[2017-06-01 18:51:50,202: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:51:52,155: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:52,157: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:52,170: INFO/command_plan] add_jobs: created Job 327 (Test for host connectivity)
[2017-06-01 18:51:52,212: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:52,212: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:52,235: INFO/command_plan] add_jobs: created Job 328 (Test for host connectivity)
[2017-06-01 18:51:52,244: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:51:52,250: INFO/job_scheduler] Job 328: RunJobThread.run
[2017-06-01 18:51:52,274: INFO/job_scheduler] Job 327: RunJobThread.run
[2017-06-01 18:51:52,285: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:51:52,369: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:52,370: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:52,392: INFO/command_plan] add_jobs: created Job 329 (Test for host connectivity)
[2017-06-01 18:51:52,465: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:51:52,488: INFO/job_scheduler] Job 329: RunJobThread.run
[2017-06-01 18:51:52,527: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:52,527: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:52,551: INFO/command_plan] add_jobs: created Job 330 (Test for host connectivity)
[2017-06-01 18:51:52,579: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:51:52,584: INFO/job_scheduler] Job 330: RunJobThread.run
[2017-06-01 18:51:54,549: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:51:54,588: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:51:54,660: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:51:54,818: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:51:55,886: INFO/job_scheduler] Job 327 finished 1 steps successfully
[2017-06-01 18:51:55,951: INFO/job_scheduler] Job 327: complete_job: Updating cache
[2017-06-01 18:51:55,952: INFO/job_scheduler] Job 327 completing (errored=False, cancelled=False)
[2017-06-01 18:51:55,961: INFO/job_scheduler] Job 328 finished 1 steps successfully
[2017-06-01 18:51:55,994: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:51:56,015: INFO/job_scheduler] Job 328: complete_job: Updating cache
[2017-06-01 18:51:56,016: INFO/job_scheduler] Job 328 completing (errored=False, cancelled=False)
[2017-06-01 18:51:56,091: INFO/job_scheduler] Job 329 finished 1 steps successfully
[2017-06-01 18:51:56,105: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:51:56,127: INFO/job_scheduler] Job 329: complete_job: Updating cache
[2017-06-01 18:51:56,127: INFO/job_scheduler] Job 329 completing (errored=False, cancelled=False)
[2017-06-01 18:51:56,198: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:51:56,233: INFO/job_scheduler] Job 330 finished 1 steps successfully
[2017-06-01 18:51:56,261: INFO/job_scheduler] Job 330: complete_job: Updating cache
[2017-06-01 18:51:56,261: INFO/job_scheduler] Job 330 completing (errored=False, cancelled=False)
[2017-06-01 18:51:56,384: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:51:58,228: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:58,229: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:58,243: INFO/command_plan] add_jobs: created Job 331 (Test for host connectivity)
[2017-06-01 18:51:58,296: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:58,312: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:58,334: INFO/command_plan] add_jobs: created Job 332 (Test for host connectivity)
[2017-06-01 18:51:58,352: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:51:58,364: INFO/job_scheduler] Job 331: RunJobThread.run
[2017-06-01 18:51:58,408: INFO/job_scheduler] Job 332: RunJobThread.run
[2017-06-01 18:51:58,421: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:51:58,485: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:58,487: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:58,510: INFO/command_plan] add_jobs: created Job 333 (Test for host connectivity)
[2017-06-01 18:51:58,570: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:51:58,580: INFO/job_scheduler] Job 333: RunJobThread.run
[2017-06-01 18:51:58,655: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:51:58,662: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:51:58,678: INFO/command_plan] add_jobs: created Job 334 (Test for host connectivity)
[2017-06-01 18:51:58,775: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:51:58,777: INFO/job_scheduler] Job 334: RunJobThread.run
[2017-06-01 18:52:00,681: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:52:00,788: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:52:00,797: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:52:00,907: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:52:02,030: INFO/job_scheduler] Job 332 finished 1 steps successfully
[2017-06-01 18:52:02,043: INFO/job_scheduler] Job 332: complete_job: Updating cache
[2017-06-01 18:52:02,043: INFO/job_scheduler] Job 332 completing (errored=False, cancelled=False)
[2017-06-01 18:52:02,084: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:52:02,119: INFO/job_scheduler] Job 331 finished 1 steps successfully
[2017-06-01 18:52:02,134: INFO/job_scheduler] Job 331: complete_job: Updating cache
[2017-06-01 18:52:02,134: INFO/job_scheduler] Job 331 completing (errored=False, cancelled=False)
[2017-06-01 18:52:02,208: INFO/job_scheduler] Job 333 finished 1 steps successfully
[2017-06-01 18:52:02,227: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:52:02,242: INFO/job_scheduler] Job 334 finished 1 steps successfully
[2017-06-01 18:52:02,242: INFO/job_scheduler] Job 333: complete_job: Updating cache
[2017-06-01 18:52:02,243: INFO/job_scheduler] Job 333 completing (errored=False, cancelled=False)
[2017-06-01 18:52:02,311: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:52:02,329: INFO/job_scheduler] Job 334: complete_job: Updating cache
[2017-06-01 18:52:02,329: INFO/job_scheduler] Job 334 completing (errored=False, cancelled=False)
[2017-06-01 18:52:02,411: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:52:04,391: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:04,393: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:04,409: INFO/command_plan] add_jobs: created Job 335 (Test for host connectivity)
[2017-06-01 18:52:04,417: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:52:04,419: INFO/job_scheduler] Job 335: RunJobThread.run
[2017-06-01 18:52:04,487: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:04,488: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:04,503: INFO/command_plan] add_jobs: created Job 336 (Test for host connectivity)
[2017-06-01 18:52:04,576: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:04,576: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:04,611: INFO/command_plan] add_jobs: created Job 337 (Test for host connectivity)
[2017-06-01 18:52:04,628: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 18:52:04,656: INFO/job_scheduler] Job 336: RunJobThread.run
[2017-06-01 18:52:04,676: INFO/job_scheduler] Job 337: RunJobThread.run
[2017-06-01 18:52:04,687: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:52:04,744: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:04,746: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:04,762: INFO/command_plan] add_jobs: created Job 338 (Test for host connectivity)
[2017-06-01 18:52:04,859: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:52:04,861: INFO/job_scheduler] Job 338: RunJobThread.run
[2017-06-01 18:52:06,734: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:52:06,841: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:52:07,038: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:52:07,057: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:52:08,048: INFO/job_scheduler] Job 335 finished 1 steps successfully
[2017-06-01 18:52:08,112: INFO/job_scheduler] Job 335: complete_job: Updating cache
[2017-06-01 18:52:08,113: INFO/job_scheduler] Job 335 completing (errored=False, cancelled=False)
[2017-06-01 18:52:08,200: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:52:08,208: INFO/job_scheduler] Job 336 finished 1 steps successfully
[2017-06-01 18:52:08,242: INFO/job_scheduler] Job 336: complete_job: Updating cache
[2017-06-01 18:52:08,242: INFO/job_scheduler] Job 336 completing (errored=False, cancelled=False)
[2017-06-01 18:52:08,310: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:52:08,404: INFO/job_scheduler] Job 338 finished 1 steps successfully
[2017-06-01 18:52:08,459: INFO/job_scheduler] Job 338: complete_job: Updating cache
[2017-06-01 18:52:08,459: INFO/job_scheduler] Job 338 completing (errored=False, cancelled=False)
[2017-06-01 18:52:08,525: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:52:08,536: INFO/job_scheduler] Job 337 finished 1 steps successfully
[2017-06-01 18:52:08,562: INFO/job_scheduler] Job 337: complete_job: Updating cache
[2017-06-01 18:52:08,562: INFO/job_scheduler] Job 337 completing (errored=False, cancelled=False)
[2017-06-01 18:52:08,621: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:52:10,496: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:10,497: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:10,516: INFO/command_plan] add_jobs: created Job 339 (Test for host connectivity)
[2017-06-01 18:52:10,543: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:52:10,558: INFO/job_scheduler] Job 339: RunJobThread.run
[2017-06-01 18:52:10,591: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:10,592: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:10,617: INFO/command_plan] add_jobs: created Job 340 (Test for host connectivity)
[2017-06-01 18:52:10,656: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:52:10,666: INFO/job_scheduler] Job 340: RunJobThread.run
[2017-06-01 18:52:10,737: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:10,737: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:10,758: INFO/command_plan] add_jobs: created Job 341 (Test for host connectivity)
[2017-06-01 18:52:10,790: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:52:10,807: INFO/job_scheduler] Job 341: RunJobThread.run
[2017-06-01 18:52:10,855: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:10,855: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:10,876: INFO/command_plan] add_jobs: created Job 342 (Test for host connectivity)
[2017-06-01 18:52:10,916: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:52:10,926: INFO/job_scheduler] Job 342: RunJobThread.run
[2017-06-01 18:52:12,749: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:52:12,981: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:52:13,075: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:52:13,191: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:52:14,076: INFO/job_scheduler] Job 339 finished 1 steps successfully
[2017-06-01 18:52:14,145: INFO/job_scheduler] Job 339: complete_job: Updating cache
[2017-06-01 18:52:14,145: INFO/job_scheduler] Job 339 completing (errored=False, cancelled=False)
[2017-06-01 18:52:14,316: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:52:14,356: INFO/job_scheduler] Job 340 finished 1 steps successfully
[2017-06-01 18:52:14,408: INFO/job_scheduler] Job 340: complete_job: Updating cache
[2017-06-01 18:52:14,408: INFO/job_scheduler] Job 340 completing (errored=False, cancelled=False)
[2017-06-01 18:52:14,449: INFO/job_scheduler] Job 341 finished 1 steps successfully
[2017-06-01 18:52:14,460: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:52:14,483: INFO/job_scheduler] Job 341: complete_job: Updating cache
[2017-06-01 18:52:14,483: INFO/job_scheduler] Job 341 completing (errored=False, cancelled=False)
[2017-06-01 18:52:14,538: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:52:14,629: INFO/job_scheduler] Job 342 finished 1 steps successfully
[2017-06-01 18:52:14,674: INFO/job_scheduler] Job 342: complete_job: Updating cache
[2017-06-01 18:52:14,675: INFO/job_scheduler] Job 342 completing (errored=False, cancelled=False)
[2017-06-01 18:52:14,719: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:52:16,594: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:16,594: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:16,624: INFO/command_plan] add_jobs: created Job 343 (Test for host connectivity)
[2017-06-01 18:52:16,639: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:52:16,643: INFO/job_scheduler] Job 343: RunJobThread.run
[2017-06-01 18:52:16,697: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:16,697: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:16,709: INFO/command_plan] add_jobs: created Job 344 (Test for host connectivity)
[2017-06-01 18:52:16,741: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:52:16,745: INFO/job_scheduler] Job 344: RunJobThread.run
[2017-06-01 18:52:16,875: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:16,875: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:16,894: INFO/command_plan] add_jobs: created Job 345 (Test for host connectivity)
[2017-06-01 18:52:16,951: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:52:16,953: INFO/job_scheduler] Job 345: RunJobThread.run
[2017-06-01 18:52:16,998: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:16,999: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:17,016: INFO/command_plan] add_jobs: created Job 346 (Test for host connectivity)
[2017-06-01 18:52:17,053: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:52:17,059: INFO/job_scheduler] Job 346: RunJobThread.run
[2017-06-01 18:52:18,953: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:52:19,052: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:52:19,247: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:52:19,266: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:52:20,352: INFO/job_scheduler] Job 343 finished 1 steps successfully
[2017-06-01 18:52:20,390: INFO/job_scheduler] Job 343: complete_job: Updating cache
[2017-06-01 18:52:20,391: INFO/job_scheduler] Job 343 completing (errored=False, cancelled=False)
[2017-06-01 18:52:20,460: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:52:20,499: INFO/job_scheduler] Job 344 finished 1 steps successfully
[2017-06-01 18:52:20,540: INFO/job_scheduler] Job 344: complete_job: Updating cache
[2017-06-01 18:52:20,541: INFO/job_scheduler] Job 344 completing (errored=False, cancelled=False)
[2017-06-01 18:52:20,582: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:52:20,624: INFO/job_scheduler] Job 346 finished 1 steps successfully
[2017-06-01 18:52:20,656: INFO/job_scheduler] Job 345 finished 1 steps successfully
[2017-06-01 18:52:20,665: INFO/job_scheduler] Job 346: complete_job: Updating cache
[2017-06-01 18:52:20,665: INFO/job_scheduler] Job 346 completing (errored=False, cancelled=False)
[2017-06-01 18:52:20,726: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:52:20,740: INFO/job_scheduler] Job 345: complete_job: Updating cache
[2017-06-01 18:52:20,741: INFO/job_scheduler] Job 345 completing (errored=False, cancelled=False)
[2017-06-01 18:52:20,803: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:52:22,825: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:22,826: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:22,848: INFO/command_plan] add_jobs: created Job 347 (Test for host connectivity)
[2017-06-01 18:52:22,870: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:52:22,876: INFO/job_scheduler] Job 347: RunJobThread.run
[2017-06-01 18:52:23,025: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:23,026: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:23,044: INFO/command_plan] add_jobs: created Job 348 (Test for host connectivity)
[2017-06-01 18:52:23,089: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:52:23,104: INFO/job_scheduler] Job 348: RunJobThread.run
[2017-06-01 18:52:23,152: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:23,153: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:23,165: INFO/command_plan] add_jobs: created Job 349 (Test for host connectivity)
[2017-06-01 18:52:23,197: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:52:23,245: INFO/job_scheduler] Job 349: RunJobThread.run
[2017-06-01 18:52:23,304: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:23,304: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:23,336: INFO/command_plan] add_jobs: created Job 350 (Test for host connectivity)
[2017-06-01 18:52:23,389: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:52:23,404: INFO/job_scheduler] Job 350: RunJobThread.run
[2017-06-01 18:52:25,079: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:52:25,379: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:52:25,562: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:52:25,632: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:52:26,376: INFO/job_scheduler] Job 347 finished 1 steps successfully
[2017-06-01 18:52:26,423: INFO/job_scheduler] Job 347: complete_job: Updating cache
[2017-06-01 18:52:26,423: INFO/job_scheduler] Job 347 completing (errored=False, cancelled=False)
[2017-06-01 18:52:26,492: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:52:26,746: INFO/job_scheduler] Job 348 finished 1 steps successfully
[2017-06-01 18:52:26,781: INFO/job_scheduler] Job 348: complete_job: Updating cache
[2017-06-01 18:52:26,781: INFO/job_scheduler] Job 348 completing (errored=False, cancelled=False)
[2017-06-01 18:52:26,815: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:52:27,006: INFO/job_scheduler] Job 349 finished 1 steps successfully
[2017-06-01 18:52:27,018: INFO/job_scheduler] Job 350 finished 1 steps successfully
[2017-06-01 18:52:27,056: INFO/job_scheduler] Job 349: complete_job: Updating cache
[2017-06-01 18:52:27,056: INFO/job_scheduler] Job 349 completing (errored=False, cancelled=False)
[2017-06-01 18:52:27,099: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:52:27,120: INFO/job_scheduler] Job 350: complete_job: Updating cache
[2017-06-01 18:52:27,120: INFO/job_scheduler] Job 350 completing (errored=False, cancelled=False)
[2017-06-01 18:52:27,207: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:52:29,057: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:29,057: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:29,081: INFO/command_plan] add_jobs: created Job 351 (Test for host connectivity)
[2017-06-01 18:52:29,126: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:29,126: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:29,149: INFO/command_plan] add_jobs: created Job 352 (Test for host connectivity)
[2017-06-01 18:52:29,157: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:52:29,167: INFO/job_scheduler] Job 352: RunJobThread.run
[2017-06-01 18:52:29,190: INFO/job_scheduler] Job 351: RunJobThread.run
[2017-06-01 18:52:29,200: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:52:29,257: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:29,257: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:29,280: INFO/command_plan] add_jobs: created Job 353 (Test for host connectivity)
[2017-06-01 18:52:29,417: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:52:29,419: INFO/job_scheduler] Job 353: RunJobThread.run
[2017-06-01 18:52:29,442: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:29,443: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:29,480: INFO/command_plan] add_jobs: created Job 354 (Test for host connectivity)
[2017-06-01 18:52:29,507: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:52:29,509: INFO/job_scheduler] Job 354: RunJobThread.run
[2017-06-01 18:52:31,496: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:52:31,529: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:52:31,684: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:52:31,799: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:52:32,888: INFO/job_scheduler] Job 352 finished 1 steps successfully
[2017-06-01 18:52:32,889: INFO/job_scheduler] Job 351 finished 1 steps successfully
[2017-06-01 18:52:32,913: INFO/job_scheduler] Job 352: complete_job: Updating cache
[2017-06-01 18:52:32,913: INFO/job_scheduler] Job 352 completing (errored=False, cancelled=False)
[2017-06-01 18:52:32,946: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:52:32,963: INFO/job_scheduler] Job 351: complete_job: Updating cache
[2017-06-01 18:52:32,964: INFO/job_scheduler] Job 351 completing (errored=False, cancelled=False)
[2017-06-01 18:52:33,086: INFO/job_scheduler] Job 353 finished 1 steps successfully
[2017-06-01 18:52:33,088: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:52:33,106: INFO/job_scheduler] Job 353: complete_job: Updating cache
[2017-06-01 18:52:33,106: INFO/job_scheduler] Job 353 completing (errored=False, cancelled=False)
[2017-06-01 18:52:33,157: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:52:33,182: INFO/job_scheduler] Job 354 finished 1 steps successfully
[2017-06-01 18:52:33,206: INFO/job_scheduler] Job 354: complete_job: Updating cache
[2017-06-01 18:52:33,206: INFO/job_scheduler] Job 354 completing (errored=False, cancelled=False)
[2017-06-01 18:52:33,238: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:52:35,178: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:35,180: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:35,197: INFO/command_plan] add_jobs: created Job 355 (Test for host connectivity)
[2017-06-01 18:52:35,241: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:52:35,263: INFO/job_scheduler] Job 355: RunJobThread.run
[2017-06-01 18:52:35,307: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:35,314: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:35,331: INFO/command_plan] add_jobs: created Job 356 (Test for host connectivity)
[2017-06-01 18:52:35,378: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:52:35,385: INFO/job_scheduler] Job 356: RunJobThread.run
[2017-06-01 18:52:35,438: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:35,440: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:35,487: INFO/command_plan] add_jobs: created Job 357 (Test for host connectivity)
[2017-06-01 18:52:35,564: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:35,565: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:35,587: INFO/command_plan] add_jobs: created Job 358 (Test for host connectivity)
[2017-06-01 18:52:35,605: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 18:52:35,619: INFO/job_scheduler] Job 357: RunJobThread.run
[2017-06-01 18:52:35,655: INFO/job_scheduler] Job 358: RunJobThread.run
[2017-06-01 18:52:35,670: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 18:52:37,445: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:52:37,530: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:52:37,919: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:52:37,977: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:52:38,818: INFO/job_scheduler] Job 355 finished 1 steps successfully
[2017-06-01 18:52:38,845: INFO/job_scheduler] Job 355: complete_job: Updating cache
[2017-06-01 18:52:38,845: INFO/job_scheduler] Job 355 completing (errored=False, cancelled=False)
[2017-06-01 18:52:38,897: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:52:38,950: INFO/job_scheduler] Job 356 finished 1 steps successfully
[2017-06-01 18:52:38,978: INFO/job_scheduler] Job 356: complete_job: Updating cache
[2017-06-01 18:52:39,002: INFO/job_scheduler] Job 356 completing (errored=False, cancelled=False)
[2017-06-01 18:52:39,056: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:52:39,290: INFO/job_scheduler] Job 358 finished 1 steps successfully
[2017-06-01 18:52:39,345: INFO/job_scheduler] Job 358: complete_job: Updating cache
[2017-06-01 18:52:39,345: INFO/job_scheduler] Job 358 completing (errored=False, cancelled=False)
[2017-06-01 18:52:39,386: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:52:39,437: INFO/job_scheduler] Job 357 finished 1 steps successfully
[2017-06-01 18:52:39,470: INFO/job_scheduler] Job 357: complete_job: Updating cache
[2017-06-01 18:52:39,470: INFO/job_scheduler] Job 357 completing (errored=False, cancelled=False)
[2017-06-01 18:52:39,504: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:52:41,338: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:41,338: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:41,345: INFO/command_plan] add_jobs: created Job 359 (Test for host connectivity)
[2017-06-01 18:52:41,377: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:52:41,385: INFO/job_scheduler] Job 359: RunJobThread.run
[2017-06-01 18:52:41,438: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:41,439: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:41,454: INFO/command_plan] add_jobs: created Job 360 (Test for host connectivity)
[2017-06-01 18:52:41,548: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:41,548: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:41,571: INFO/command_plan] add_jobs: created Job 361 (Test for host connectivity)
[2017-06-01 18:52:41,602: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 18:52:41,606: INFO/job_scheduler] Job 360: RunJobThread.run
[2017-06-01 18:52:41,632: INFO/job_scheduler] Job 361: RunJobThread.run
[2017-06-01 18:52:41,647: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:52:41,702: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:41,703: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:41,714: INFO/command_plan] add_jobs: created Job 362 (Test for host connectivity)
[2017-06-01 18:52:41,878: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:52:41,881: INFO/job_scheduler] Job 362: RunJobThread.run
[2017-06-01 18:52:43,651: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:52:43,895: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:52:43,932: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:52:44,086: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:52:45,002: INFO/job_scheduler] Job 359 finished 1 steps successfully
[2017-06-01 18:52:45,044: INFO/job_scheduler] Job 359: complete_job: Updating cache
[2017-06-01 18:52:45,044: INFO/job_scheduler] Job 359 completing (errored=False, cancelled=False)
[2017-06-01 18:52:45,117: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:52:45,277: INFO/job_scheduler] Job 361 finished 1 steps successfully
[2017-06-01 18:52:45,386: INFO/job_scheduler] Job 360 finished 1 steps successfully
[2017-06-01 18:52:45,410: INFO/job_scheduler] Job 361: complete_job: Updating cache
[2017-06-01 18:52:45,410: INFO/job_scheduler] Job 361 completing (errored=False, cancelled=False)
[2017-06-01 18:52:45,469: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:52:45,483: INFO/job_scheduler] Job 362 finished 1 steps successfully
[2017-06-01 18:52:45,485: INFO/job_scheduler] Job 360: complete_job: Updating cache
[2017-06-01 18:52:45,485: INFO/job_scheduler] Job 360 completing (errored=False, cancelled=False)
[2017-06-01 18:52:45,537: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:52:45,569: INFO/job_scheduler] Job 362: complete_job: Updating cache
[2017-06-01 18:52:45,569: INFO/job_scheduler] Job 362 completing (errored=False, cancelled=False)
[2017-06-01 18:52:45,619: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:52:47,520: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:47,522: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:47,535: INFO/command_plan] add_jobs: created Job 363 (Test for host connectivity)
[2017-06-01 18:52:47,594: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:52:47,611: INFO/job_scheduler] Job 363: RunJobThread.run
[2017-06-01 18:52:47,637: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:47,637: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:47,710: INFO/command_plan] add_jobs: created Job 364 (Test for host connectivity)
[2017-06-01 18:52:47,747: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:52:47,769: INFO/job_scheduler] Job 364: RunJobThread.run
[2017-06-01 18:52:47,807: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:47,807: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:47,837: INFO/command_plan] add_jobs: created Job 365 (Test for host connectivity)
[2017-06-01 18:52:47,988: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:47,989: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:48,002: INFO/command_plan] add_jobs: created Job 366 (Test for host connectivity)
[2017-06-01 18:52:48,010: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 18:52:48,014: INFO/job_scheduler] Job 365: RunJobThread.run
[2017-06-01 18:52:48,015: INFO/job_scheduler] Job 366: RunJobThread.run
[2017-06-01 18:52:48,027: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 18:52:49,888: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:52:50,004: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:52:50,354: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:52:50,363: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:52:51,249: INFO/job_scheduler] Job 363 finished 1 steps successfully
[2017-06-01 18:52:51,284: INFO/job_scheduler] Job 363: complete_job: Updating cache
[2017-06-01 18:52:51,284: INFO/job_scheduler] Job 363 completing (errored=False, cancelled=False)
[2017-06-01 18:52:51,366: INFO/job_scheduler] Job 364 finished 1 steps successfully
[2017-06-01 18:52:51,401: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:52:51,437: INFO/job_scheduler] Job 364: complete_job: Updating cache
[2017-06-01 18:52:51,437: INFO/job_scheduler] Job 364 completing (errored=False, cancelled=False)
[2017-06-01 18:52:51,526: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:52:51,759: INFO/job_scheduler] Job 366 finished 1 steps successfully
[2017-06-01 18:52:51,767: INFO/job_scheduler] Job 366: complete_job: Updating cache
[2017-06-01 18:52:51,767: INFO/job_scheduler] Job 366 completing (errored=False, cancelled=False)
[2017-06-01 18:52:51,827: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:52:51,918: INFO/job_scheduler] Job 365 finished 1 steps successfully
[2017-06-01 18:52:51,959: INFO/job_scheduler] Job 365: complete_job: Updating cache
[2017-06-01 18:52:51,960: INFO/job_scheduler] Job 365 completing (errored=False, cancelled=False)
[2017-06-01 18:52:52,015: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:52:53,682: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:53,682: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:53,692: INFO/command_plan] add_jobs: created Job 367 (Test for host connectivity)
[2017-06-01 18:52:53,738: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:52:53,740: INFO/job_scheduler] Job 367: RunJobThread.run
[2017-06-01 18:52:53,786: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:53,788: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:53,808: INFO/command_plan] add_jobs: created Job 368 (Test for host connectivity)
[2017-06-01 18:52:53,841: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:52:53,851: INFO/job_scheduler] Job 368: RunJobThread.run
[2017-06-01 18:52:53,949: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:53,949: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:53,992: INFO/command_plan] add_jobs: created Job 369 (Test for host connectivity)
[2017-06-01 18:52:54,033: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:52:54,060: INFO/job_scheduler] Job 369: RunJobThread.run
[2017-06-01 18:52:54,089: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:54,089: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:54,110: INFO/command_plan] add_jobs: created Job 370 (Test for host connectivity)
[2017-06-01 18:52:54,151: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:52:54,182: INFO/job_scheduler] Job 370: RunJobThread.run
[2017-06-01 18:52:56,051: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:52:56,130: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:52:56,328: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:52:56,407: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:52:57,514: INFO/job_scheduler] Job 368 finished 1 steps successfully
[2017-06-01 18:52:57,515: INFO/job_scheduler] Job 367 finished 1 steps successfully
[2017-06-01 18:52:57,566: INFO/job_scheduler] Job 368: complete_job: Updating cache
[2017-06-01 18:52:57,566: INFO/job_scheduler] Job 368 completing (errored=False, cancelled=False)
[2017-06-01 18:52:57,609: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:52:57,636: INFO/job_scheduler] Job 367: complete_job: Updating cache
[2017-06-01 18:52:57,636: INFO/job_scheduler] Job 367 completing (errored=False, cancelled=False)
[2017-06-01 18:52:57,696: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:52:57,733: INFO/job_scheduler] Job 370 finished 1 steps successfully
[2017-06-01 18:52:57,736: INFO/job_scheduler] Job 369 finished 1 steps successfully
[2017-06-01 18:52:57,753: INFO/job_scheduler] Job 370: complete_job: Updating cache
[2017-06-01 18:52:57,753: INFO/job_scheduler] Job 370 completing (errored=False, cancelled=False)
[2017-06-01 18:52:57,800: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:52:57,830: INFO/job_scheduler] Job 369: complete_job: Updating cache
[2017-06-01 18:52:57,830: INFO/job_scheduler] Job 369 completing (errored=False, cancelled=False)
[2017-06-01 18:52:57,896: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:52:59,856: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:59,857: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:59,876: INFO/command_plan] add_jobs: created Job 371 (Test for host connectivity)
[2017-06-01 18:52:59,905: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:52:59,909: INFO/job_scheduler] Job 371: RunJobThread.run
[2017-06-01 18:52:59,945: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:52:59,946: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:52:59,988: INFO/command_plan] add_jobs: created Job 372 (Test for host connectivity)
[2017-06-01 18:53:00,023: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:53:00,047: INFO/job_scheduler] Job 372: RunJobThread.run
[2017-06-01 18:53:00,117: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:00,118: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:00,149: INFO/command_plan] add_jobs: created Job 373 (Test for host connectivity)
[2017-06-01 18:53:00,227: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:53:00,250: INFO/job_scheduler] Job 373: RunJobThread.run
[2017-06-01 18:53:00,322: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:00,323: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:00,349: INFO/command_plan] add_jobs: created Job 374 (Test for host connectivity)
[2017-06-01 18:53:00,402: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:53:00,417: INFO/job_scheduler] Job 374: RunJobThread.run
[2017-06-01 18:53:02,226: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:53:02,392: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:53:02,536: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:53:02,648: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:53:03,572: INFO/job_scheduler] Job 371 finished 1 steps successfully
[2017-06-01 18:53:03,640: INFO/job_scheduler] Job 371: complete_job: Updating cache
[2017-06-01 18:53:03,640: INFO/job_scheduler] Job 371 completing (errored=False, cancelled=False)
[2017-06-01 18:53:03,720: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:53:03,777: INFO/job_scheduler] Job 372 finished 1 steps successfully
[2017-06-01 18:53:03,806: INFO/job_scheduler] Job 372: complete_job: Updating cache
[2017-06-01 18:53:03,807: INFO/job_scheduler] Job 372 completing (errored=False, cancelled=False)
[2017-06-01 18:53:03,858: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:53:03,921: INFO/job_scheduler] Job 373 finished 1 steps successfully
[2017-06-01 18:53:03,956: INFO/job_scheduler] Job 373: complete_job: Updating cache
[2017-06-01 18:53:03,957: INFO/job_scheduler] Job 373 completing (errored=False, cancelled=False)
[2017-06-01 18:53:04,048: INFO/job_scheduler] Job 374 finished 1 steps successfully
[2017-06-01 18:53:04,081: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:53:04,090: INFO/job_scheduler] Job 374: complete_job: Updating cache
[2017-06-01 18:53:04,090: INFO/job_scheduler] Job 374 completing (errored=False, cancelled=False)
[2017-06-01 18:53:04,165: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:53:06,034: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:06,034: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:06,049: INFO/command_plan] add_jobs: created Job 375 (Test for host connectivity)
[2017-06-01 18:53:06,093: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:06,094: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:06,113: INFO/command_plan] add_jobs: created Job 376 (Test for host connectivity)
[2017-06-01 18:53:06,128: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:53:06,155: INFO/job_scheduler] Job 376: RunJobThread.run
[2017-06-01 18:53:06,163: INFO/job_scheduler] Job 375: RunJobThread.run
[2017-06-01 18:53:06,174: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:53:06,222: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:06,222: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:06,245: INFO/command_plan] add_jobs: created Job 377 (Test for host connectivity)
[2017-06-01 18:53:06,281: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:53:06,296: INFO/job_scheduler] Job 377: RunJobThread.run
[2017-06-01 18:53:06,361: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:06,361: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:06,392: INFO/command_plan] add_jobs: created Job 378 (Test for host connectivity)
[2017-06-01 18:53:06,445: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:53:06,469: INFO/job_scheduler] Job 378: RunJobThread.run
[2017-06-01 18:53:08,421: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:53:08,454: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:53:08,635: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:53:08,720: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:53:09,789: INFO/job_scheduler] Job 376 finished 1 steps successfully
[2017-06-01 18:53:09,810: INFO/job_scheduler] Job 375 finished 1 steps successfully
[2017-06-01 18:53:09,839: INFO/job_scheduler] Job 376: complete_job: Updating cache
[2017-06-01 18:53:09,839: INFO/job_scheduler] Job 376 completing (errored=False, cancelled=False)
[2017-06-01 18:53:09,908: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:53:09,925: INFO/job_scheduler] Job 375: complete_job: Updating cache
[2017-06-01 18:53:09,925: INFO/job_scheduler] Job 375 completing (errored=False, cancelled=False)
[2017-06-01 18:53:09,954: INFO/job_scheduler] Job 377 finished 1 steps successfully
[2017-06-01 18:53:09,981: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:53:10,064: INFO/job_scheduler] Job 377: complete_job: Updating cache
[2017-06-01 18:53:10,066: INFO/job_scheduler] Job 377 completing (errored=False, cancelled=False)
[2017-06-01 18:53:10,115: INFO/job_scheduler] Job 378 finished 1 steps successfully
[2017-06-01 18:53:10,124: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:53:10,145: INFO/job_scheduler] Job 378: complete_job: Updating cache
[2017-06-01 18:53:10,145: INFO/job_scheduler] Job 378 completing (errored=False, cancelled=False)
[2017-06-01 18:53:10,232: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:53:11,541: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:11,541: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:11,556: INFO/command_plan] add_jobs: created Job 379 (Test for host connectivity)
[2017-06-01 18:53:11,607: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:11,607: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:11,638: INFO/command_plan] add_jobs: created Job 380 (Test for host connectivity)
[2017-06-01 18:53:11,648: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:53:11,655: INFO/job_scheduler] Job 379: RunJobThread.run
[2017-06-01 18:53:11,656: INFO/job_scheduler] Job 380: RunJobThread.run
[2017-06-01 18:53:11,666: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:53:11,803: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:11,805: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:11,830: INFO/command_plan] add_jobs: created Job 381 (Test for host connectivity)
[2017-06-01 18:53:11,878: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:53:11,901: INFO/job_scheduler] Job 381: RunJobThread.run
[2017-06-01 18:53:11,970: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:11,970: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:12,013: INFO/command_plan] add_jobs: created Job 382 (Test for host connectivity)
[2017-06-01 18:53:12,041: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:53:12,058: INFO/job_scheduler] Job 382: RunJobThread.run
[2017-06-01 18:53:13,962: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:53:14,047: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:53:14,311: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:53:14,373: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:53:15,402: INFO/job_scheduler] Job 380 finished 1 steps successfully
[2017-06-01 18:53:15,446: INFO/job_scheduler] Job 380: complete_job: Updating cache
[2017-06-01 18:53:15,446: INFO/job_scheduler] Job 380 completing (errored=False, cancelled=False)
[2017-06-01 18:53:15,453: INFO/job_scheduler] Job 379 finished 1 steps successfully
[2017-06-01 18:53:15,521: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:53:15,540: INFO/job_scheduler] Job 379: complete_job: Updating cache
[2017-06-01 18:53:15,540: INFO/job_scheduler] Job 379 completing (errored=False, cancelled=False)
[2017-06-01 18:53:15,650: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:53:15,762: INFO/job_scheduler] Job 381 finished 1 steps successfully
[2017-06-01 18:53:15,796: INFO/job_scheduler] Job 381: complete_job: Updating cache
[2017-06-01 18:53:15,796: INFO/job_scheduler] Job 381 completing (errored=False, cancelled=False)
[2017-06-01 18:53:15,848: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:53:15,909: INFO/job_scheduler] Job 382 finished 1 steps successfully
[2017-06-01 18:53:15,937: INFO/job_scheduler] Job 382: complete_job: Updating cache
[2017-06-01 18:53:15,937: INFO/job_scheduler] Job 382 completing (errored=False, cancelled=False)
[2017-06-01 18:53:15,997: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:53:17,696: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:17,697: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:17,716: INFO/command_plan] add_jobs: created Job 383 (Test for host connectivity)
[2017-06-01 18:53:17,774: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:17,774: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:17,796: INFO/command_plan] add_jobs: created Job 384 (Test for host connectivity)
[2017-06-01 18:53:17,806: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:53:17,809: INFO/job_scheduler] Job 384: RunJobThread.run
[2017-06-01 18:53:17,825: INFO/job_scheduler] Job 383: RunJobThread.run
[2017-06-01 18:53:17,838: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:53:17,912: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:17,912: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:17,940: INFO/command_plan] add_jobs: created Job 385 (Test for host connectivity)
[2017-06-01 18:53:17,992: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:53:18,029: INFO/job_scheduler] Job 385: RunJobThread.run
[2017-06-01 18:53:18,080: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:18,081: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:18,109: INFO/command_plan] add_jobs: created Job 386 (Test for host connectivity)
[2017-06-01 18:53:18,143: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:53:18,152: INFO/job_scheduler] Job 386: RunJobThread.run
[2017-06-01 18:53:20,101: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:53:20,252: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:53:20,366: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:53:20,415: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:53:21,500: INFO/job_scheduler] Job 384 finished 1 steps successfully
[2017-06-01 18:53:21,562: INFO/job_scheduler] Job 383 finished 1 steps successfully
[2017-06-01 18:53:21,562: INFO/job_scheduler] Job 384: complete_job: Updating cache
[2017-06-01 18:53:21,562: INFO/job_scheduler] Job 384 completing (errored=False, cancelled=False)
[2017-06-01 18:53:21,598: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:53:21,609: INFO/job_scheduler] Job 386 finished 1 steps successfully
[2017-06-01 18:53:21,625: INFO/job_scheduler] Job 383: complete_job: Updating cache
[2017-06-01 18:53:21,625: INFO/job_scheduler] Job 383 completing (errored=False, cancelled=False)
[2017-06-01 18:53:21,675: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:53:21,724: INFO/job_scheduler] Job 386: complete_job: Updating cache
[2017-06-01 18:53:21,724: INFO/job_scheduler] Job 386 completing (errored=False, cancelled=False)
[2017-06-01 18:53:21,782: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:53:21,842: INFO/job_scheduler] Job 385 finished 1 steps successfully
[2017-06-01 18:53:21,873: INFO/job_scheduler] Job 385: complete_job: Updating cache
[2017-06-01 18:53:21,873: INFO/job_scheduler] Job 385 completing (errored=False, cancelled=False)
[2017-06-01 18:53:21,917: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:53:23,765: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:23,767: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:23,782: INFO/command_plan] add_jobs: created Job 387 (Test for host connectivity)
[2017-06-01 18:53:23,843: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:23,843: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:23,872: INFO/command_plan] add_jobs: created Job 388 (Test for host connectivity)
[2017-06-01 18:53:23,881: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:53:23,896: INFO/job_scheduler] Job 387: RunJobThread.run
[2017-06-01 18:53:23,911: INFO/job_scheduler] Job 388: RunJobThread.run
[2017-06-01 18:53:23,923: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:53:23,981: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:23,982: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:24,012: INFO/command_plan] add_jobs: created Job 389 (Test for host connectivity)
[2017-06-01 18:53:24,095: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:53:24,116: INFO/job_scheduler] Job 389: RunJobThread.run
[2017-06-01 18:53:24,177: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:24,178: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:24,189: INFO/command_plan] add_jobs: created Job 390 (Test for host connectivity)
[2017-06-01 18:53:24,247: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:53:24,281: INFO/job_scheduler] Job 390: RunJobThread.run
[2017-06-01 18:53:26,145: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:53:26,185: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:53:26,442: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:53:26,531: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:53:27,516: INFO/job_scheduler] Job 388 finished 1 steps successfully
[2017-06-01 18:53:27,539: INFO/job_scheduler] Job 387 finished 1 steps successfully
[2017-06-01 18:53:27,571: INFO/job_scheduler] Job 388: complete_job: Updating cache
[2017-06-01 18:53:27,572: INFO/job_scheduler] Job 388 completing (errored=False, cancelled=False)
[2017-06-01 18:53:27,680: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:53:27,730: INFO/job_scheduler] Job 387: complete_job: Updating cache
[2017-06-01 18:53:27,730: INFO/job_scheduler] Job 387 completing (errored=False, cancelled=False)
[2017-06-01 18:53:27,792: INFO/job_scheduler] Job 389 finished 1 steps successfully
[2017-06-01 18:53:27,793: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:53:27,847: INFO/job_scheduler] Job 389: complete_job: Updating cache
[2017-06-01 18:53:27,847: INFO/job_scheduler] Job 389 completing (errored=False, cancelled=False)
[2017-06-01 18:53:27,919: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:53:27,928: INFO/job_scheduler] Job 390 finished 1 steps successfully
[2017-06-01 18:53:27,940: INFO/job_scheduler] Job 390: complete_job: Updating cache
[2017-06-01 18:53:27,940: INFO/job_scheduler] Job 390 completing (errored=False, cancelled=False)
[2017-06-01 18:53:28,015: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:53:29,831: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:29,832: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:29,847: INFO/command_plan] add_jobs: created Job 391 (Test for host connectivity)
[2017-06-01 18:53:29,890: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:29,890: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:29,930: INFO/command_plan] add_jobs: created Job 392 (Test for host connectivity)
[2017-06-01 18:53:29,939: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:53:29,954: INFO/job_scheduler] Job 392: RunJobThread.run
[2017-06-01 18:53:29,962: INFO/job_scheduler] Job 391: RunJobThread.run
[2017-06-01 18:53:29,977: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:53:30,025: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:30,026: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:30,064: INFO/command_plan] add_jobs: created Job 393 (Test for host connectivity)
[2017-06-01 18:53:30,084: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:53:30,128: INFO/job_scheduler] Job 393: RunJobThread.run
[2017-06-01 18:53:30,179: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:30,180: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:30,227: INFO/command_plan] add_jobs: created Job 394 (Test for host connectivity)
[2017-06-01 18:53:30,299: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:53:30,301: INFO/job_scheduler] Job 394: RunJobThread.run
[2017-06-01 18:53:32,404: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:53:32,550: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:53:32,633: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:53:32,652: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:53:33,729: INFO/job_scheduler] Job 392 finished 1 steps successfully
[2017-06-01 18:53:33,787: INFO/job_scheduler] Job 392: complete_job: Updating cache
[2017-06-01 18:53:33,788: INFO/job_scheduler] Job 392 completing (errored=False, cancelled=False)
[2017-06-01 18:53:33,879: INFO/job_scheduler] Job 391 finished 1 steps successfully
[2017-06-01 18:53:33,979: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:53:33,997: INFO/job_scheduler] Job 394 finished 1 steps successfully
[2017-06-01 18:53:34,021: INFO/job_scheduler] Job 391: complete_job: Updating cache
[2017-06-01 18:53:34,021: INFO/job_scheduler] Job 391 completing (errored=False, cancelled=False)
[2017-06-01 18:53:34,086: INFO/job_scheduler] Job 393 finished 1 steps successfully
[2017-06-01 18:53:34,086: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:53:34,104: INFO/job_scheduler] Job 394: complete_job: Updating cache
[2017-06-01 18:53:34,104: INFO/job_scheduler] Job 394 completing (errored=False, cancelled=False)
[2017-06-01 18:53:34,166: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:53:34,191: INFO/job_scheduler] Job 393: complete_job: Updating cache
[2017-06-01 18:53:34,191: INFO/job_scheduler] Job 393 completing (errored=False, cancelled=False)
[2017-06-01 18:53:34,237: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:53:36,591: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:36,593: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:36,620: INFO/command_plan] add_jobs: created Job 395 (Test for host connectivity)
[2017-06-01 18:53:36,656: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:53:36,660: INFO/job_scheduler] Job 395: RunJobThread.run
[2017-06-01 18:53:36,720: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:36,721: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:36,736: INFO/command_plan] add_jobs: created Job 396 (Test for host connectivity)
[2017-06-01 18:53:36,832: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:36,834: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:36,855: INFO/command_plan] add_jobs: created Job 397 (Test for host connectivity)
[2017-06-01 18:53:36,873: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 18:53:36,887: INFO/job_scheduler] Job 396: RunJobThread.run
[2017-06-01 18:53:36,909: INFO/job_scheduler] Job 397: RunJobThread.run
[2017-06-01 18:53:36,944: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:36,945: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:36,973: INFO/command_plan] add_jobs: created Job 398 (Test for host connectivity)
[2017-06-01 18:53:36,998: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:53:37,024: INFO/job_scheduler] Job 398: RunJobThread.run
[2017-06-01 18:53:37,081: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 18:53:38,969: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:53:39,187: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:53:39,420: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:53:39,426: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:53:40,295: INFO/job_scheduler] Job 395 finished 1 steps successfully
[2017-06-01 18:53:40,328: INFO/job_scheduler] Job 395: complete_job: Updating cache
[2017-06-01 18:53:40,328: INFO/job_scheduler] Job 395 completing (errored=False, cancelled=False)
[2017-06-01 18:53:40,373: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:53:40,572: INFO/job_scheduler] Job 396 finished 1 steps successfully
[2017-06-01 18:53:40,628: INFO/job_scheduler] Job 396: complete_job: Updating cache
[2017-06-01 18:53:40,628: INFO/job_scheduler] Job 396 completing (errored=False, cancelled=False)
[2017-06-01 18:53:40,670: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:53:40,804: INFO/job_scheduler] Job 398 finished 1 steps successfully
[2017-06-01 18:53:40,852: INFO/job_scheduler] Job 398: complete_job: Updating cache
[2017-06-01 18:53:40,852: INFO/job_scheduler] Job 398 completing (errored=False, cancelled=False)
[2017-06-01 18:53:40,855: INFO/job_scheduler] Job 397 finished 1 steps successfully
[2017-06-01 18:53:40,921: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:53:41,044: INFO/job_scheduler] Job 397: complete_job: Updating cache
[2017-06-01 18:53:41,044: INFO/job_scheduler] Job 397 completing (errored=False, cancelled=False)
[2017-06-01 18:53:41,086: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:53:42,641: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:42,643: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:42,652: INFO/command_plan] add_jobs: created Job 399 (Test for host connectivity)
[2017-06-01 18:53:42,720: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:42,721: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:42,738: INFO/command_plan] add_jobs: created Job 400 (Test for host connectivity)
[2017-06-01 18:53:42,755: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:53:42,760: INFO/job_scheduler] Job 400: RunJobThread.run
[2017-06-01 18:53:42,760: INFO/job_scheduler] Job 399: RunJobThread.run
[2017-06-01 18:53:42,771: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:53:42,802: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:42,802: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:42,829: INFO/command_plan] add_jobs: created Job 401 (Test for host connectivity)
[2017-06-01 18:53:42,861: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:53:42,888: INFO/job_scheduler] Job 401: RunJobThread.run
[2017-06-01 18:53:42,979: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:42,979: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:43,019: INFO/command_plan] add_jobs: created Job 402 (Test for host connectivity)
[2017-06-01 18:53:43,086: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:53:43,097: INFO/job_scheduler] Job 402: RunJobThread.run
[2017-06-01 18:53:45,000: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:53:45,032: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:53:45,180: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:53:45,275: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:53:46,356: INFO/job_scheduler] Job 399 finished 1 steps successfully
[2017-06-01 18:53:46,360: INFO/job_scheduler] Job 400 finished 1 steps successfully
[2017-06-01 18:53:46,410: INFO/job_scheduler] Job 399: complete_job: Updating cache
[2017-06-01 18:53:46,410: INFO/job_scheduler] Job 399 completing (errored=False, cancelled=False)
[2017-06-01 18:53:46,479: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:53:46,493: INFO/job_scheduler] Job 400: complete_job: Updating cache
[2017-06-01 18:53:46,493: INFO/job_scheduler] Job 400 completing (errored=False, cancelled=False)
[2017-06-01 18:53:46,560: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:53:46,626: INFO/job_scheduler] Job 401 finished 1 steps successfully
[2017-06-01 18:53:46,652: INFO/job_scheduler] Job 401: complete_job: Updating cache
[2017-06-01 18:53:46,652: INFO/job_scheduler] Job 401 completing (errored=False, cancelled=False)
[2017-06-01 18:53:46,701: INFO/job_scheduler] Job 402 finished 1 steps successfully
[2017-06-01 18:53:46,712: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:53:46,732: INFO/job_scheduler] Job 402: complete_job: Updating cache
[2017-06-01 18:53:46,732: INFO/job_scheduler] Job 402 completing (errored=False, cancelled=False)
[2017-06-01 18:53:46,851: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:53:48,733: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:48,734: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:48,751: INFO/command_plan] add_jobs: created Job 403 (Test for host connectivity)
[2017-06-01 18:53:48,769: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:53:48,776: INFO/job_scheduler] Job 403: RunJobThread.run
[2017-06-01 18:53:48,850: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:48,853: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:48,869: INFO/command_plan] add_jobs: created Job 404 (Test for host connectivity)
[2017-06-01 18:53:48,958: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:48,959: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:48,977: INFO/command_plan] add_jobs: created Job 405 (Test for host connectivity)
[2017-06-01 18:53:48,997: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 18:53:49,022: INFO/job_scheduler] Job 404: RunJobThread.run
[2017-06-01 18:53:49,023: INFO/job_scheduler] Job 405: RunJobThread.run
[2017-06-01 18:53:49,028: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:53:49,098: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:49,106: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:49,133: INFO/command_plan] add_jobs: created Job 406 (Test for host connectivity)
[2017-06-01 18:53:49,206: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:53:49,234: INFO/job_scheduler] Job 406: RunJobThread.run
[2017-06-01 18:53:51,066: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:53:51,389: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:53:51,582: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:53:51,637: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:53:52,410: INFO/job_scheduler] Job 403 finished 1 steps successfully
[2017-06-01 18:53:52,450: INFO/job_scheduler] Job 403: complete_job: Updating cache
[2017-06-01 18:53:52,450: INFO/job_scheduler] Job 403 completing (errored=False, cancelled=False)
[2017-06-01 18:53:52,542: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:53:52,728: INFO/job_scheduler] Job 404 finished 1 steps successfully
[2017-06-01 18:53:52,800: INFO/job_scheduler] Job 404: complete_job: Updating cache
[2017-06-01 18:53:52,800: INFO/job_scheduler] Job 404 completing (errored=False, cancelled=False)
[2017-06-01 18:53:52,868: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:53:53,011: INFO/job_scheduler] Job 405 finished 1 steps successfully
[2017-06-01 18:53:53,038: INFO/job_scheduler] Job 406 finished 1 steps successfully
[2017-06-01 18:53:53,067: INFO/job_scheduler] Job 405: complete_job: Updating cache
[2017-06-01 18:53:53,067: INFO/job_scheduler] Job 405 completing (errored=False, cancelled=False)
[2017-06-01 18:53:53,133: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:53:53,258: INFO/job_scheduler] Job 406: complete_job: Updating cache
[2017-06-01 18:53:53,258: INFO/job_scheduler] Job 406 completing (errored=False, cancelled=False)
[2017-06-01 18:53:53,315: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:53:55,260: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:55,260: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:55,324: INFO/command_plan] add_jobs: created Job 407 (Test for host connectivity)
[2017-06-01 18:53:55,359: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:55,359: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:55,402: INFO/command_plan] add_jobs: created Job 408 (Test for host connectivity)
[2017-06-01 18:53:55,416: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:53:55,429: INFO/job_scheduler] Job 408: RunJobThread.run
[2017-06-01 18:53:55,430: INFO/job_scheduler] Job 407: RunJobThread.run
[2017-06-01 18:53:55,443: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:53:55,510: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:55,511: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:55,558: INFO/command_plan] add_jobs: created Job 409 (Test for host connectivity)
[2017-06-01 18:53:55,647: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:53:55,652: INFO/job_scheduler] Job 409: RunJobThread.run
[2017-06-01 18:53:55,727: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:53:55,728: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:53:55,757: INFO/command_plan] add_jobs: created Job 410 (Test for host connectivity)
[2017-06-01 18:53:55,790: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:53:55,798: INFO/job_scheduler] Job 410: RunJobThread.run
[2017-06-01 18:53:57,703: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:53:57,764: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:53:57,902: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:53:57,937: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:53:59,016: INFO/job_scheduler] Job 407 finished 1 steps successfully
[2017-06-01 18:53:59,141: INFO/job_scheduler] Job 407: complete_job: Updating cache
[2017-06-01 18:53:59,141: INFO/job_scheduler] Job 407 completing (errored=False, cancelled=False)
[2017-06-01 18:53:59,182: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:53:59,199: INFO/job_scheduler] Job 408 finished 1 steps successfully
[2017-06-01 18:53:59,217: INFO/job_scheduler] Job 408: complete_job: Updating cache
[2017-06-01 18:53:59,218: INFO/job_scheduler] Job 408 completing (errored=False, cancelled=False)
[2017-06-01 18:53:59,265: INFO/job_scheduler] Job 410 finished 1 steps successfully
[2017-06-01 18:53:59,293: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:53:59,319: INFO/job_scheduler] Job 409 finished 1 steps successfully
[2017-06-01 18:53:59,336: INFO/job_scheduler] Job 410: complete_job: Updating cache
[2017-06-01 18:53:59,336: INFO/job_scheduler] Job 410 completing (errored=False, cancelled=False)
[2017-06-01 18:53:59,372: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:53:59,391: INFO/job_scheduler] Job 409: complete_job: Updating cache
[2017-06-01 18:53:59,392: INFO/job_scheduler] Job 409 completing (errored=False, cancelled=False)
[2017-06-01 18:53:59,434: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:54:01,412: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:01,414: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:01,423: INFO/command_plan] add_jobs: created Job 411 (Test for host connectivity)
[2017-06-01 18:54:01,454: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:54:01,456: INFO/job_scheduler] Job 411: RunJobThread.run
[2017-06-01 18:54:01,522: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:01,522: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:01,541: INFO/command_plan] add_jobs: created Job 412 (Test for host connectivity)
[2017-06-01 18:54:01,603: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:54:01,607: INFO/job_scheduler] Job 412: RunJobThread.run
[2017-06-01 18:54:01,663: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:01,665: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:01,765: INFO/command_plan] add_jobs: created Job 413 (Test for host connectivity)
[2017-06-01 18:54:01,874: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:54:01,876: INFO/job_scheduler] Job 413: RunJobThread.run
[2017-06-01 18:54:01,898: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:01,901: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:02,040: INFO/command_plan] add_jobs: created Job 414 (Test for host connectivity)
[2017-06-01 18:54:02,066: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:54:02,070: INFO/job_scheduler] Job 414: RunJobThread.run
[2017-06-01 18:54:03,864: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:54:03,916: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:54:04,174: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:54:04,306: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:54:05,303: INFO/job_scheduler] Job 411 finished 1 steps successfully
[2017-06-01 18:54:05,348: INFO/job_scheduler] Job 411: complete_job: Updating cache
[2017-06-01 18:54:05,348: INFO/job_scheduler] Job 411 completing (errored=False, cancelled=False)
[2017-06-01 18:54:05,365: INFO/job_scheduler] Job 412 finished 1 steps successfully
[2017-06-01 18:54:05,416: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:54:05,432: INFO/job_scheduler] Job 412: complete_job: Updating cache
[2017-06-01 18:54:05,433: INFO/job_scheduler] Job 412 completing (errored=False, cancelled=False)
[2017-06-01 18:54:05,516: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:54:05,596: INFO/job_scheduler] Job 413 finished 1 steps successfully
[2017-06-01 18:54:05,628: INFO/job_scheduler] Job 414 finished 1 steps successfully
[2017-06-01 18:54:05,648: INFO/job_scheduler] Job 413: complete_job: Updating cache
[2017-06-01 18:54:05,648: INFO/job_scheduler] Job 413 completing (errored=False, cancelled=False)
[2017-06-01 18:54:05,707: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:54:05,722: INFO/job_scheduler] Job 414: complete_job: Updating cache
[2017-06-01 18:54:05,723: INFO/job_scheduler] Job 414 completing (errored=False, cancelled=False)
[2017-06-01 18:54:05,817: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:54:07,772: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:07,772: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:07,791: INFO/command_plan] add_jobs: created Job 415 (Test for host connectivity)
[2017-06-01 18:54:07,819: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:54:07,821: INFO/job_scheduler] Job 415: RunJobThread.run
[2017-06-01 18:54:07,854: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:07,854: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:07,881: INFO/command_plan] add_jobs: created Job 416 (Test for host connectivity)
[2017-06-01 18:54:07,902: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:54:07,914: INFO/job_scheduler] Job 416: RunJobThread.run
[2017-06-01 18:54:07,953: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:07,953: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:08,014: INFO/command_plan] add_jobs: created Job 417 (Test for host connectivity)
[2017-06-01 18:54:08,095: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:54:08,100: INFO/job_scheduler] Job 417: RunJobThread.run
[2017-06-01 18:54:08,149: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:08,150: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:08,180: INFO/command_plan] add_jobs: created Job 418 (Test for host connectivity)
[2017-06-01 18:54:08,278: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:54:08,282: INFO/job_scheduler] Job 418: RunJobThread.run
[2017-06-01 18:54:10,182: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:54:10,209: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:54:10,326: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:54:10,478: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:54:11,435: INFO/job_scheduler] Job 415 finished 1 steps successfully
[2017-06-01 18:54:11,480: INFO/job_scheduler] Job 415: complete_job: Updating cache
[2017-06-01 18:54:11,480: INFO/job_scheduler] Job 415 completing (errored=False, cancelled=False)
[2017-06-01 18:54:11,543: INFO/job_scheduler] Job 416 finished 1 steps successfully
[2017-06-01 18:54:11,565: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:54:11,581: INFO/job_scheduler] Job 416: complete_job: Updating cache
[2017-06-01 18:54:11,588: INFO/job_scheduler] Job 416 completing (errored=False, cancelled=False)
[2017-06-01 18:54:11,681: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:54:11,745: INFO/job_scheduler] Job 417 finished 1 steps successfully
[2017-06-01 18:54:11,765: INFO/job_scheduler] Job 417: complete_job: Updating cache
[2017-06-01 18:54:11,765: INFO/job_scheduler] Job 417 completing (errored=False, cancelled=False)
[2017-06-01 18:54:11,801: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:54:11,841: INFO/job_scheduler] Job 418 finished 1 steps successfully
[2017-06-01 18:54:11,898: INFO/job_scheduler] Job 418: complete_job: Updating cache
[2017-06-01 18:54:11,899: INFO/job_scheduler] Job 418 completing (errored=False, cancelled=False)
[2017-06-01 18:54:11,941: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:54:13,855: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:13,857: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:13,878: INFO/command_plan] add_jobs: created Job 419 (Test for host connectivity)
[2017-06-01 18:54:13,919: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:54:13,955: INFO/job_scheduler] Job 419: RunJobThread.run
[2017-06-01 18:54:13,991: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:13,991: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:14,015: INFO/command_plan] add_jobs: created Job 420 (Test for host connectivity)
[2017-06-01 18:54:14,064: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:54:14,068: INFO/job_scheduler] Job 420: RunJobThread.run
[2017-06-01 18:54:14,141: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:14,141: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:14,178: INFO/command_plan] add_jobs: created Job 421 (Test for host connectivity)
[2017-06-01 18:54:14,246: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:54:14,252: INFO/job_scheduler] Job 421: RunJobThread.run
[2017-06-01 18:54:14,326: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:14,330: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:14,350: INFO/command_plan] add_jobs: created Job 422 (Test for host connectivity)
[2017-06-01 18:54:14,386: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:54:14,394: INFO/job_scheduler] Job 422: RunJobThread.run
[2017-06-01 18:54:16,221: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:54:16,396: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:54:16,491: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:54:16,555: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:54:17,546: INFO/job_scheduler] Job 419 finished 1 steps successfully
[2017-06-01 18:54:17,614: INFO/job_scheduler] Job 419: complete_job: Updating cache
[2017-06-01 18:54:17,614: INFO/job_scheduler] Job 419 completing (errored=False, cancelled=False)
[2017-06-01 18:54:17,698: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:54:17,730: INFO/job_scheduler] Job 420 finished 1 steps successfully
[2017-06-01 18:54:17,756: INFO/job_scheduler] Job 420: complete_job: Updating cache
[2017-06-01 18:54:17,756: INFO/job_scheduler] Job 420 completing (errored=False, cancelled=False)
[2017-06-01 18:54:17,815: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:54:17,888: INFO/job_scheduler] Job 421 finished 1 steps successfully
[2017-06-01 18:54:17,922: INFO/job_scheduler] Job 421: complete_job: Updating cache
[2017-06-01 18:54:17,922: INFO/job_scheduler] Job 421 completing (errored=False, cancelled=False)
[2017-06-01 18:54:17,982: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:54:17,996: INFO/job_scheduler] Job 422 finished 1 steps successfully
[2017-06-01 18:54:18,032: INFO/job_scheduler] Job 422: complete_job: Updating cache
[2017-06-01 18:54:18,032: INFO/job_scheduler] Job 422 completing (errored=False, cancelled=False)
[2017-06-01 18:54:18,083: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:54:20,062: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:20,063: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:20,073: INFO/command_plan] add_jobs: created Job 423 (Test for host connectivity)
[2017-06-01 18:54:20,095: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:54:20,122: INFO/job_scheduler] Job 423: RunJobThread.run
[2017-06-01 18:54:20,188: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:20,190: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:20,208: INFO/command_plan] add_jobs: created Job 424 (Test for host connectivity)
[2017-06-01 18:54:20,309: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:20,310: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:20,323: INFO/command_plan] add_jobs: created Job 425 (Test for host connectivity)
[2017-06-01 18:54:20,350: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 18:54:20,376: INFO/job_scheduler] Job 424: RunJobThread.run
[2017-06-01 18:54:20,393: INFO/job_scheduler] Job 425: RunJobThread.run
[2017-06-01 18:54:20,421: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:20,424: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:20,442: INFO/command_plan] add_jobs: created Job 426 (Test for host connectivity)
[2017-06-01 18:54:20,472: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:54:20,528: INFO/job_scheduler] Job 426: RunJobThread.run
[2017-06-01 18:54:20,655: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 18:54:22,339: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:54:22,764: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:54:22,794: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:54:22,827: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:54:23,682: INFO/job_scheduler] Job 423 finished 1 steps successfully
[2017-06-01 18:54:23,730: INFO/job_scheduler] Job 423: complete_job: Updating cache
[2017-06-01 18:54:23,730: INFO/job_scheduler] Job 423 completing (errored=False, cancelled=False)
[2017-06-01 18:54:23,779: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:54:24,303: INFO/job_scheduler] Job 424 finished 1 steps successfully
[2017-06-01 18:54:24,311: INFO/job_scheduler] Job 425 finished 1 steps successfully
[2017-06-01 18:54:24,320: INFO/job_scheduler] Job 426 finished 1 steps successfully
[2017-06-01 18:54:24,338: INFO/job_scheduler] Job 424: complete_job: Updating cache
[2017-06-01 18:54:24,338: INFO/job_scheduler] Job 424 completing (errored=False, cancelled=False)
[2017-06-01 18:54:24,399: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:54:24,421: INFO/job_scheduler] Job 425: complete_job: Updating cache
[2017-06-01 18:54:24,422: INFO/job_scheduler] Job 425 completing (errored=False, cancelled=False)
[2017-06-01 18:54:24,504: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:54:24,513: INFO/job_scheduler] Job 426: complete_job: Updating cache
[2017-06-01 18:54:24,513: INFO/job_scheduler] Job 426 completing (errored=False, cancelled=False)
[2017-06-01 18:54:24,596: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:54:26,286: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:26,287: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:26,305: INFO/command_plan] add_jobs: created Job 427 (Test for host connectivity)
[2017-06-01 18:54:26,314: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:54:26,316: INFO/job_scheduler] Job 427: RunJobThread.run
[2017-06-01 18:54:26,404: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:26,407: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:26,445: INFO/command_plan] add_jobs: created Job 428 (Test for host connectivity)
[2017-06-01 18:54:26,503: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:54:26,510: INFO/job_scheduler] Job 428: RunJobThread.run
[2017-06-01 18:54:26,534: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:26,535: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:26,551: INFO/command_plan] add_jobs: created Job 429 (Test for host connectivity)
[2017-06-01 18:54:26,584: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:54:26,594: INFO/job_scheduler] Job 429: RunJobThread.run
[2017-06-01 18:54:26,632: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:26,633: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:26,671: INFO/command_plan] add_jobs: created Job 430 (Test for host connectivity)
[2017-06-01 18:54:26,789: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:54:26,794: INFO/job_scheduler] Job 430: RunJobThread.run
[2017-06-01 18:54:28,591: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:54:28,770: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:54:28,960: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:54:29,096: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:54:29,977: INFO/job_scheduler] Job 427 finished 1 steps successfully
[2017-06-01 18:54:30,020: INFO/job_scheduler] Job 427: complete_job: Updating cache
[2017-06-01 18:54:30,020: INFO/job_scheduler] Job 427 completing (errored=False, cancelled=False)
[2017-06-01 18:54:30,054: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:54:30,119: INFO/job_scheduler] Job 428 finished 1 steps successfully
[2017-06-01 18:54:30,162: INFO/job_scheduler] Job 428: complete_job: Updating cache
[2017-06-01 18:54:30,162: INFO/job_scheduler] Job 428 completing (errored=False, cancelled=False)
[2017-06-01 18:54:30,234: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:54:30,256: INFO/job_scheduler] Job 430 finished 1 steps successfully
[2017-06-01 18:54:30,286: INFO/job_scheduler] Job 430: complete_job: Updating cache
[2017-06-01 18:54:30,287: INFO/job_scheduler] Job 430 completing (errored=False, cancelled=False)
[2017-06-01 18:54:30,337: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:54:30,543: INFO/job_scheduler] Job 429 finished 1 steps successfully
[2017-06-01 18:54:30,586: INFO/job_scheduler] Job 429: complete_job: Updating cache
[2017-06-01 18:54:30,586: INFO/job_scheduler] Job 429 completing (errored=False, cancelled=False)
[2017-06-01 18:54:30,638: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:54:32,342: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:32,344: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:32,386: INFO/command_plan] add_jobs: created Job 431 (Test for host connectivity)
[2017-06-01 18:54:32,411: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:54:32,414: INFO/job_scheduler] Job 431: RunJobThread.run
[2017-06-01 18:54:32,447: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:32,448: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:32,486: INFO/command_plan] add_jobs: created Job 432 (Test for host connectivity)
[2017-06-01 18:54:32,511: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:54:32,550: INFO/job_scheduler] Job 432: RunJobThread.run
[2017-06-01 18:54:32,592: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:32,592: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:32,613: INFO/command_plan] add_jobs: created Job 433 (Test for host connectivity)
[2017-06-01 18:54:32,665: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:54:32,680: INFO/job_scheduler] Job 433: RunJobThread.run
[2017-06-01 18:54:32,748: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:32,748: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:32,772: INFO/command_plan] add_jobs: created Job 434 (Test for host connectivity)
[2017-06-01 18:54:32,814: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:54:32,824: INFO/job_scheduler] Job 434: RunJobThread.run
[2017-06-01 18:54:34,726: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:54:34,742: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:54:34,929: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:54:35,071: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:54:36,084: INFO/job_scheduler] Job 432 finished 1 steps successfully
[2017-06-01 18:54:36,091: INFO/job_scheduler] Job 431 finished 1 steps successfully
[2017-06-01 18:54:36,119: INFO/job_scheduler] Job 432: complete_job: Updating cache
[2017-06-01 18:54:36,119: INFO/job_scheduler] Job 432 completing (errored=False, cancelled=False)
[2017-06-01 18:54:36,179: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:54:36,186: INFO/job_scheduler] Job 431: complete_job: Updating cache
[2017-06-01 18:54:36,186: INFO/job_scheduler] Job 431 completing (errored=False, cancelled=False)
[2017-06-01 18:54:36,318: INFO/job_scheduler] Job 433 finished 1 steps successfully
[2017-06-01 18:54:36,344: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:54:36,372: INFO/job_scheduler] Job 433: complete_job: Updating cache
[2017-06-01 18:54:36,372: INFO/job_scheduler] Job 433 completing (errored=False, cancelled=False)
[2017-06-01 18:54:36,443: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:54:36,616: INFO/job_scheduler] Job 434 finished 1 steps successfully
[2017-06-01 18:54:36,643: INFO/job_scheduler] Job 434: complete_job: Updating cache
[2017-06-01 18:54:36,644: INFO/job_scheduler] Job 434 completing (errored=False, cancelled=False)
[2017-06-01 18:54:36,669: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:54:37,912: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:37,913: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:37,927: INFO/command_plan] add_jobs: created Job 435 (Test for host connectivity)
[2017-06-01 18:54:37,986: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:54:37,989: INFO/job_scheduler] Job 435: RunJobThread.run
[2017-06-01 18:54:38,030: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:38,034: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:38,052: INFO/command_plan] add_jobs: created Job 436 (Test for host connectivity)
[2017-06-01 18:54:38,112: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:38,113: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:38,160: INFO/command_plan] add_jobs: created Job 437 (Test for host connectivity)
[2017-06-01 18:54:38,170: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 18:54:38,178: INFO/job_scheduler] Job 436: RunJobThread.run
[2017-06-01 18:54:38,228: INFO/job_scheduler] Job 437: RunJobThread.run
[2017-06-01 18:54:38,263: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:38,264: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:38,287: INFO/command_plan] add_jobs: created Job 438 (Test for host connectivity)
[2017-06-01 18:54:38,303: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:54:38,351: INFO/job_scheduler] Job 438: RunJobThread.run
[2017-06-01 18:54:38,443: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 18:54:40,229: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:54:40,542: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:54:40,636: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:54:40,674: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:54:41,593: INFO/job_scheduler] Job 435 finished 1 steps successfully
[2017-06-01 18:54:41,626: INFO/job_scheduler] Job 435: complete_job: Updating cache
[2017-06-01 18:54:41,626: INFO/job_scheduler] Job 435 completing (errored=False, cancelled=False)
[2017-06-01 18:54:41,693: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:54:41,877: INFO/job_scheduler] Job 436 finished 1 steps successfully
[2017-06-01 18:54:41,901: INFO/job_scheduler] Job 436: complete_job: Updating cache
[2017-06-01 18:54:41,901: INFO/job_scheduler] Job 436 completing (errored=False, cancelled=False)
[2017-06-01 18:54:41,973: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:54:42,093: INFO/job_scheduler] Job 437 finished 1 steps successfully
[2017-06-01 18:54:42,151: INFO/job_scheduler] Job 437: complete_job: Updating cache
[2017-06-01 18:54:42,151: INFO/job_scheduler] Job 437 completing (errored=False, cancelled=False)
[2017-06-01 18:54:42,173: INFO/job_scheduler] Job 438 finished 1 steps successfully
[2017-06-01 18:54:42,210: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:54:42,226: INFO/job_scheduler] Job 438: complete_job: Updating cache
[2017-06-01 18:54:42,226: INFO/job_scheduler] Job 438 completing (errored=False, cancelled=False)
[2017-06-01 18:54:42,261: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:54:43,973: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:43,974: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:43,985: INFO/command_plan] add_jobs: created Job 439 (Test for host connectivity)
[2017-06-01 18:54:44,038: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:44,039: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:44,073: INFO/command_plan] add_jobs: created Job 440 (Test for host connectivity)
[2017-06-01 18:54:44,087: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:54:44,107: INFO/job_scheduler] Job 440: RunJobThread.run
[2017-06-01 18:54:44,108: INFO/job_scheduler] Job 439: RunJobThread.run
[2017-06-01 18:54:44,121: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:54:44,184: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:44,185: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:44,201: INFO/command_plan] add_jobs: created Job 441 (Test for host connectivity)
[2017-06-01 18:54:44,261: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:44,261: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:44,326: INFO/command_plan] add_jobs: created Job 442 (Test for host connectivity)
[2017-06-01 18:54:44,376: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 18:54:44,404: INFO/job_scheduler] Job 441: RunJobThread.run
[2017-06-01 18:54:44,405: INFO/job_scheduler] Job 442: RunJobThread.run
[2017-06-01 18:54:44,410: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 18:54:46,369: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:54:46,419: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:54:46,665: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:54:46,717: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:54:47,753: INFO/job_scheduler] Job 439 finished 1 steps successfully
[2017-06-01 18:54:47,800: INFO/job_scheduler] Job 439: complete_job: Updating cache
[2017-06-01 18:54:47,800: INFO/job_scheduler] Job 439 completing (errored=False, cancelled=False)
[2017-06-01 18:54:47,824: INFO/job_scheduler] Job 440 finished 1 steps successfully
[2017-06-01 18:54:47,864: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:54:47,897: INFO/job_scheduler] Job 440: complete_job: Updating cache
[2017-06-01 18:54:47,901: INFO/job_scheduler] Job 440 completing (errored=False, cancelled=False)
[2017-06-01 18:54:47,981: INFO/job_scheduler] Job 442 finished 1 steps successfully
[2017-06-01 18:54:48,025: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:54:48,050: INFO/job_scheduler] Job 442: complete_job: Updating cache
[2017-06-01 18:54:48,050: INFO/job_scheduler] Job 442 completing (errored=False, cancelled=False)
[2017-06-01 18:54:48,124: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:54:48,239: INFO/job_scheduler] Job 441 finished 1 steps successfully
[2017-06-01 18:54:48,275: INFO/job_scheduler] Job 441: complete_job: Updating cache
[2017-06-01 18:54:48,275: INFO/job_scheduler] Job 441 completing (errored=False, cancelled=False)
[2017-06-01 18:54:48,310: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:54:50,127: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:50,128: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:50,141: INFO/command_plan] add_jobs: created Job 443 (Test for host connectivity)
[2017-06-01 18:54:50,179: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:54:50,185: INFO/job_scheduler] Job 443: RunJobThread.run
[2017-06-01 18:54:50,247: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:50,248: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:50,268: INFO/command_plan] add_jobs: created Job 444 (Test for host connectivity)
[2017-06-01 18:54:50,361: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:54:50,365: INFO/job_scheduler] Job 444: RunJobThread.run
[2017-06-01 18:54:50,415: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:50,415: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:50,441: INFO/command_plan] add_jobs: created Job 445 (Test for host connectivity)
[2017-06-01 18:54:50,481: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:54:50,501: INFO/job_scheduler] Job 445: RunJobThread.run
[2017-06-01 18:54:50,547: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:50,548: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:50,566: INFO/command_plan] add_jobs: created Job 446 (Test for host connectivity)
[2017-06-01 18:54:50,601: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:54:50,603: INFO/job_scheduler] Job 446: RunJobThread.run
[2017-06-01 18:54:52,501: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:54:52,597: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:54:52,699: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:54:52,795: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:54:53,819: INFO/job_scheduler] Job 443 finished 1 steps successfully
[2017-06-01 18:54:53,874: INFO/job_scheduler] Job 443: complete_job: Updating cache
[2017-06-01 18:54:53,874: INFO/job_scheduler] Job 443 completing (errored=False, cancelled=False)
[2017-06-01 18:54:53,915: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:54:53,963: INFO/job_scheduler] Job 444 finished 1 steps successfully
[2017-06-01 18:54:53,991: INFO/job_scheduler] Job 444: complete_job: Updating cache
[2017-06-01 18:54:53,991: INFO/job_scheduler] Job 444 completing (errored=False, cancelled=False)
[2017-06-01 18:54:54,047: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:54:54,065: INFO/job_scheduler] Job 445 finished 1 steps successfully
[2017-06-01 18:54:54,087: INFO/job_scheduler] Job 445: complete_job: Updating cache
[2017-06-01 18:54:54,087: INFO/job_scheduler] Job 445 completing (errored=False, cancelled=False)
[2017-06-01 18:54:54,096: INFO/job_scheduler] Job 446 finished 1 steps successfully
[2017-06-01 18:54:54,147: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:54:54,195: INFO/job_scheduler] Job 446: complete_job: Updating cache
[2017-06-01 18:54:54,195: INFO/job_scheduler] Job 446 completing (errored=False, cancelled=False)
[2017-06-01 18:54:54,242: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:54:56,325: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:56,325: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:56,337: INFO/command_plan] add_jobs: created Job 447 (Test for host connectivity)
[2017-06-01 18:54:56,358: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:54:56,361: INFO/job_scheduler] Job 447: RunJobThread.run
[2017-06-01 18:54:56,393: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:56,394: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:56,504: INFO/command_plan] add_jobs: created Job 448 (Test for host connectivity)
[2017-06-01 18:54:56,608: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:56,609: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:56,621: INFO/command_plan] add_jobs: created Job 449 (Test for host connectivity)
[2017-06-01 18:54:56,642: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 18:54:56,644: INFO/job_scheduler] Job 448: RunJobThread.run
[2017-06-01 18:54:56,654: INFO/job_scheduler] Job 449: RunJobThread.run
[2017-06-01 18:54:56,663: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:54:56,745: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:54:56,746: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:54:56,773: INFO/command_plan] add_jobs: created Job 450 (Test for host connectivity)
[2017-06-01 18:54:56,798: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:54:56,853: INFO/job_scheduler] Job 450: RunJobThread.run
[2017-06-01 18:54:58,685: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:54:58,936: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:54:59,017: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:54:59,060: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:55:00,003: INFO/job_scheduler] Job 447 finished 1 steps successfully
[2017-06-01 18:55:00,092: INFO/job_scheduler] Job 447: complete_job: Updating cache
[2017-06-01 18:55:00,092: INFO/job_scheduler] Job 447 completing (errored=False, cancelled=False)
[2017-06-01 18:55:00,177: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:55:00,273: INFO/job_scheduler] Job 448 finished 1 steps successfully
[2017-06-01 18:55:00,331: INFO/job_scheduler] Job 448: complete_job: Updating cache
[2017-06-01 18:55:00,331: INFO/job_scheduler] Job 448 completing (errored=False, cancelled=False)
[2017-06-01 18:55:00,366: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:55:00,462: INFO/job_scheduler] Job 449 finished 1 steps successfully
[2017-06-01 18:55:00,514: INFO/job_scheduler] Job 449: complete_job: Updating cache
[2017-06-01 18:55:00,514: INFO/job_scheduler] Job 449 completing (errored=False, cancelled=False)
[2017-06-01 18:55:00,543: INFO/job_scheduler] Job 450 finished 1 steps successfully
[2017-06-01 18:55:00,560: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:55:00,573: INFO/job_scheduler] Job 450: complete_job: Updating cache
[2017-06-01 18:55:00,573: INFO/job_scheduler] Job 450 completing (errored=False, cancelled=False)
[2017-06-01 18:55:00,627: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:55:02,473: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:02,475: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:02,491: INFO/command_plan] add_jobs: created Job 451 (Test for host connectivity)
[2017-06-01 18:55:02,532: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:02,557: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:02,564: INFO/command_plan] add_jobs: created Job 452 (Test for host connectivity)
[2017-06-01 18:55:02,581: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:55:02,611: INFO/job_scheduler] Job 451: RunJobThread.run
[2017-06-01 18:55:02,612: INFO/job_scheduler] Job 452: RunJobThread.run
[2017-06-01 18:55:02,622: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:55:02,696: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:02,697: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:02,741: INFO/command_plan] add_jobs: created Job 453 (Test for host connectivity)
[2017-06-01 18:55:02,778: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:55:02,810: INFO/job_scheduler] Job 453: RunJobThread.run
[2017-06-01 18:55:02,851: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:02,851: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:02,968: INFO/command_plan] add_jobs: created Job 454 (Test for host connectivity)
[2017-06-01 18:55:03,022: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:55:03,025: INFO/job_scheduler] Job 454: RunJobThread.run
[2017-06-01 18:55:04,920: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:55:04,925: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:55:05,130: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:55:05,330: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:55:06,286: INFO/job_scheduler] Job 452 finished 1 steps successfully
[2017-06-01 18:55:06,323: INFO/job_scheduler] Job 452: complete_job: Updating cache
[2017-06-01 18:55:06,324: INFO/job_scheduler] Job 452 completing (errored=False, cancelled=False)
[2017-06-01 18:55:06,383: INFO/job_scheduler] Job 451 finished 1 steps successfully
[2017-06-01 18:55:06,386: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:55:06,412: INFO/job_scheduler] Job 451: complete_job: Updating cache
[2017-06-01 18:55:06,412: INFO/job_scheduler] Job 451 completing (errored=False, cancelled=False)
[2017-06-01 18:55:06,467: INFO/job_scheduler] Job 453 finished 1 steps successfully
[2017-06-01 18:55:06,498: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:55:06,536: INFO/job_scheduler] Job 453: complete_job: Updating cache
[2017-06-01 18:55:06,536: INFO/job_scheduler] Job 453 completing (errored=False, cancelled=False)
[2017-06-01 18:55:06,572: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:55:06,674: INFO/job_scheduler] Job 454 finished 1 steps successfully
[2017-06-01 18:55:06,688: INFO/job_scheduler] Job 454: complete_job: Updating cache
[2017-06-01 18:55:06,688: INFO/job_scheduler] Job 454 completing (errored=False, cancelled=False)
[2017-06-01 18:55:06,740: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:55:08,682: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:08,682: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:08,714: INFO/command_plan] add_jobs: created Job 455 (Test for host connectivity)
[2017-06-01 18:55:08,771: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:08,771: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:08,863: INFO/command_plan] add_jobs: created Job 456 (Test for host connectivity)
[2017-06-01 18:55:08,873: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:55:08,878: INFO/job_scheduler] Job 456: RunJobThread.run
[2017-06-01 18:55:08,879: INFO/job_scheduler] Job 455: RunJobThread.run
[2017-06-01 18:55:08,893: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:55:08,939: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:08,939: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:08,965: INFO/command_plan] add_jobs: created Job 457 (Test for host connectivity)
[2017-06-01 18:55:09,067: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:55:09,071: INFO/job_scheduler] Job 457: RunJobThread.run
[2017-06-01 18:55:09,129: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:09,131: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:09,154: INFO/command_plan] add_jobs: created Job 458 (Test for host connectivity)
[2017-06-01 18:55:09,187: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:55:09,214: INFO/job_scheduler] Job 458: RunJobThread.run
[2017-06-01 18:55:11,166: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:55:11,270: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:55:11,277: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:55:11,541: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:55:12,504: INFO/job_scheduler] Job 455 finished 1 steps successfully
[2017-06-01 18:55:12,545: INFO/job_scheduler] Job 455: complete_job: Updating cache
[2017-06-01 18:55:12,545: INFO/job_scheduler] Job 455 completing (errored=False, cancelled=False)
[2017-06-01 18:55:12,598: INFO/job_scheduler] Job 456 finished 1 steps successfully
[2017-06-01 18:55:12,670: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:55:12,694: INFO/job_scheduler] Job 457 finished 1 steps successfully
[2017-06-01 18:55:12,742: INFO/job_scheduler] Job 456: complete_job: Updating cache
[2017-06-01 18:55:12,742: INFO/job_scheduler] Job 456 completing (errored=False, cancelled=False)
[2017-06-01 18:55:12,787: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:55:12,804: INFO/job_scheduler] Job 457: complete_job: Updating cache
[2017-06-01 18:55:12,805: INFO/job_scheduler] Job 457 completing (errored=False, cancelled=False)
[2017-06-01 18:55:12,837: INFO/job_scheduler] Job 458 finished 1 steps successfully
[2017-06-01 18:55:12,876: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:55:12,905: INFO/job_scheduler] Job 458: complete_job: Updating cache
[2017-06-01 18:55:12,905: INFO/job_scheduler] Job 458 completing (errored=False, cancelled=False)
[2017-06-01 18:55:12,974: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:55:14,940: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:14,942: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:14,953: INFO/command_plan] add_jobs: created Job 459 (Test for host connectivity)
[2017-06-01 18:55:14,977: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:55:14,986: INFO/job_scheduler] Job 459: RunJobThread.run
[2017-06-01 18:55:15,044: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:15,044: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:15,081: INFO/command_plan] add_jobs: created Job 460 (Test for host connectivity)
[2017-06-01 18:55:15,144: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:55:15,155: INFO/job_scheduler] Job 460: RunJobThread.run
[2017-06-01 18:55:15,202: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:15,202: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:15,231: INFO/command_plan] add_jobs: created Job 461 (Test for host connectivity)
[2017-06-01 18:55:15,316: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:55:15,321: INFO/job_scheduler] Job 461: RunJobThread.run
[2017-06-01 18:55:15,356: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:15,357: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:15,416: INFO/command_plan] add_jobs: created Job 462 (Test for host connectivity)
[2017-06-01 18:55:15,461: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:55:15,524: INFO/job_scheduler] Job 462: RunJobThread.run
[2017-06-01 18:55:17,172: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:55:17,398: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:55:17,531: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:55:17,780: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:55:18,428: INFO/job_scheduler] Job 459 finished 1 steps successfully
[2017-06-01 18:55:18,452: INFO/job_scheduler] Job 459: complete_job: Updating cache
[2017-06-01 18:55:18,453: INFO/job_scheduler] Job 459 completing (errored=False, cancelled=False)
[2017-06-01 18:55:18,513: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:55:18,743: INFO/job_scheduler] Job 460 finished 1 steps successfully
[2017-06-01 18:55:18,836: INFO/job_scheduler] Job 460: complete_job: Updating cache
[2017-06-01 18:55:18,836: INFO/job_scheduler] Job 460 completing (errored=False, cancelled=False)
[2017-06-01 18:55:18,886: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:55:18,982: INFO/job_scheduler] Job 461 finished 1 steps successfully
[2017-06-01 18:55:19,111: INFO/job_scheduler] Job 461: complete_job: Updating cache
[2017-06-01 18:55:19,111: INFO/job_scheduler] Job 461 completing (errored=False, cancelled=False)
[2017-06-01 18:55:19,236: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:55:19,274: INFO/job_scheduler] Job 462 finished 1 steps successfully
[2017-06-01 18:55:19,319: INFO/job_scheduler] Job 462: complete_job: Updating cache
[2017-06-01 18:55:19,319: INFO/job_scheduler] Job 462 completing (errored=False, cancelled=False)
[2017-06-01 18:55:19,362: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:55:21,140: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:21,140: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:21,164: INFO/command_plan] add_jobs: created Job 463 (Test for host connectivity)
[2017-06-01 18:55:21,213: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:21,213: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:21,232: INFO/command_plan] add_jobs: created Job 464 (Test for host connectivity)
[2017-06-01 18:55:21,256: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:55:21,272: INFO/job_scheduler] Job 464: RunJobThread.run
[2017-06-01 18:55:21,279: INFO/job_scheduler] Job 463: RunJobThread.run
[2017-06-01 18:55:21,291: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:55:21,352: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:21,353: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:21,377: INFO/command_plan] add_jobs: created Job 465 (Test for host connectivity)
[2017-06-01 18:55:21,482: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:21,483: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:21,556: INFO/command_plan] add_jobs: created Job 466 (Test for host connectivity)
[2017-06-01 18:55:21,591: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 18:55:21,596: INFO/job_scheduler] Job 465: RunJobThread.run
[2017-06-01 18:55:21,607: INFO/job_scheduler] Job 466: RunJobThread.run
[2017-06-01 18:55:21,632: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 18:55:23,547: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:55:23,557: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:55:23,810: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:55:23,961: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:55:24,929: INFO/job_scheduler] Job 463 finished 1 steps successfully
[2017-06-01 18:55:24,934: INFO/job_scheduler] Job 464 finished 1 steps successfully
[2017-06-01 18:55:24,988: INFO/job_scheduler] Job 463: complete_job: Updating cache
[2017-06-01 18:55:24,989: INFO/job_scheduler] Job 463 completing (errored=False, cancelled=False)
[2017-06-01 18:55:25,039: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:55:25,066: INFO/job_scheduler] Job 464: complete_job: Updating cache
[2017-06-01 18:55:25,067: INFO/job_scheduler] Job 464 completing (errored=False, cancelled=False)
[2017-06-01 18:55:25,130: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:55:25,211: INFO/job_scheduler] Job 466 finished 1 steps successfully
[2017-06-01 18:55:25,248: INFO/job_scheduler] Job 466: complete_job: Updating cache
[2017-06-01 18:55:25,248: INFO/job_scheduler] Job 466 completing (errored=False, cancelled=False)
[2017-06-01 18:55:25,305: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:55:25,499: INFO/job_scheduler] Job 465 finished 1 steps successfully
[2017-06-01 18:55:25,547: INFO/job_scheduler] Job 465: complete_job: Updating cache
[2017-06-01 18:55:25,547: INFO/job_scheduler] Job 465 completing (errored=False, cancelled=False)
[2017-06-01 18:55:25,590: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:55:27,290: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:27,291: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:27,321: INFO/command_plan] add_jobs: created Job 467 (Test for host connectivity)
[2017-06-01 18:55:27,376: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:27,376: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:27,393: INFO/command_plan] add_jobs: created Job 468 (Test for host connectivity)
[2017-06-01 18:55:27,413: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:55:27,422: INFO/job_scheduler] Job 467: RunJobThread.run
[2017-06-01 18:55:27,423: INFO/job_scheduler] Job 468: RunJobThread.run
[2017-06-01 18:55:27,445: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:55:27,529: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:27,532: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:27,556: INFO/command_plan] add_jobs: created Job 469 (Test for host connectivity)
[2017-06-01 18:55:27,587: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:55:27,635: INFO/job_scheduler] Job 469: RunJobThread.run
[2017-06-01 18:55:27,687: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:27,688: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:27,855: INFO/command_plan] add_jobs: created Job 470 (Test for host connectivity)
[2017-06-01 18:55:27,936: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:55:27,943: INFO/job_scheduler] Job 470: RunJobThread.run
[2017-06-01 18:55:29,807: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:55:29,840: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:55:29,950: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:55:30,168: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:55:31,154: INFO/job_scheduler] Job 467 finished 1 steps successfully
[2017-06-01 18:55:31,161: INFO/job_scheduler] Job 468 finished 1 steps successfully
[2017-06-01 18:55:31,204: INFO/job_scheduler] Job 467: complete_job: Updating cache
[2017-06-01 18:55:31,204: INFO/job_scheduler] Job 467 completing (errored=False, cancelled=False)
[2017-06-01 18:55:31,272: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:55:31,282: INFO/job_scheduler] Job 469 finished 1 steps successfully
[2017-06-01 18:55:31,288: INFO/job_scheduler] Job 468: complete_job: Updating cache
[2017-06-01 18:55:31,289: INFO/job_scheduler] Job 468 completing (errored=False, cancelled=False)
[2017-06-01 18:55:31,340: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:55:31,356: INFO/job_scheduler] Job 469: complete_job: Updating cache
[2017-06-01 18:55:31,356: INFO/job_scheduler] Job 469 completing (errored=False, cancelled=False)
[2017-06-01 18:55:31,412: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:55:31,496: INFO/job_scheduler] Job 470 finished 1 steps successfully
[2017-06-01 18:55:31,545: INFO/job_scheduler] Job 470: complete_job: Updating cache
[2017-06-01 18:55:31,546: INFO/job_scheduler] Job 470 completing (errored=False, cancelled=False)
[2017-06-01 18:55:31,609: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:55:33,601: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:33,601: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:33,612: INFO/command_plan] add_jobs: created Job 471 (Test for host connectivity)
[2017-06-01 18:55:33,626: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:55:33,628: INFO/job_scheduler] Job 471: RunJobThread.run
[2017-06-01 18:55:33,686: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:33,686: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:33,745: INFO/command_plan] add_jobs: created Job 472 (Test for host connectivity)
[2017-06-01 18:55:33,797: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:55:33,799: INFO/job_scheduler] Job 472: RunJobThread.run
[2017-06-01 18:55:33,878: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:33,880: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:33,904: INFO/command_plan] add_jobs: created Job 473 (Test for host connectivity)
[2017-06-01 18:55:33,985: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:33,985: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:33,996: INFO/command_plan] add_jobs: created Job 474 (Test for host connectivity)
[2017-06-01 18:55:34,019: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 18:55:34,042: INFO/job_scheduler] Job 473: RunJobThread.run
[2017-06-01 18:55:34,063: INFO/job_scheduler] Job 474: RunJobThread.run
[2017-06-01 18:55:34,072: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 18:55:36,017: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:55:36,109: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:55:36,275: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:55:36,357: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:55:37,348: INFO/job_scheduler] Job 471 finished 1 steps successfully
[2017-06-01 18:55:37,369: INFO/job_scheduler] Job 471: complete_job: Updating cache
[2017-06-01 18:55:37,370: INFO/job_scheduler] Job 471 completing (errored=False, cancelled=False)
[2017-06-01 18:55:37,438: INFO/job_scheduler] Job 472 finished 1 steps successfully
[2017-06-01 18:55:37,454: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:55:37,478: INFO/job_scheduler] Job 472: complete_job: Updating cache
[2017-06-01 18:55:37,478: INFO/job_scheduler] Job 472 completing (errored=False, cancelled=False)
[2017-06-01 18:55:37,570: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:55:37,640: INFO/job_scheduler] Job 473 finished 1 steps successfully
[2017-06-01 18:55:37,678: INFO/job_scheduler] Job 473: complete_job: Updating cache
[2017-06-01 18:55:37,679: INFO/job_scheduler] Job 473 completing (errored=False, cancelled=False)
[2017-06-01 18:55:37,687: INFO/job_scheduler] Job 474 finished 1 steps successfully
[2017-06-01 18:55:37,720: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:55:37,760: INFO/job_scheduler] Job 474: complete_job: Updating cache
[2017-06-01 18:55:37,760: INFO/job_scheduler] Job 474 completing (errored=False, cancelled=False)
[2017-06-01 18:55:37,798: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:55:39,714: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:39,715: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:39,736: INFO/command_plan] add_jobs: created Job 475 (Test for host connectivity)
[2017-06-01 18:55:39,766: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:55:39,778: INFO/job_scheduler] Job 475: RunJobThread.run
[2017-06-01 18:55:39,823: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:39,823: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:39,838: INFO/command_plan] add_jobs: created Job 476 (Test for host connectivity)
[2017-06-01 18:55:39,914: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:55:39,916: INFO/job_scheduler] Job 476: RunJobThread.run
[2017-06-01 18:55:39,954: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:39,955: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:40,002: INFO/command_plan] add_jobs: created Job 477 (Test for host connectivity)
[2017-06-01 18:55:40,090: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:55:40,091: INFO/job_scheduler] Job 477: RunJobThread.run
[2017-06-01 18:55:40,158: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:40,158: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:40,179: INFO/command_plan] add_jobs: created Job 478 (Test for host connectivity)
[2017-06-01 18:55:40,278: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:55:40,280: INFO/job_scheduler] Job 478: RunJobThread.run
[2017-06-01 18:55:42,051: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:55:42,236: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:55:42,304: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:55:42,528: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:55:43,385: INFO/job_scheduler] Job 475 finished 1 steps successfully
[2017-06-01 18:55:43,424: INFO/job_scheduler] Job 475: complete_job: Updating cache
[2017-06-01 18:55:43,426: INFO/job_scheduler] Job 475 completing (errored=False, cancelled=False)
[2017-06-01 18:55:43,487: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:55:43,571: INFO/job_scheduler] Job 476 finished 1 steps successfully
[2017-06-01 18:55:43,618: INFO/job_scheduler] Job 476: complete_job: Updating cache
[2017-06-01 18:55:43,618: INFO/job_scheduler] Job 476 completing (errored=False, cancelled=False)
[2017-06-01 18:55:43,678: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:55:43,724: INFO/job_scheduler] Job 477 finished 1 steps successfully
[2017-06-01 18:55:43,760: INFO/job_scheduler] Job 477: complete_job: Updating cache
[2017-06-01 18:55:43,761: INFO/job_scheduler] Job 477 completing (errored=False, cancelled=False)
[2017-06-01 18:55:43,812: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:55:43,884: INFO/job_scheduler] Job 478 finished 1 steps successfully
[2017-06-01 18:55:43,910: INFO/job_scheduler] Job 478: complete_job: Updating cache
[2017-06-01 18:55:43,910: INFO/job_scheduler] Job 478 completing (errored=False, cancelled=False)
[2017-06-01 18:55:43,944: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:55:45,928: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:45,928: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:45,944: INFO/command_plan] add_jobs: created Job 479 (Test for host connectivity)
[2017-06-01 18:55:45,967: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:55:45,969: INFO/job_scheduler] Job 479: RunJobThread.run
[2017-06-01 18:55:46,124: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:46,125: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:46,152: INFO/command_plan] add_jobs: created Job 480 (Test for host connectivity)
[2017-06-01 18:55:46,183: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:55:46,187: INFO/job_scheduler] Job 480: RunJobThread.run
[2017-06-01 18:55:46,228: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:46,229: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:46,256: INFO/command_plan] add_jobs: created Job 481 (Test for host connectivity)
[2017-06-01 18:55:46,298: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:55:46,310: INFO/job_scheduler] Job 481: RunJobThread.run
[2017-06-01 18:55:46,345: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:46,345: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:46,368: INFO/command_plan] add_jobs: created Job 482 (Test for host connectivity)
[2017-06-01 18:55:46,392: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:55:46,439: INFO/job_scheduler] Job 482: RunJobThread.run
[2017-06-01 18:55:48,319: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:55:48,483: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:55:48,674: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:55:48,717: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:55:49,663: INFO/job_scheduler] Job 479 finished 1 steps successfully
[2017-06-01 18:55:49,725: INFO/job_scheduler] Job 479: complete_job: Updating cache
[2017-06-01 18:55:49,725: INFO/job_scheduler] Job 479 completing (errored=False, cancelled=False)
[2017-06-01 18:55:49,776: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:55:49,780: INFO/job_scheduler] Job 480 finished 1 steps successfully
[2017-06-01 18:55:49,818: INFO/job_scheduler] Job 480: complete_job: Updating cache
[2017-06-01 18:55:49,818: INFO/job_scheduler] Job 480 completing (errored=False, cancelled=False)
[2017-06-01 18:55:49,899: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:55:50,006: INFO/job_scheduler] Job 482 finished 1 steps successfully
[2017-06-01 18:55:50,034: INFO/job_scheduler] Job 482: complete_job: Updating cache
[2017-06-01 18:55:50,034: INFO/job_scheduler] Job 482 completing (errored=False, cancelled=False)
[2017-06-01 18:55:50,071: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:55:50,118: INFO/job_scheduler] Job 481 finished 1 steps successfully
[2017-06-01 18:55:50,142: INFO/job_scheduler] Job 481: complete_job: Updating cache
[2017-06-01 18:55:50,142: INFO/job_scheduler] Job 481 completing (errored=False, cancelled=False)
[2017-06-01 18:55:50,184: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:55:52,080: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:52,082: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:52,100: INFO/command_plan] add_jobs: created Job 483 (Test for host connectivity)
[2017-06-01 18:55:52,186: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:52,191: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:52,283: INFO/command_plan] add_jobs: created Job 484 (Test for host connectivity)
[2017-06-01 18:55:52,293: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:55:52,296: INFO/job_scheduler] Job 483: RunJobThread.run
[2017-06-01 18:55:52,296: INFO/job_scheduler] Job 484: RunJobThread.run
[2017-06-01 18:55:52,301: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:55:52,344: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:52,345: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:52,387: INFO/command_plan] add_jobs: created Job 485 (Test for host connectivity)
[2017-06-01 18:55:52,529: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:52,529: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:52,560: INFO/command_plan] add_jobs: created Job 486 (Test for host connectivity)
[2017-06-01 18:55:52,576: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 18:55:52,592: INFO/job_scheduler] Job 485: RunJobThread.run
[2017-06-01 18:55:52,610: INFO/job_scheduler] Job 486: RunJobThread.run
[2017-06-01 18:55:52,625: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 18:55:54,544: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:55:54,708: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:55:54,834: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:55:54,927: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:55:55,942: INFO/job_scheduler] Job 484 finished 1 steps successfully
[2017-06-01 18:55:55,991: INFO/job_scheduler] Job 484: complete_job: Updating cache
[2017-06-01 18:55:55,991: INFO/job_scheduler] Job 484 completing (errored=False, cancelled=False)
[2017-06-01 18:55:56,048: INFO/job_scheduler] Job 483 finished 1 steps successfully
[2017-06-01 18:55:56,091: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:55:56,107: INFO/job_scheduler] Job 483: complete_job: Updating cache
[2017-06-01 18:55:56,107: INFO/job_scheduler] Job 483 completing (errored=False, cancelled=False)
[2017-06-01 18:55:56,193: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:55:56,195: INFO/job_scheduler] Job 486 finished 1 steps successfully
[2017-06-01 18:55:56,219: INFO/job_scheduler] Job 486: complete_job: Updating cache
[2017-06-01 18:55:56,219: INFO/job_scheduler] Job 486 completing (errored=False, cancelled=False)
[2017-06-01 18:55:56,266: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:55:56,340: INFO/job_scheduler] Job 485 finished 1 steps successfully
[2017-06-01 18:55:56,424: INFO/job_scheduler] Job 485: complete_job: Updating cache
[2017-06-01 18:55:56,424: INFO/job_scheduler] Job 485 completing (errored=False, cancelled=False)
[2017-06-01 18:55:56,490: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:55:58,275: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:58,276: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:58,283: INFO/command_plan] add_jobs: created Job 487 (Test for host connectivity)
[2017-06-01 18:55:58,308: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:55:58,312: INFO/job_scheduler] Job 487: RunJobThread.run
[2017-06-01 18:55:58,361: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:58,362: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:58,379: INFO/command_plan] add_jobs: created Job 488 (Test for host connectivity)
[2017-06-01 18:55:58,523: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:58,525: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:58,540: INFO/command_plan] add_jobs: created Job 489 (Test for host connectivity)
[2017-06-01 18:55:58,561: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 18:55:58,566: INFO/job_scheduler] Job 488: RunJobThread.run
[2017-06-01 18:55:58,599: INFO/job_scheduler] Job 489: RunJobThread.run
[2017-06-01 18:55:58,677: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:55:58,678: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:55:58,691: INFO/command_plan] add_jobs: created Job 490 (Test for host connectivity)
[2017-06-01 18:55:58,751: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:55:58,784: INFO/job_scheduler] Job 490: RunJobThread.run
[2017-06-01 18:55:58,941: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 18:56:00,568: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:56:00,760: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:56:00,874: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:56:01,082: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:56:01,921: INFO/job_scheduler] Job 487 finished 1 steps successfully
[2017-06-01 18:56:01,948: INFO/job_scheduler] Job 487: complete_job: Updating cache
[2017-06-01 18:56:01,948: INFO/job_scheduler] Job 487 completing (errored=False, cancelled=False)
[2017-06-01 18:56:01,999: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:56:02,150: INFO/job_scheduler] Job 488 finished 1 steps successfully
[2017-06-01 18:56:02,220: INFO/job_scheduler] Job 489 finished 1 steps successfully
[2017-06-01 18:56:02,232: INFO/job_scheduler] Job 488: complete_job: Updating cache
[2017-06-01 18:56:02,232: INFO/job_scheduler] Job 488 completing (errored=False, cancelled=False)
[2017-06-01 18:56:02,294: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:56:02,332: INFO/job_scheduler] Job 489: complete_job: Updating cache
[2017-06-01 18:56:02,332: INFO/job_scheduler] Job 489 completing (errored=False, cancelled=False)
[2017-06-01 18:56:02,392: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:56:02,428: INFO/job_scheduler] Job 490 finished 1 steps successfully
[2017-06-01 18:56:02,449: INFO/job_scheduler] Job 490: complete_job: Updating cache
[2017-06-01 18:56:02,449: INFO/job_scheduler] Job 490 completing (errored=False, cancelled=False)
[2017-06-01 18:56:02,500: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:56:05,164: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:05,165: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:05,233: INFO/command_plan] add_jobs: created Job 491 (Test for host connectivity)
[2017-06-01 18:56:05,270: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:56:05,276: INFO/job_scheduler] Job 491: RunJobThread.run
[2017-06-01 18:56:05,360: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:05,367: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:05,388: INFO/command_plan] add_jobs: created Job 492 (Test for host connectivity)
[2017-06-01 18:56:05,437: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:56:05,471: INFO/job_scheduler] Job 492: RunJobThread.run
[2017-06-01 18:56:05,497: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:05,497: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:05,531: INFO/command_plan] add_jobs: created Job 493 (Test for host connectivity)
[2017-06-01 18:56:05,562: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:56:05,578: INFO/job_scheduler] Job 493: RunJobThread.run
[2017-06-01 18:56:05,630: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:05,630: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:05,679: INFO/command_plan] add_jobs: created Job 494 (Test for host connectivity)
[2017-06-01 18:56:05,750: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:56:05,772: INFO/job_scheduler] Job 494: RunJobThread.run
[2017-06-01 18:56:07,644: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:56:07,802: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:56:07,834: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:56:07,919: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:56:08,955: INFO/job_scheduler] Job 491 finished 1 steps successfully
[2017-06-01 18:56:09,016: INFO/job_scheduler] Job 491: complete_job: Updating cache
[2017-06-01 18:56:09,016: INFO/job_scheduler] Job 491 completing (errored=False, cancelled=False)
[2017-06-01 18:56:09,066: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:56:09,133: INFO/job_scheduler] Job 492 finished 1 steps successfully
[2017-06-01 18:56:09,151: INFO/job_scheduler] Job 492: complete_job: Updating cache
[2017-06-01 18:56:09,151: INFO/job_scheduler] Job 492 completing (errored=False, cancelled=False)
[2017-06-01 18:56:09,209: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:56:09,223: INFO/job_scheduler] Job 493 finished 1 steps successfully
[2017-06-01 18:56:09,254: INFO/job_scheduler] Job 493: complete_job: Updating cache
[2017-06-01 18:56:09,254: INFO/job_scheduler] Job 493 completing (errored=False, cancelled=False)
[2017-06-01 18:56:09,267: INFO/job_scheduler] Job 494 finished 1 steps successfully
[2017-06-01 18:56:09,300: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:56:09,316: INFO/job_scheduler] Job 494: complete_job: Updating cache
[2017-06-01 18:56:09,316: INFO/job_scheduler] Job 494 completing (errored=False, cancelled=False)
[2017-06-01 18:56:09,369: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:56:11,349: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:11,351: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:11,366: INFO/command_plan] add_jobs: created Job 495 (Test for host connectivity)
[2017-06-01 18:56:11,455: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:11,456: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:11,474: INFO/command_plan] add_jobs: created Job 496 (Test for host connectivity)
[2017-06-01 18:56:11,493: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:56:11,515: INFO/job_scheduler] Job 496: RunJobThread.run
[2017-06-01 18:56:11,540: INFO/job_scheduler] Job 495: RunJobThread.run
[2017-06-01 18:56:11,549: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:56:11,622: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:11,622: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:11,634: INFO/command_plan] add_jobs: created Job 497 (Test for host connectivity)
[2017-06-01 18:56:11,688: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:56:11,691: INFO/job_scheduler] Job 497: RunJobThread.run
[2017-06-01 18:56:11,764: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:11,764: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:11,794: INFO/command_plan] add_jobs: created Job 498 (Test for host connectivity)
[2017-06-01 18:56:11,868: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:56:11,898: INFO/job_scheduler] Job 498: RunJobThread.run
[2017-06-01 18:56:13,814: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:56:13,847: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:56:14,033: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:56:14,058: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:56:15,163: INFO/job_scheduler] Job 495 finished 1 steps successfully
[2017-06-01 18:56:15,183: INFO/job_scheduler] Job 496 finished 1 steps successfully
[2017-06-01 18:56:15,214: INFO/job_scheduler] Job 495: complete_job: Updating cache
[2017-06-01 18:56:15,214: INFO/job_scheduler] Job 495 completing (errored=False, cancelled=False)
[2017-06-01 18:56:15,258: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:56:15,273: INFO/job_scheduler] Job 496: complete_job: Updating cache
[2017-06-01 18:56:15,274: INFO/job_scheduler] Job 496 completing (errored=False, cancelled=False)
[2017-06-01 18:56:15,351: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:56:15,381: INFO/job_scheduler] Job 498 finished 1 steps successfully
[2017-06-01 18:56:15,399: INFO/job_scheduler] Job 498: complete_job: Updating cache
[2017-06-01 18:56:15,399: INFO/job_scheduler] Job 498 completing (errored=False, cancelled=False)
[2017-06-01 18:56:15,420: INFO/job_scheduler] Job 497 finished 1 steps successfully
[2017-06-01 18:56:15,490: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:56:15,506: INFO/job_scheduler] Job 497: complete_job: Updating cache
[2017-06-01 18:56:15,506: INFO/job_scheduler] Job 497 completing (errored=False, cancelled=False)
[2017-06-01 18:56:15,574: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:56:17,529: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:17,530: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:17,547: INFO/command_plan] add_jobs: created Job 499 (Test for host connectivity)
[2017-06-01 18:56:17,584: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:56:17,587: INFO/job_scheduler] Job 499: RunJobThread.run
[2017-06-01 18:56:17,637: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:17,637: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:17,664: INFO/command_plan] add_jobs: created Job 500 (Test for host connectivity)
[2017-06-01 18:56:17,762: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:17,763: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:17,819: INFO/command_plan] add_jobs: created Job 501 (Test for host connectivity)
[2017-06-01 18:56:17,846: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 18:56:17,880: INFO/job_scheduler] Job 500: RunJobThread.run
[2017-06-01 18:56:17,881: INFO/job_scheduler] Job 501: RunJobThread.run
[2017-06-01 18:56:17,893: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:56:17,951: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:17,952: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:17,965: INFO/command_plan] add_jobs: created Job 502 (Test for host connectivity)
[2017-06-01 18:56:18,003: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:56:18,107: INFO/job_scheduler] Job 502: RunJobThread.run
[2017-06-01 18:56:19,976: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:56:20,121: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:56:20,304: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:56:20,404: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:56:21,305: INFO/job_scheduler] Job 499 finished 1 steps successfully
[2017-06-01 18:56:21,376: INFO/job_scheduler] Job 499: complete_job: Updating cache
[2017-06-01 18:56:21,379: INFO/job_scheduler] Job 499 completing (errored=False, cancelled=False)
[2017-06-01 18:56:21,460: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:56:21,473: INFO/job_scheduler] Job 500 finished 1 steps successfully
[2017-06-01 18:56:21,613: INFO/job_scheduler] Job 500: complete_job: Updating cache
[2017-06-01 18:56:21,613: INFO/job_scheduler] Job 500 completing (errored=False, cancelled=False)
[2017-06-01 18:56:21,655: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:56:21,705: INFO/job_scheduler] Job 501 finished 1 steps successfully
[2017-06-01 18:56:21,730: INFO/job_scheduler] Job 501: complete_job: Updating cache
[2017-06-01 18:56:21,730: INFO/job_scheduler] Job 501 completing (errored=False, cancelled=False)
[2017-06-01 18:56:21,772: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:56:21,774: INFO/job_scheduler] Job 502 finished 1 steps successfully
[2017-06-01 18:56:21,788: INFO/job_scheduler] Job 502: complete_job: Updating cache
[2017-06-01 18:56:21,788: INFO/job_scheduler] Job 502 completing (errored=False, cancelled=False)
[2017-06-01 18:56:21,868: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:56:23,714: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:23,715: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:23,746: INFO/command_plan] add_jobs: created Job 503 (Test for host connectivity)
[2017-06-01 18:56:23,783: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:23,783: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:23,809: INFO/command_plan] add_jobs: created Job 504 (Test for host connectivity)
[2017-06-01 18:56:23,831: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:56:23,845: INFO/job_scheduler] Job 504: RunJobThread.run
[2017-06-01 18:56:23,852: INFO/job_scheduler] Job 503: RunJobThread.run
[2017-06-01 18:56:23,863: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:56:23,935: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:23,937: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:23,974: INFO/command_plan] add_jobs: created Job 505 (Test for host connectivity)
[2017-06-01 18:56:24,020: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:56:24,056: INFO/job_scheduler] Job 505: RunJobThread.run
[2017-06-01 18:56:24,090: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:24,097: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:24,146: INFO/command_plan] add_jobs: created Job 506 (Test for host connectivity)
[2017-06-01 18:56:24,202: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:56:24,252: INFO/job_scheduler] Job 506: RunJobThread.run
[2017-06-01 18:56:25,977: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:56:26,198: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:56:26,286: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:56:26,512: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:56:27,314: INFO/job_scheduler] Job 503 finished 1 steps successfully
[2017-06-01 18:56:27,331: INFO/job_scheduler] Job 503: complete_job: Updating cache
[2017-06-01 18:56:27,331: INFO/job_scheduler] Job 503 completing (errored=False, cancelled=False)
[2017-06-01 18:56:27,399: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:56:27,632: INFO/job_scheduler] Job 504 finished 1 steps successfully
[2017-06-01 18:56:27,667: INFO/job_scheduler] Job 505 finished 1 steps successfully
[2017-06-01 18:56:27,720: INFO/job_scheduler] Job 504: complete_job: Updating cache
[2017-06-01 18:56:27,720: INFO/job_scheduler] Job 504 completing (errored=False, cancelled=False)
[2017-06-01 18:56:27,771: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:56:27,803: INFO/job_scheduler] Job 505: complete_job: Updating cache
[2017-06-01 18:56:27,803: INFO/job_scheduler] Job 505 completing (errored=False, cancelled=False)
[2017-06-01 18:56:27,869: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:56:27,886: INFO/job_scheduler] Job 506 finished 1 steps successfully
[2017-06-01 18:56:27,912: INFO/job_scheduler] Job 506: complete_job: Updating cache
[2017-06-01 18:56:27,912: INFO/job_scheduler] Job 506 completing (errored=False, cancelled=False)
[2017-06-01 18:56:27,951: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:56:29,815: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:29,816: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:29,828: INFO/command_plan] add_jobs: created Job 507 (Test for host connectivity)
[2017-06-01 18:56:29,883: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:29,883: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:29,912: INFO/command_plan] add_jobs: created Job 508 (Test for host connectivity)
[2017-06-01 18:56:29,920: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:56:29,926: INFO/job_scheduler] Job 507: RunJobThread.run
[2017-06-01 18:56:29,927: INFO/job_scheduler] Job 508: RunJobThread.run
[2017-06-01 18:56:29,940: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:56:29,996: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:29,996: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:30,018: INFO/command_plan] add_jobs: created Job 509 (Test for host connectivity)
[2017-06-01 18:56:30,106: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:56:30,111: INFO/job_scheduler] Job 509: RunJobThread.run
[2017-06-01 18:56:30,169: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:30,176: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:30,187: INFO/command_plan] add_jobs: created Job 510 (Test for host connectivity)
[2017-06-01 18:56:30,225: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:56:30,238: INFO/job_scheduler] Job 510: RunJobThread.run
[2017-06-01 18:56:32,142: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:56:32,171: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:56:32,438: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:56:32,463: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:56:33,472: INFO/job_scheduler] Job 508 finished 1 steps successfully
[2017-06-01 18:56:33,492: INFO/job_scheduler] Job 507 finished 1 steps successfully
[2017-06-01 18:56:33,494: INFO/job_scheduler] Job 508: complete_job: Updating cache
[2017-06-01 18:56:33,494: INFO/job_scheduler] Job 508 completing (errored=False, cancelled=False)
[2017-06-01 18:56:33,574: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:56:33,599: INFO/job_scheduler] Job 507: complete_job: Updating cache
[2017-06-01 18:56:33,599: INFO/job_scheduler] Job 507 completing (errored=False, cancelled=False)
[2017-06-01 18:56:33,752: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:56:33,783: INFO/job_scheduler] Job 510 finished 1 steps successfully
[2017-06-01 18:56:33,804: INFO/job_scheduler] Job 510: complete_job: Updating cache
[2017-06-01 18:56:33,804: INFO/job_scheduler] Job 510 completing (errored=False, cancelled=False)
[2017-06-01 18:56:33,888: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:56:33,919: INFO/job_scheduler] Job 509 finished 1 steps successfully
[2017-06-01 18:56:33,944: INFO/job_scheduler] Job 509: complete_job: Updating cache
[2017-06-01 18:56:33,944: INFO/job_scheduler] Job 509 completing (errored=False, cancelled=False)
[2017-06-01 18:56:34,003: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:56:35,845: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:35,846: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:35,853: INFO/command_plan] add_jobs: created Job 511 (Test for host connectivity)
[2017-06-01 18:56:35,871: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:56:35,873: INFO/job_scheduler] Job 511: RunJobThread.run
[2017-06-01 18:56:35,909: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:35,909: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:35,921: INFO/command_plan] add_jobs: created Job 512 (Test for host connectivity)
[2017-06-01 18:56:36,071: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:56:36,073: INFO/job_scheduler] Job 512: RunJobThread.run
[2017-06-01 18:56:36,137: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:36,141: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:36,162: INFO/command_plan] add_jobs: created Job 513 (Test for host connectivity)
[2017-06-01 18:56:36,197: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:56:36,210: INFO/job_scheduler] Job 513: RunJobThread.run
[2017-06-01 18:56:36,285: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:36,286: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:36,311: INFO/command_plan] add_jobs: created Job 514 (Test for host connectivity)
[2017-06-01 18:56:36,352: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:56:36,417: INFO/job_scheduler] Job 514: RunJobThread.run
[2017-06-01 18:56:38,274: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:56:38,281: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:56:38,630: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:56:38,654: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:56:39,633: INFO/job_scheduler] Job 512 finished 1 steps successfully
[2017-06-01 18:56:39,666: INFO/job_scheduler] Job 511 finished 1 steps successfully
[2017-06-01 18:56:39,692: INFO/job_scheduler] Job 512: complete_job: Updating cache
[2017-06-01 18:56:39,694: INFO/job_scheduler] Job 512 completing (errored=False, cancelled=False)
[2017-06-01 18:56:39,786: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:56:39,810: INFO/job_scheduler] Job 511: complete_job: Updating cache
[2017-06-01 18:56:39,810: INFO/job_scheduler] Job 511 completing (errored=False, cancelled=False)
[2017-06-01 18:56:39,935: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:56:40,010: INFO/job_scheduler] Job 514 finished 1 steps successfully
[2017-06-01 18:56:40,047: INFO/job_scheduler] Job 513 finished 1 steps successfully
[2017-06-01 18:56:40,068: INFO/job_scheduler] Job 514: complete_job: Updating cache
[2017-06-01 18:56:40,068: INFO/job_scheduler] Job 514 completing (errored=False, cancelled=False)
[2017-06-01 18:56:40,119: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:56:40,145: INFO/job_scheduler] Job 513: complete_job: Updating cache
[2017-06-01 18:56:40,145: INFO/job_scheduler] Job 513 completing (errored=False, cancelled=False)
[2017-06-01 18:56:40,203: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:56:42,006: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:42,008: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:42,034: INFO/command_plan] add_jobs: created Job 515 (Test for host connectivity)
[2017-06-01 18:56:42,070: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:42,071: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:42,113: INFO/command_plan] add_jobs: created Job 516 (Test for host connectivity)
[2017-06-01 18:56:42,126: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:56:42,131: INFO/job_scheduler] Job 515: RunJobThread.run
[2017-06-01 18:56:42,148: INFO/job_scheduler] Job 516: RunJobThread.run
[2017-06-01 18:56:42,164: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:56:42,228: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:42,229: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:42,269: INFO/command_plan] add_jobs: created Job 517 (Test for host connectivity)
[2017-06-01 18:56:42,392: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:42,393: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:42,434: INFO/command_plan] add_jobs: created Job 518 (Test for host connectivity)
[2017-06-01 18:56:42,443: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 18:56:42,446: INFO/job_scheduler] Job 517: RunJobThread.run
[2017-06-01 18:56:42,454: INFO/job_scheduler] Job 518: RunJobThread.run
[2017-06-01 18:56:42,467: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 18:56:44,401: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:56:44,484: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:56:44,744: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:56:44,784: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:56:45,731: INFO/job_scheduler] Job 515 finished 1 steps successfully
[2017-06-01 18:56:45,814: INFO/job_scheduler] Job 515: complete_job: Updating cache
[2017-06-01 18:56:45,815: INFO/job_scheduler] Job 515 completing (errored=False, cancelled=False)
[2017-06-01 18:56:45,866: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:56:45,877: INFO/job_scheduler] Job 516 finished 1 steps successfully
[2017-06-01 18:56:45,932: INFO/job_scheduler] Job 516: complete_job: Updating cache
[2017-06-01 18:56:45,933: INFO/job_scheduler] Job 516 completing (errored=False, cancelled=False)
[2017-06-01 18:56:46,001: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:56:46,100: INFO/job_scheduler] Job 517 finished 1 steps successfully
[2017-06-01 18:56:46,150: INFO/job_scheduler] Job 517: complete_job: Updating cache
[2017-06-01 18:56:46,150: INFO/job_scheduler] Job 517 completing (errored=False, cancelled=False)
[2017-06-01 18:56:46,205: INFO/job_scheduler] Job 518 finished 1 steps successfully
[2017-06-01 18:56:46,209: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:56:46,225: INFO/job_scheduler] Job 518: complete_job: Updating cache
[2017-06-01 18:56:46,237: INFO/job_scheduler] Job 518 completing (errored=False, cancelled=False)
[2017-06-01 18:56:46,310: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:56:48,092: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:48,092: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:48,125: INFO/command_plan] add_jobs: created Job 519 (Test for host connectivity)
[2017-06-01 18:56:48,187: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:48,187: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:48,218: INFO/command_plan] add_jobs: created Job 520 (Test for host connectivity)
[2017-06-01 18:56:48,234: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:56:48,258: INFO/job_scheduler] Job 520: RunJobThread.run
[2017-06-01 18:56:48,261: INFO/job_scheduler] Job 519: RunJobThread.run
[2017-06-01 18:56:48,272: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:56:48,301: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:48,302: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:48,326: INFO/command_plan] add_jobs: created Job 521 (Test for host connectivity)
[2017-06-01 18:56:48,481: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:48,481: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:48,508: INFO/command_plan] add_jobs: created Job 522 (Test for host connectivity)
[2017-06-01 18:56:48,537: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 18:56:48,544: INFO/job_scheduler] Job 521: RunJobThread.run
[2017-06-01 18:56:48,549: INFO/job_scheduler] Job 522: RunJobThread.run
[2017-06-01 18:56:48,559: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 18:56:50,528: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:56:50,584: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:56:50,718: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:56:50,749: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:56:51,867: INFO/job_scheduler] Job 519 finished 1 steps successfully
[2017-06-01 18:56:51,924: INFO/job_scheduler] Job 519: complete_job: Updating cache
[2017-06-01 18:56:51,924: INFO/job_scheduler] Job 519 completing (errored=False, cancelled=False)
[2017-06-01 18:56:51,954: INFO/job_scheduler] Job 520 finished 1 steps successfully
[2017-06-01 18:56:51,994: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:56:52,007: INFO/job_scheduler] Job 520: complete_job: Updating cache
[2017-06-01 18:56:52,007: INFO/job_scheduler] Job 520 completing (errored=False, cancelled=False)
[2017-06-01 18:56:52,064: INFO/job_scheduler] Job 521 finished 1 steps successfully
[2017-06-01 18:56:52,064: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:56:52,067: INFO/job_scheduler] Job 522 finished 1 steps successfully
[2017-06-01 18:56:52,106: INFO/job_scheduler] Job 521: complete_job: Updating cache
[2017-06-01 18:56:52,107: INFO/job_scheduler] Job 521 completing (errored=False, cancelled=False)
[2017-06-01 18:56:52,207: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:56:52,249: INFO/job_scheduler] Job 522: complete_job: Updating cache
[2017-06-01 18:56:52,249: INFO/job_scheduler] Job 522 completing (errored=False, cancelled=False)
[2017-06-01 18:56:52,300: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:56:54,294: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:54,294: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:54,307: INFO/command_plan] add_jobs: created Job 523 (Test for host connectivity)
[2017-06-01 18:56:54,332: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:56:54,333: INFO/job_scheduler] Job 523: RunJobThread.run
[2017-06-01 18:56:54,385: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:54,386: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:54,399: INFO/command_plan] add_jobs: created Job 524 (Test for host connectivity)
[2017-06-01 18:56:54,470: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:56:54,472: INFO/job_scheduler] Job 524: RunJobThread.run
[2017-06-01 18:56:54,517: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:54,520: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:54,532: INFO/command_plan] add_jobs: created Job 525 (Test for host connectivity)
[2017-06-01 18:56:54,640: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:56:54,640: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:56:54,657: INFO/command_plan] add_jobs: created Job 526 (Test for host connectivity)
[2017-06-01 18:56:54,673: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 18:56:54,685: INFO/job_scheduler] Job 525: RunJobThread.run
[2017-06-01 18:56:54,686: INFO/job_scheduler] Job 526: RunJobThread.run
[2017-06-01 18:56:54,702: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 18:56:56,628: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:56:56,743: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:56:57,003: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:56:57,086: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:56:57,967: INFO/job_scheduler] Job 523 finished 1 steps successfully
[2017-06-01 18:56:57,997: INFO/job_scheduler] Job 523: complete_job: Updating cache
[2017-06-01 18:56:57,997: INFO/job_scheduler] Job 523 completing (errored=False, cancelled=False)
[2017-06-01 18:56:58,045: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:56:58,166: INFO/job_scheduler] Job 524 finished 1 steps successfully
[2017-06-01 18:56:58,205: INFO/job_scheduler] Job 524: complete_job: Updating cache
[2017-06-01 18:56:58,206: INFO/job_scheduler] Job 524 completing (errored=False, cancelled=False)
[2017-06-01 18:56:58,272: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:56:58,356: INFO/job_scheduler] Job 526 finished 1 steps successfully
[2017-06-01 18:56:58,413: INFO/job_scheduler] Job 526: complete_job: Updating cache
[2017-06-01 18:56:58,413: INFO/job_scheduler] Job 526 completing (errored=False, cancelled=False)
[2017-06-01 18:56:58,464: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:56:58,526: INFO/job_scheduler] Job 525 finished 1 steps successfully
[2017-06-01 18:56:58,572: INFO/job_scheduler] Job 525: complete_job: Updating cache
[2017-06-01 18:56:58,572: INFO/job_scheduler] Job 525 completing (errored=False, cancelled=False)
[2017-06-01 18:56:58,605: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:57:00,355: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:00,355: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:00,372: INFO/command_plan] add_jobs: created Job 527 (Test for host connectivity)
[2017-06-01 18:57:00,426: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:57:00,429: INFO/job_scheduler] Job 527: RunJobThread.run
[2017-06-01 18:57:00,470: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:00,471: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:00,492: INFO/command_plan] add_jobs: created Job 528 (Test for host connectivity)
[2017-06-01 18:57:00,519: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:57:00,546: INFO/job_scheduler] Job 528: RunJobThread.run
[2017-06-01 18:57:00,575: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:00,575: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:00,622: INFO/command_plan] add_jobs: created Job 529 (Test for host connectivity)
[2017-06-01 18:57:00,647: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:57:00,656: INFO/job_scheduler] Job 529: RunJobThread.run
[2017-06-01 18:57:00,721: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:00,721: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:00,738: INFO/command_plan] add_jobs: created Job 530 (Test for host connectivity)
[2017-06-01 18:57:00,806: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:57:00,811: INFO/job_scheduler] Job 530: RunJobThread.run
[2017-06-01 18:57:02,644: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:57:02,784: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:57:02,992: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:57:03,020: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:57:03,926: INFO/job_scheduler] Job 527 finished 1 steps successfully
[2017-06-01 18:57:04,028: INFO/job_scheduler] Job 527: complete_job: Updating cache
[2017-06-01 18:57:04,032: INFO/job_scheduler] Job 527 completing (errored=False, cancelled=False)
[2017-06-01 18:57:04,099: INFO/job_scheduler] Job 528 finished 1 steps successfully
[2017-06-01 18:57:04,105: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:57:04,130: INFO/job_scheduler] Job 528: complete_job: Updating cache
[2017-06-01 18:57:04,130: INFO/job_scheduler] Job 528 completing (errored=False, cancelled=False)
[2017-06-01 18:57:04,182: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:57:04,346: INFO/job_scheduler] Job 529 finished 1 steps successfully
[2017-06-01 18:57:04,362: INFO/job_scheduler] Job 530 finished 1 steps successfully
[2017-06-01 18:57:04,379: INFO/job_scheduler] Job 529: complete_job: Updating cache
[2017-06-01 18:57:04,379: INFO/job_scheduler] Job 529 completing (errored=False, cancelled=False)
[2017-06-01 18:57:04,423: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:57:04,438: INFO/job_scheduler] Job 530: complete_job: Updating cache
[2017-06-01 18:57:04,439: INFO/job_scheduler] Job 530 completing (errored=False, cancelled=False)
[2017-06-01 18:57:04,517: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:57:06,484: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:06,485: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:06,505: INFO/command_plan] add_jobs: created Job 531 (Test for host connectivity)
[2017-06-01 18:57:06,525: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:57:06,551: INFO/job_scheduler] Job 531: RunJobThread.run
[2017-06-01 18:57:06,587: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:06,587: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:06,605: INFO/command_plan] add_jobs: created Job 532 (Test for host connectivity)
[2017-06-01 18:57:06,639: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:57:06,647: INFO/job_scheduler] Job 532: RunJobThread.run
[2017-06-01 18:57:06,743: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:06,743: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:06,771: INFO/command_plan] add_jobs: created Job 533 (Test for host connectivity)
[2017-06-01 18:57:06,843: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:57:06,848: INFO/job_scheduler] Job 533: RunJobThread.run
[2017-06-01 18:57:06,874: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:06,875: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:06,905: INFO/command_plan] add_jobs: created Job 534 (Test for host connectivity)
[2017-06-01 18:57:06,957: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:57:06,978: INFO/job_scheduler] Job 534: RunJobThread.run
[2017-06-01 18:57:08,810: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:57:08,933: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:57:09,161: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:57:09,203: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:57:10,159: INFO/job_scheduler] Job 531 finished 1 steps successfully
[2017-06-01 18:57:10,279: INFO/job_scheduler] Job 531: complete_job: Updating cache
[2017-06-01 18:57:10,279: INFO/job_scheduler] Job 531 completing (errored=False, cancelled=False)
[2017-06-01 18:57:10,329: INFO/job_scheduler] Job 532 finished 1 steps successfully
[2017-06-01 18:57:10,354: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:57:10,479: INFO/job_scheduler] Job 532: complete_job: Updating cache
[2017-06-01 18:57:10,479: INFO/job_scheduler] Job 532 completing (errored=False, cancelled=False)
[2017-06-01 18:57:10,521: INFO/job_scheduler] Job 533 finished 1 steps successfully
[2017-06-01 18:57:10,531: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:57:10,551: INFO/job_scheduler] Job 533: complete_job: Updating cache
[2017-06-01 18:57:10,551: INFO/job_scheduler] Job 533 completing (errored=False, cancelled=False)
[2017-06-01 18:57:10,595: INFO/job_scheduler] Job 534 finished 1 steps successfully
[2017-06-01 18:57:10,639: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:57:10,661: INFO/job_scheduler] Job 534: complete_job: Updating cache
[2017-06-01 18:57:10,661: INFO/job_scheduler] Job 534 completing (errored=False, cancelled=False)
[2017-06-01 18:57:10,746: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:57:12,584: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:12,586: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:12,604: INFO/command_plan] add_jobs: created Job 535 (Test for host connectivity)
[2017-06-01 18:57:12,661: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:12,661: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:12,720: INFO/command_plan] add_jobs: created Job 536 (Test for host connectivity)
[2017-06-01 18:57:12,730: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:57:12,763: INFO/job_scheduler] Job 536: RunJobThread.run
[2017-06-01 18:57:12,769: INFO/job_scheduler] Job 535: RunJobThread.run
[2017-06-01 18:57:12,792: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:12,792: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:12,829: INFO/command_plan] add_jobs: created Job 537 (Test for host connectivity)
[2017-06-01 18:57:12,861: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:57:12,871: INFO/job_scheduler] Job 537: RunJobThread.run
[2017-06-01 18:57:12,954: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:57:13,003: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:13,005: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:13,042: INFO/command_plan] add_jobs: created Job 538 (Test for host connectivity)
[2017-06-01 18:57:13,077: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:57:13,085: INFO/job_scheduler] Job 538: RunJobThread.run
[2017-06-01 18:57:14,944: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:57:15,050: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:57:15,195: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:57:15,207: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:57:16,283: INFO/job_scheduler] Job 535 finished 1 steps successfully
[2017-06-01 18:57:16,328: INFO/job_scheduler] Job 535: complete_job: Updating cache
[2017-06-01 18:57:16,328: INFO/job_scheduler] Job 535 completing (errored=False, cancelled=False)
[2017-06-01 18:57:16,380: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:57:16,383: INFO/job_scheduler] Job 536 finished 1 steps successfully
[2017-06-01 18:57:16,405: INFO/job_scheduler] Job 536: complete_job: Updating cache
[2017-06-01 18:57:16,408: INFO/job_scheduler] Job 536 completing (errored=False, cancelled=False)
[2017-06-01 18:57:16,461: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:57:16,600: INFO/job_scheduler] Job 537 finished 1 steps successfully
[2017-06-01 18:57:16,636: INFO/job_scheduler] Job 538 finished 1 steps successfully
[2017-06-01 18:57:16,670: INFO/job_scheduler] Job 537: complete_job: Updating cache
[2017-06-01 18:57:16,670: INFO/job_scheduler] Job 537 completing (errored=False, cancelled=False)
[2017-06-01 18:57:16,733: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:57:16,756: INFO/job_scheduler] Job 538: complete_job: Updating cache
[2017-06-01 18:57:16,756: INFO/job_scheduler] Job 538 completing (errored=False, cancelled=False)
[2017-06-01 18:57:16,805: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:57:18,685: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:18,686: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:18,725: INFO/command_plan] add_jobs: created Job 539 (Test for host connectivity)
[2017-06-01 18:57:18,774: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:57:18,783: INFO/job_scheduler] Job 539: RunJobThread.run
[2017-06-01 18:57:18,827: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:18,832: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:18,854: INFO/command_plan] add_jobs: created Job 540 (Test for host connectivity)
[2017-06-01 18:57:18,950: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:18,950: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:19,000: INFO/command_plan] add_jobs: created Job 541 (Test for host connectivity)
[2017-06-01 18:57:19,019: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 18:57:19,040: INFO/job_scheduler] Job 540: RunJobThread.run
[2017-06-01 18:57:19,041: INFO/job_scheduler] Job 541: RunJobThread.run
[2017-06-01 18:57:19,058: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:57:19,105: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:19,105: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:19,145: INFO/command_plan] add_jobs: created Job 542 (Test for host connectivity)
[2017-06-01 18:57:19,160: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:57:19,184: INFO/job_scheduler] Job 542: RunJobThread.run
[2017-06-01 18:57:21,092: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:57:21,205: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:57:21,417: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:57:21,507: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:57:22,407: INFO/job_scheduler] Job 539 finished 1 steps successfully
[2017-06-01 18:57:22,458: INFO/job_scheduler] Job 539: complete_job: Updating cache
[2017-06-01 18:57:22,458: INFO/job_scheduler] Job 539 completing (errored=False, cancelled=False)
[2017-06-01 18:57:22,503: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:57:22,560: INFO/job_scheduler] Job 540 finished 1 steps successfully
[2017-06-01 18:57:22,608: INFO/job_scheduler] Job 540: complete_job: Updating cache
[2017-06-01 18:57:22,609: INFO/job_scheduler] Job 540 completing (errored=False, cancelled=False)
[2017-06-01 18:57:22,667: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:57:22,831: INFO/job_scheduler] Job 541 finished 1 steps successfully
[2017-06-01 18:57:22,875: INFO/job_scheduler] Job 541: complete_job: Updating cache
[2017-06-01 18:57:22,875: INFO/job_scheduler] Job 541 completing (errored=False, cancelled=False)
[2017-06-01 18:57:22,878: INFO/job_scheduler] Job 542 finished 1 steps successfully
[2017-06-01 18:57:22,916: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:57:22,924: INFO/job_scheduler] Job 542: complete_job: Updating cache
[2017-06-01 18:57:22,925: INFO/job_scheduler] Job 542 completing (errored=False, cancelled=False)
[2017-06-01 18:57:22,996: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:57:24,899: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:24,899: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:24,919: INFO/command_plan] add_jobs: created Job 543 (Test for host connectivity)
[2017-06-01 18:57:24,964: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:57:24,967: INFO/job_scheduler] Job 543: RunJobThread.run
[2017-06-01 18:57:25,013: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:25,026: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:25,043: INFO/command_plan] add_jobs: created Job 544 (Test for host connectivity)
[2017-06-01 18:57:25,112: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:57:25,118: INFO/job_scheduler] Job 544: RunJobThread.run
[2017-06-01 18:57:25,151: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:25,160: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:25,183: INFO/command_plan] add_jobs: created Job 545 (Test for host connectivity)
[2017-06-01 18:57:25,210: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:57:25,240: INFO/job_scheduler] Job 545: RunJobThread.run
[2017-06-01 18:57:25,295: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:25,295: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:25,315: INFO/command_plan] add_jobs: created Job 546 (Test for host connectivity)
[2017-06-01 18:57:25,394: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:57:25,398: INFO/job_scheduler] Job 546: RunJobThread.run
[2017-06-01 18:57:27,224: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:57:27,318: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:57:27,598: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:57:27,654: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:57:28,573: INFO/job_scheduler] Job 543 finished 1 steps successfully
[2017-06-01 18:57:28,625: INFO/job_scheduler] Job 543: complete_job: Updating cache
[2017-06-01 18:57:28,625: INFO/job_scheduler] Job 543 completing (errored=False, cancelled=False)
[2017-06-01 18:57:28,644: INFO/job_scheduler] Job 544 finished 1 steps successfully
[2017-06-01 18:57:28,674: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:57:28,701: INFO/job_scheduler] Job 544: complete_job: Updating cache
[2017-06-01 18:57:28,701: INFO/job_scheduler] Job 544 completing (errored=False, cancelled=False)
[2017-06-01 18:57:28,802: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:57:28,926: INFO/job_scheduler] Job 546 finished 1 steps successfully
[2017-06-01 18:57:28,982: INFO/job_scheduler] Job 546: complete_job: Updating cache
[2017-06-01 18:57:28,982: INFO/job_scheduler] Job 546 completing (errored=False, cancelled=False)
[2017-06-01 18:57:29,024: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:57:29,178: INFO/job_scheduler] Job 545 finished 1 steps successfully
[2017-06-01 18:57:29,241: INFO/job_scheduler] Job 545: complete_job: Updating cache
[2017-06-01 18:57:29,241: INFO/job_scheduler] Job 545 completing (errored=False, cancelled=False)
[2017-06-01 18:57:29,275: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:57:30,992: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:30,992: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:31,008: INFO/command_plan] add_jobs: created Job 547 (Test for host connectivity)
[2017-06-01 18:57:31,081: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:31,082: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:31,090: INFO/command_plan] add_jobs: created Job 548 (Test for host connectivity)
[2017-06-01 18:57:31,100: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:57:31,128: INFO/job_scheduler] Job 547: RunJobThread.run
[2017-06-01 18:57:31,145: INFO/job_scheduler] Job 548: RunJobThread.run
[2017-06-01 18:57:31,169: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:31,169: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:31,196: INFO/command_plan] add_jobs: created Job 549 (Test for host connectivity)
[2017-06-01 18:57:31,212: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:57:31,253: INFO/job_scheduler] Job 549: RunJobThread.run
[2017-06-01 18:57:31,308: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:57:31,358: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:31,358: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:31,376: INFO/command_plan] add_jobs: created Job 550 (Test for host connectivity)
[2017-06-01 18:57:31,415: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:57:31,457: INFO/job_scheduler] Job 550: RunJobThread.run
[2017-06-01 18:57:33,354: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:57:33,451: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:57:33,487: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:57:33,699: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:57:34,693: INFO/job_scheduler] Job 547 finished 1 steps successfully
[2017-06-01 18:57:34,754: INFO/job_scheduler] Job 547: complete_job: Updating cache
[2017-06-01 18:57:34,755: INFO/job_scheduler] Job 547 completing (errored=False, cancelled=False)
[2017-06-01 18:57:34,798: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:57:34,818: INFO/job_scheduler] Job 548 finished 1 steps successfully
[2017-06-01 18:57:34,875: INFO/job_scheduler] Job 548: complete_job: Updating cache
[2017-06-01 18:57:34,875: INFO/job_scheduler] Job 548 completing (errored=False, cancelled=False)
[2017-06-01 18:57:34,891: INFO/job_scheduler] Job 549 finished 1 steps successfully
[2017-06-01 18:57:34,923: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:57:34,948: INFO/job_scheduler] Job 549: complete_job: Updating cache
[2017-06-01 18:57:34,948: INFO/job_scheduler] Job 549 completing (errored=False, cancelled=False)
[2017-06-01 18:57:35,021: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:57:35,036: INFO/job_scheduler] Job 550 finished 1 steps successfully
[2017-06-01 18:57:35,122: INFO/job_scheduler] Job 550: complete_job: Updating cache
[2017-06-01 18:57:35,122: INFO/job_scheduler] Job 550 completing (errored=False, cancelled=False)
[2017-06-01 18:57:35,156: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:57:36,635: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:36,635: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:36,667: INFO/command_plan] add_jobs: created Job 551 (Test for host connectivity)
[2017-06-01 18:57:36,707: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:57:36,724: INFO/job_scheduler] Job 551: RunJobThread.run
[2017-06-01 18:57:36,744: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:36,746: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:36,765: INFO/command_plan] add_jobs: created Job 552 (Test for host connectivity)
[2017-06-01 18:57:36,837: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:57:36,850: INFO/job_scheduler] Job 552: RunJobThread.run
[2017-06-01 18:57:36,902: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:36,902: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:36,917: INFO/command_plan] add_jobs: created Job 553 (Test for host connectivity)
[2017-06-01 18:57:36,961: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:57:36,989: INFO/job_scheduler] Job 553: RunJobThread.run
[2017-06-01 18:57:37,056: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:37,058: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:37,097: INFO/command_plan] add_jobs: created Job 554 (Test for host connectivity)
[2017-06-01 18:57:37,168: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:57:37,190: INFO/job_scheduler] Job 554: RunJobThread.run
[2017-06-01 18:57:39,024: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:57:39,197: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:57:39,328: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:57:39,399: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:57:40,332: INFO/job_scheduler] Job 551 finished 1 steps successfully
[2017-06-01 18:57:40,387: INFO/job_scheduler] Job 551: complete_job: Updating cache
[2017-06-01 18:57:40,387: INFO/job_scheduler] Job 551 completing (errored=False, cancelled=False)
[2017-06-01 18:57:40,446: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:57:40,554: INFO/job_scheduler] Job 552 finished 1 steps successfully
[2017-06-01 18:57:40,612: INFO/job_scheduler] Job 552: complete_job: Updating cache
[2017-06-01 18:57:40,612: INFO/job_scheduler] Job 552 completing (errored=False, cancelled=False)
[2017-06-01 18:57:40,654: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:57:40,703: INFO/job_scheduler] Job 553 finished 1 steps successfully
[2017-06-01 18:57:40,739: INFO/job_scheduler] Job 554 finished 1 steps successfully
[2017-06-01 18:57:40,753: INFO/job_scheduler] Job 553: complete_job: Updating cache
[2017-06-01 18:57:40,753: INFO/job_scheduler] Job 553 completing (errored=False, cancelled=False)
[2017-06-01 18:57:40,795: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:57:40,813: INFO/job_scheduler] Job 554: complete_job: Updating cache
[2017-06-01 18:57:40,813: INFO/job_scheduler] Job 554 completing (errored=False, cancelled=False)
[2017-06-01 18:57:40,872: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:57:42,940: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:42,940: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:42,953: INFO/command_plan] add_jobs: created Job 555 (Test for host connectivity)
[2017-06-01 18:57:42,988: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:57:42,991: INFO/job_scheduler] Job 555: RunJobThread.run
[2017-06-01 18:57:43,037: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:43,037: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:43,067: INFO/command_plan] add_jobs: created Job 556 (Test for host connectivity)
[2017-06-01 18:57:43,123: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:57:43,133: INFO/job_scheduler] Job 556: RunJobThread.run
[2017-06-01 18:57:43,192: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:43,193: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:43,229: INFO/command_plan] add_jobs: created Job 557 (Test for host connectivity)
[2017-06-01 18:57:43,298: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:57:43,308: INFO/job_scheduler] Job 557: RunJobThread.run
[2017-06-01 18:57:43,333: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:43,333: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:43,354: INFO/command_plan] add_jobs: created Job 558 (Test for host connectivity)
[2017-06-01 18:57:43,391: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:57:43,396: INFO/job_scheduler] Job 558: RunJobThread.run
[2017-06-01 18:57:45,252: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:57:45,519: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:57:45,663: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:57:45,709: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:57:46,596: INFO/job_scheduler] Job 555 finished 1 steps successfully
[2017-06-01 18:57:46,652: INFO/job_scheduler] Job 555: complete_job: Updating cache
[2017-06-01 18:57:46,652: INFO/job_scheduler] Job 555 completing (errored=False, cancelled=False)
[2017-06-01 18:57:46,706: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:57:46,866: INFO/job_scheduler] Job 556 finished 1 steps successfully
[2017-06-01 18:57:46,913: INFO/job_scheduler] Job 556: complete_job: Updating cache
[2017-06-01 18:57:46,913: INFO/job_scheduler] Job 556 completing (errored=False, cancelled=False)
[2017-06-01 18:57:46,978: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:57:47,142: INFO/job_scheduler] Job 558 finished 1 steps successfully
[2017-06-01 18:57:47,160: INFO/job_scheduler] Job 558: complete_job: Updating cache
[2017-06-01 18:57:47,161: INFO/job_scheduler] Job 558 completing (errored=False, cancelled=False)
[2017-06-01 18:57:47,167: INFO/job_scheduler] Job 557 finished 1 steps successfully
[2017-06-01 18:57:47,194: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:57:47,221: INFO/job_scheduler] Job 557: complete_job: Updating cache
[2017-06-01 18:57:47,221: INFO/job_scheduler] Job 557 completing (errored=False, cancelled=False)
[2017-06-01 18:57:47,311: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:57:49,210: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:49,210: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:49,227: INFO/command_plan] add_jobs: created Job 559 (Test for host connectivity)
[2017-06-01 18:57:49,283: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:49,283: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:49,318: INFO/command_plan] add_jobs: created Job 560 (Test for host connectivity)
[2017-06-01 18:57:49,328: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:57:49,332: INFO/job_scheduler] Job 560: RunJobThread.run
[2017-06-01 18:57:49,347: INFO/job_scheduler] Job 559: RunJobThread.run
[2017-06-01 18:57:49,358: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:57:49,432: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:49,432: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:49,554: INFO/command_plan] add_jobs: created Job 561 (Test for host connectivity)
[2017-06-01 18:57:49,583: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:57:49,591: INFO/job_scheduler] Job 561: RunJobThread.run
[2017-06-01 18:57:49,664: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:49,665: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:49,679: INFO/command_plan] add_jobs: created Job 562 (Test for host connectivity)
[2017-06-01 18:57:49,710: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:57:49,724: INFO/job_scheduler] Job 562: RunJobThread.run
[2017-06-01 18:57:51,592: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:57:51,788: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:57:51,895: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:57:51,994: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:57:52,986: INFO/job_scheduler] Job 560 finished 1 steps successfully
[2017-06-01 18:57:53,051: INFO/job_scheduler] Job 560: complete_job: Updating cache
[2017-06-01 18:57:53,051: INFO/job_scheduler] Job 560 completing (errored=False, cancelled=False)
[2017-06-01 18:57:53,093: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:57:53,102: INFO/job_scheduler] Job 559 finished 1 steps successfully
[2017-06-01 18:57:53,110: INFO/job_scheduler] Job 559: complete_job: Updating cache
[2017-06-01 18:57:53,113: INFO/job_scheduler] Job 559 completing (errored=False, cancelled=False)
[2017-06-01 18:57:53,202: INFO/job_scheduler] Job 562 finished 1 steps successfully
[2017-06-01 18:57:53,239: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:57:53,292: INFO/job_scheduler] Job 562: complete_job: Updating cache
[2017-06-01 18:57:53,293: INFO/job_scheduler] Job 562 completing (errored=False, cancelled=False)
[2017-06-01 18:57:53,360: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:57:53,364: INFO/job_scheduler] Job 561 finished 1 steps successfully
[2017-06-01 18:57:53,381: INFO/job_scheduler] Job 561: complete_job: Updating cache
[2017-06-01 18:57:53,381: INFO/job_scheduler] Job 561 completing (errored=False, cancelled=False)
[2017-06-01 18:57:53,470: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:57:55,610: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:55,611: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:55,626: INFO/command_plan] add_jobs: created Job 563 (Test for host connectivity)
[2017-06-01 18:57:55,636: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:57:55,640: INFO/job_scheduler] Job 563: RunJobThread.run
[2017-06-01 18:57:55,720: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:55,721: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:55,744: INFO/command_plan] add_jobs: created Job 564 (Test for host connectivity)
[2017-06-01 18:57:55,821: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:57:55,836: INFO/job_scheduler] Job 564: RunJobThread.run
[2017-06-01 18:57:55,876: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:55,878: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:55,914: INFO/command_plan] add_jobs: created Job 565 (Test for host connectivity)
[2017-06-01 18:57:56,012: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:57:56,021: INFO/job_scheduler] Job 565: RunJobThread.run
[2017-06-01 18:57:56,054: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:57:56,054: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:57:56,068: INFO/command_plan] add_jobs: created Job 566 (Test for host connectivity)
[2017-06-01 18:57:56,122: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:57:56,124: INFO/job_scheduler] Job 566: RunJobThread.run
[2017-06-01 18:57:57,989: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:57:58,122: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:57:58,210: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:57:58,408: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:57:59,259: INFO/job_scheduler] Job 563 finished 1 steps successfully
[2017-06-01 18:57:59,320: INFO/job_scheduler] Job 563: complete_job: Updating cache
[2017-06-01 18:57:59,320: INFO/job_scheduler] Job 563 completing (errored=False, cancelled=False)
[2017-06-01 18:57:59,390: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:57:59,459: INFO/job_scheduler] Job 564 finished 1 steps successfully
[2017-06-01 18:57:59,501: INFO/job_scheduler] Job 564: complete_job: Updating cache
[2017-06-01 18:57:59,502: INFO/job_scheduler] Job 564 completing (errored=False, cancelled=False)
[2017-06-01 18:57:59,600: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:57:59,628: INFO/job_scheduler] Job 565 finished 1 steps successfully
[2017-06-01 18:57:59,655: INFO/job_scheduler] Job 565: complete_job: Updating cache
[2017-06-01 18:57:59,656: INFO/job_scheduler] Job 565 completing (errored=False, cancelled=False)
[2017-06-01 18:57:59,688: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:57:59,825: INFO/job_scheduler] Job 566 finished 1 steps successfully
[2017-06-01 18:57:59,847: INFO/job_scheduler] Job 566: complete_job: Updating cache
[2017-06-01 18:57:59,847: INFO/job_scheduler] Job 566 completing (errored=False, cancelled=False)
[2017-06-01 18:57:59,888: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:58:01,903: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:01,903: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:01,914: INFO/command_plan] add_jobs: created Job 567 (Test for host connectivity)
[2017-06-01 18:58:01,930: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:58:01,934: INFO/job_scheduler] Job 567: RunJobThread.run
[2017-06-01 18:58:01,989: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:01,989: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:02,023: INFO/command_plan] add_jobs: created Job 568 (Test for host connectivity)
[2017-06-01 18:58:02,085: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:58:02,092: INFO/job_scheduler] Job 568: RunJobThread.run
[2017-06-01 18:58:02,135: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:02,135: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:02,196: INFO/command_plan] add_jobs: created Job 569 (Test for host connectivity)
[2017-06-01 18:58:02,225: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:58:02,247: INFO/job_scheduler] Job 569: RunJobThread.run
[2017-06-01 18:58:02,283: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:02,284: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:02,354: INFO/command_plan] add_jobs: created Job 570 (Test for host connectivity)
[2017-06-01 18:58:02,405: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:58:02,434: INFO/job_scheduler] Job 570: RunJobThread.run
[2017-06-01 18:58:04,302: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:58:04,331: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:58:04,560: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:58:04,690: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:58:05,680: INFO/job_scheduler] Job 568 finished 1 steps successfully
[2017-06-01 18:58:05,740: INFO/job_scheduler] Job 567 finished 1 steps successfully
[2017-06-01 18:58:05,745: INFO/job_scheduler] Job 568: complete_job: Updating cache
[2017-06-01 18:58:05,745: INFO/job_scheduler] Job 568 completing (errored=False, cancelled=False)
[2017-06-01 18:58:05,779: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:58:05,796: INFO/job_scheduler] Job 567: complete_job: Updating cache
[2017-06-01 18:58:05,798: INFO/job_scheduler] Job 567 completing (errored=False, cancelled=False)
[2017-06-01 18:58:05,937: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:58:05,970: INFO/job_scheduler] Job 569 finished 1 steps successfully
[2017-06-01 18:58:05,991: INFO/job_scheduler] Job 570 finished 1 steps successfully
[2017-06-01 18:58:06,038: INFO/job_scheduler] Job 569: complete_job: Updating cache
[2017-06-01 18:58:06,038: INFO/job_scheduler] Job 569 completing (errored=False, cancelled=False)
[2017-06-01 18:58:06,087: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:58:06,139: INFO/job_scheduler] Job 570: complete_job: Updating cache
[2017-06-01 18:58:06,139: INFO/job_scheduler] Job 570 completing (errored=False, cancelled=False)
[2017-06-01 18:58:06,187: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:58:08,214: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:08,215: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:08,229: INFO/command_plan] add_jobs: created Job 571 (Test for host connectivity)
[2017-06-01 18:58:08,264: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:08,265: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:08,307: INFO/command_plan] add_jobs: created Job 572 (Test for host connectivity)
[2017-06-01 18:58:08,329: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:58:08,353: INFO/job_scheduler] Job 571: RunJobThread.run
[2017-06-01 18:58:08,354: INFO/job_scheduler] Job 572: RunJobThread.run
[2017-06-01 18:58:08,390: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:08,391: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:08,420: INFO/command_plan] add_jobs: created Job 573 (Test for host connectivity)
[2017-06-01 18:58:08,459: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:58:08,501: INFO/job_scheduler] Job 573: RunJobThread.run
[2017-06-01 18:58:08,573: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:08,581: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:08,661: INFO/command_plan] add_jobs: created Job 574 (Test for host connectivity)
[2017-06-01 18:58:08,701: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:58:08,719: INFO/job_scheduler] Job 574: RunJobThread.run
[2017-06-01 18:58:08,745: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 18:58:10,577: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:58:10,637: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:58:10,819: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:58:10,958: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:58:11,903: INFO/job_scheduler] Job 571 finished 1 steps successfully
[2017-06-01 18:58:11,972: INFO/job_scheduler] Job 571: complete_job: Updating cache
[2017-06-01 18:58:11,972: INFO/job_scheduler] Job 571 completing (errored=False, cancelled=False)
[2017-06-01 18:58:12,010: INFO/job_scheduler] Job 572 finished 1 steps successfully
[2017-06-01 18:58:12,036: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:58:12,079: INFO/job_scheduler] Job 572: complete_job: Updating cache
[2017-06-01 18:58:12,079: INFO/job_scheduler] Job 572 completing (errored=False, cancelled=False)
[2017-06-01 18:58:12,158: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:58:12,265: INFO/job_scheduler] Job 573 finished 1 steps successfully
[2017-06-01 18:58:12,276: INFO/job_scheduler] Job 574 finished 1 steps successfully
[2017-06-01 18:58:12,311: INFO/job_scheduler] Job 573: complete_job: Updating cache
[2017-06-01 18:58:12,311: INFO/job_scheduler] Job 573 completing (errored=False, cancelled=False)
[2017-06-01 18:58:12,347: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:58:12,363: INFO/job_scheduler] Job 574: complete_job: Updating cache
[2017-06-01 18:58:12,363: INFO/job_scheduler] Job 574 completing (errored=False, cancelled=False)
[2017-06-01 18:58:12,441: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:58:14,504: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:14,505: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:14,519: INFO/command_plan] add_jobs: created Job 575 (Test for host connectivity)
[2017-06-01 18:58:14,558: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:58:14,567: INFO/job_scheduler] Job 575: RunJobThread.run
[2017-06-01 18:58:14,611: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:14,612: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:14,627: INFO/command_plan] add_jobs: created Job 576 (Test for host connectivity)
[2017-06-01 18:58:14,659: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:58:14,668: INFO/job_scheduler] Job 576: RunJobThread.run
[2017-06-01 18:58:14,703: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:14,719: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:14,775: INFO/command_plan] add_jobs: created Job 577 (Test for host connectivity)
[2017-06-01 18:58:14,936: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:58:14,938: INFO/job_scheduler] Job 577: RunJobThread.run
[2017-06-01 18:58:14,989: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:14,990: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:15,003: INFO/command_plan] add_jobs: created Job 578 (Test for host connectivity)
[2017-06-01 18:58:15,025: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:58:15,065: INFO/job_scheduler] Job 578: RunJobThread.run
[2017-06-01 18:58:16,950: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:58:16,977: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:58:17,207: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:58:17,368: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:58:18,320: INFO/job_scheduler] Job 576 finished 1 steps successfully
[2017-06-01 18:58:18,327: INFO/job_scheduler] Job 575 finished 1 steps successfully
[2017-06-01 18:58:18,385: INFO/job_scheduler] Job 576: complete_job: Updating cache
[2017-06-01 18:58:18,385: INFO/job_scheduler] Job 576 completing (errored=False, cancelled=False)
[2017-06-01 18:58:18,445: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:58:18,451: INFO/job_scheduler] Job 575: complete_job: Updating cache
[2017-06-01 18:58:18,451: INFO/job_scheduler] Job 575 completing (errored=False, cancelled=False)
[2017-06-01 18:58:18,529: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:58:18,616: INFO/job_scheduler] Job 577 finished 1 steps successfully
[2017-06-01 18:58:18,651: INFO/job_scheduler] Job 577: complete_job: Updating cache
[2017-06-01 18:58:18,651: INFO/job_scheduler] Job 577 completing (errored=False, cancelled=False)
[2017-06-01 18:58:18,726: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:58:18,828: INFO/job_scheduler] Job 578 finished 1 steps successfully
[2017-06-01 18:58:18,984: INFO/job_scheduler] Job 578: complete_job: Updating cache
[2017-06-01 18:58:18,985: INFO/job_scheduler] Job 578 completing (errored=False, cancelled=False)
[2017-06-01 18:58:19,020: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:58:20,822: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:20,822: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:20,840: INFO/command_plan] add_jobs: created Job 579 (Test for host connectivity)
[2017-06-01 18:58:20,890: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:58:20,905: INFO/job_scheduler] Job 579: RunJobThread.run
[2017-06-01 18:58:20,948: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:20,950: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:20,961: INFO/command_plan] add_jobs: created Job 580 (Test for host connectivity)
[2017-06-01 18:58:21,033: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:58:21,052: INFO/job_scheduler] Job 580: RunJobThread.run
[2017-06-01 18:58:21,075: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:21,075: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:21,084: INFO/command_plan] add_jobs: created Job 581 (Test for host connectivity)
[2017-06-01 18:58:21,111: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:58:21,118: INFO/job_scheduler] Job 581: RunJobThread.run
[2017-06-01 18:58:21,278: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:21,279: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:21,334: INFO/command_plan] add_jobs: created Job 582 (Test for host connectivity)
[2017-06-01 18:58:21,372: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:58:21,396: INFO/job_scheduler] Job 582: RunJobThread.run
[2017-06-01 18:58:23,392: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:58:23,443: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:58:23,494: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:58:23,669: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:58:24,766: INFO/job_scheduler] Job 579 finished 1 steps successfully
[2017-06-01 18:58:24,791: INFO/job_scheduler] Job 579: complete_job: Updating cache
[2017-06-01 18:58:24,792: INFO/job_scheduler] Job 579 completing (errored=False, cancelled=False)
[2017-06-01 18:58:24,795: INFO/job_scheduler] Job 580 finished 1 steps successfully
[2017-06-01 18:58:24,833: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:58:24,850: INFO/job_scheduler] Job 580: complete_job: Updating cache
[2017-06-01 18:58:24,851: INFO/job_scheduler] Job 580 completing (errored=False, cancelled=False)
[2017-06-01 18:58:24,913: INFO/job_scheduler] Job 582 finished 1 steps successfully
[2017-06-01 18:58:24,913: INFO/job_scheduler] Job 581 finished 1 steps successfully
[2017-06-01 18:58:24,982: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:58:25,004: INFO/job_scheduler] Job 582: complete_job: Updating cache
[2017-06-01 18:58:25,004: INFO/job_scheduler] Job 582 completing (errored=False, cancelled=False)
[2017-06-01 18:58:25,067: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:58:25,085: INFO/job_scheduler] Job 581: complete_job: Updating cache
[2017-06-01 18:58:25,085: INFO/job_scheduler] Job 581 completing (errored=False, cancelled=False)
[2017-06-01 18:58:25,208: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:58:27,156: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:27,157: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:27,191: INFO/command_plan] add_jobs: created Job 583 (Test for host connectivity)
[2017-06-01 18:58:27,217: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:58:27,218: INFO/job_scheduler] Job 583: RunJobThread.run
[2017-06-01 18:58:27,280: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:27,280: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:27,319: INFO/command_plan] add_jobs: created Job 584 (Test for host connectivity)
[2017-06-01 18:58:27,376: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:58:27,383: INFO/job_scheduler] Job 584: RunJobThread.run
[2017-06-01 18:58:27,414: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:27,415: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:27,442: INFO/command_plan] add_jobs: created Job 585 (Test for host connectivity)
[2017-06-01 18:58:27,491: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:58:27,537: INFO/job_scheduler] Job 585: RunJobThread.run
[2017-06-01 18:58:27,568: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:27,572: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:27,617: INFO/command_plan] add_jobs: created Job 586 (Test for host connectivity)
[2017-06-01 18:58:27,643: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:58:27,664: INFO/job_scheduler] Job 586: RunJobThread.run
[2017-06-01 18:58:29,455: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:58:29,667: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:58:29,804: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:58:29,973: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:58:30,814: INFO/job_scheduler] Job 583 finished 1 steps successfully
[2017-06-01 18:58:30,866: INFO/job_scheduler] Job 583: complete_job: Updating cache
[2017-06-01 18:58:30,866: INFO/job_scheduler] Job 583 completing (errored=False, cancelled=False)
[2017-06-01 18:58:30,949: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:58:31,022: INFO/job_scheduler] Job 584 finished 1 steps successfully
[2017-06-01 18:58:31,074: INFO/job_scheduler] Job 584: complete_job: Updating cache
[2017-06-01 18:58:31,074: INFO/job_scheduler] Job 584 completing (errored=False, cancelled=False)
[2017-06-01 18:58:31,135: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:58:31,189: INFO/job_scheduler] Job 585 finished 1 steps successfully
[2017-06-01 18:58:31,232: INFO/job_scheduler] Job 585: complete_job: Updating cache
[2017-06-01 18:58:31,233: INFO/job_scheduler] Job 585 completing (errored=False, cancelled=False)
[2017-06-01 18:58:31,276: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:58:31,382: INFO/job_scheduler] Job 586 finished 1 steps successfully
[2017-06-01 18:58:31,432: INFO/job_scheduler] Job 586: complete_job: Updating cache
[2017-06-01 18:58:31,432: INFO/job_scheduler] Job 586 completing (errored=False, cancelled=False)
[2017-06-01 18:58:31,492: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:58:33,407: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:33,409: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:33,425: INFO/command_plan] add_jobs: created Job 587 (Test for host connectivity)
[2017-06-01 18:58:33,473: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:33,473: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:33,489: INFO/command_plan] add_jobs: created Job 588 (Test for host connectivity)
[2017-06-01 18:58:33,513: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:58:33,518: INFO/job_scheduler] Job 587: RunJobThread.run
[2017-06-01 18:58:33,536: INFO/job_scheduler] Job 588: RunJobThread.run
[2017-06-01 18:58:33,541: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:58:33,595: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:33,596: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:33,682: INFO/command_plan] add_jobs: created Job 589 (Test for host connectivity)
[2017-06-01 18:58:33,799: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:33,799: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:33,818: INFO/command_plan] add_jobs: created Job 590 (Test for host connectivity)
[2017-06-01 18:58:33,842: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 18:58:33,876: INFO/job_scheduler] Job 589: RunJobThread.run
[2017-06-01 18:58:33,891: INFO/job_scheduler] Job 590: RunJobThread.run
[2017-06-01 18:58:33,907: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 18:58:35,790: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:58:35,839: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:58:36,097: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:58:36,261: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:58:37,113: INFO/job_scheduler] Job 588 finished 1 steps successfully
[2017-06-01 18:58:37,139: INFO/job_scheduler] Job 588: complete_job: Updating cache
[2017-06-01 18:58:37,139: INFO/job_scheduler] Job 588 completing (errored=False, cancelled=False)
[2017-06-01 18:58:37,186: INFO/job_scheduler] Job 587 finished 1 steps successfully
[2017-06-01 18:58:37,215: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:58:37,236: INFO/job_scheduler] Job 587: complete_job: Updating cache
[2017-06-01 18:58:37,236: INFO/job_scheduler] Job 587 completing (errored=False, cancelled=False)
[2017-06-01 18:58:37,348: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:58:37,424: INFO/job_scheduler] Job 590 finished 1 steps successfully
[2017-06-01 18:58:37,449: INFO/job_scheduler] Job 590: complete_job: Updating cache
[2017-06-01 18:58:37,449: INFO/job_scheduler] Job 590 completing (errored=False, cancelled=False)
[2017-06-01 18:58:37,509: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:58:37,639: INFO/job_scheduler] Job 589 finished 1 steps successfully
[2017-06-01 18:58:37,706: INFO/job_scheduler] Job 589: complete_job: Updating cache
[2017-06-01 18:58:37,706: INFO/job_scheduler] Job 589 completing (errored=False, cancelled=False)
[2017-06-01 18:58:37,740: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:58:39,717: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:39,717: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:39,730: INFO/command_plan] add_jobs: created Job 591 (Test for host connectivity)
[2017-06-01 18:58:39,751: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:58:39,752: INFO/job_scheduler] Job 591: RunJobThread.run
[2017-06-01 18:58:39,790: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:39,790: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:39,834: INFO/command_plan] add_jobs: created Job 592 (Test for host connectivity)
[2017-06-01 18:58:39,871: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:58:39,894: INFO/job_scheduler] Job 592: RunJobThread.run
[2017-06-01 18:58:39,941: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:39,941: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:39,965: INFO/command_plan] add_jobs: created Job 593 (Test for host connectivity)
[2017-06-01 18:58:40,040: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:40,040: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:40,101: INFO/command_plan] add_jobs: created Job 594 (Test for host connectivity)
[2017-06-01 18:58:40,132: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 18:58:40,136: INFO/job_scheduler] Job 593: RunJobThread.run
[2017-06-01 18:58:40,136: INFO/job_scheduler] Job 594: RunJobThread.run
[2017-06-01 18:58:40,148: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 18:58:42,033: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:58:42,255: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:58:42,364: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:58:42,536: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:58:43,290: INFO/job_scheduler] Job 591 finished 1 steps successfully
[2017-06-01 18:58:43,313: INFO/job_scheduler] Job 591: complete_job: Updating cache
[2017-06-01 18:58:43,313: INFO/job_scheduler] Job 591 completing (errored=False, cancelled=False)
[2017-06-01 18:58:43,401: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:58:43,583: INFO/job_scheduler] Job 592 finished 1 steps successfully
[2017-06-01 18:58:43,646: INFO/job_scheduler] Job 592: complete_job: Updating cache
[2017-06-01 18:58:43,646: INFO/job_scheduler] Job 592 completing (errored=False, cancelled=False)
[2017-06-01 18:58:43,688: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:58:43,731: INFO/job_scheduler] Job 594 finished 1 steps successfully
[2017-06-01 18:58:43,746: INFO/job_scheduler] Job 594: complete_job: Updating cache
[2017-06-01 18:58:43,747: INFO/job_scheduler] Job 594 completing (errored=False, cancelled=False)
[2017-06-01 18:58:43,816: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:58:43,912: INFO/job_scheduler] Job 593 finished 1 steps successfully
[2017-06-01 18:58:43,971: INFO/job_scheduler] Job 593: complete_job: Updating cache
[2017-06-01 18:58:43,971: INFO/job_scheduler] Job 593 completing (errored=False, cancelled=False)
[2017-06-01 18:58:44,013: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:58:46,020: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:46,020: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:46,054: INFO/command_plan] add_jobs: created Job 595 (Test for host connectivity)
[2017-06-01 18:58:46,069: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:58:46,079: INFO/job_scheduler] Job 595: RunJobThread.run
[2017-06-01 18:58:46,124: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:46,125: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:46,155: INFO/command_plan] add_jobs: created Job 596 (Test for host connectivity)
[2017-06-01 18:58:46,249: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:46,250: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:46,288: INFO/command_plan] add_jobs: created Job 597 (Test for host connectivity)
[2017-06-01 18:58:46,342: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 18:58:46,351: INFO/job_scheduler] Job 596: RunJobThread.run
[2017-06-01 18:58:46,352: INFO/job_scheduler] Job 597: RunJobThread.run
[2017-06-01 18:58:46,364: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:58:46,402: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:46,403: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:46,533: INFO/command_plan] add_jobs: created Job 598 (Test for host connectivity)
[2017-06-01 18:58:46,583: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:58:46,592: INFO/job_scheduler] Job 598: RunJobThread.run
[2017-06-01 18:58:48,355: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:58:48,588: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:58:48,589: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:58:48,808: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:58:49,759: INFO/job_scheduler] Job 595 finished 1 steps successfully
[2017-06-01 18:58:49,812: INFO/job_scheduler] Job 595: complete_job: Updating cache
[2017-06-01 18:58:49,812: INFO/job_scheduler] Job 595 completing (errored=False, cancelled=False)
[2017-06-01 18:58:49,915: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:58:49,971: INFO/job_scheduler] Job 596 finished 1 steps successfully
[2017-06-01 18:58:49,995: INFO/job_scheduler] Job 596: complete_job: Updating cache
[2017-06-01 18:58:49,995: INFO/job_scheduler] Job 596 completing (errored=False, cancelled=False)
[2017-06-01 18:58:50,010: INFO/job_scheduler] Job 597 finished 1 steps successfully
[2017-06-01 18:58:50,049: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:58:50,085: INFO/job_scheduler] Job 597: complete_job: Updating cache
[2017-06-01 18:58:50,085: INFO/job_scheduler] Job 597 completing (errored=False, cancelled=False)
[2017-06-01 18:58:50,182: INFO/job_scheduler] Job 598 finished 1 steps successfully
[2017-06-01 18:58:50,195: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:58:50,235: INFO/job_scheduler] Job 598: complete_job: Updating cache
[2017-06-01 18:58:50,235: INFO/job_scheduler] Job 598 completing (errored=False, cancelled=False)
[2017-06-01 18:58:50,279: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:58:52,413: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:52,414: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:52,432: INFO/command_plan] add_jobs: created Job 599 (Test for host connectivity)
[2017-06-01 18:58:52,447: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:58:52,464: INFO/job_scheduler] Job 599: RunJobThread.run
[2017-06-01 18:58:52,509: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:52,510: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:52,539: INFO/command_plan] add_jobs: created Job 600 (Test for host connectivity)
[2017-06-01 18:58:52,566: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:58:52,586: INFO/job_scheduler] Job 600: RunJobThread.run
[2017-06-01 18:58:52,654: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:52,655: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:52,672: INFO/command_plan] add_jobs: created Job 601 (Test for host connectivity)
[2017-06-01 18:58:52,707: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:58:52,746: INFO/job_scheduler] Job 601: RunJobThread.run
[2017-06-01 18:58:52,888: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:52,889: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:52,930: INFO/command_plan] add_jobs: created Job 602 (Test for host connectivity)
[2017-06-01 18:58:53,003: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:58:53,032: INFO/job_scheduler] Job 602: RunJobThread.run
[2017-06-01 18:58:54,829: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:58:54,864: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:58:54,961: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:58:55,314: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:58:56,165: INFO/job_scheduler] Job 599 finished 1 steps successfully
[2017-06-01 18:58:56,212: INFO/job_scheduler] Job 599: complete_job: Updating cache
[2017-06-01 18:58:56,212: INFO/job_scheduler] Job 599 completing (errored=False, cancelled=False)
[2017-06-01 18:58:56,263: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:58:56,274: INFO/job_scheduler] Job 600 finished 1 steps successfully
[2017-06-01 18:58:56,301: INFO/job_scheduler] Job 600: complete_job: Updating cache
[2017-06-01 18:58:56,301: INFO/job_scheduler] Job 600 completing (errored=False, cancelled=False)
[2017-06-01 18:58:56,346: INFO/job_scheduler] Job 601 finished 1 steps successfully
[2017-06-01 18:58:56,349: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:58:56,377: INFO/job_scheduler] Job 601: complete_job: Updating cache
[2017-06-01 18:58:56,377: INFO/job_scheduler] Job 601 completing (errored=False, cancelled=False)
[2017-06-01 18:58:56,485: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:58:56,797: INFO/job_scheduler] Job 602 finished 1 steps successfully
[2017-06-01 18:58:56,829: INFO/job_scheduler] Job 602: complete_job: Updating cache
[2017-06-01 18:58:56,829: INFO/job_scheduler] Job 602 completing (errored=False, cancelled=False)
[2017-06-01 18:58:56,865: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:58:58,764: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:58,765: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:58,780: INFO/command_plan] add_jobs: created Job 603 (Test for host connectivity)
[2017-06-01 18:58:58,842: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:58,842: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:58,871: INFO/command_plan] add_jobs: created Job 604 (Test for host connectivity)
[2017-06-01 18:58:58,885: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:58:58,897: INFO/job_scheduler] Job 603: RunJobThread.run
[2017-06-01 18:58:58,898: INFO/job_scheduler] Job 604: RunJobThread.run
[2017-06-01 18:58:58,915: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:58:58,971: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:58,971: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:59,096: INFO/command_plan] add_jobs: created Job 605 (Test for host connectivity)
[2017-06-01 18:58:59,164: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:58:59,164: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:58:59,179: INFO/command_plan] add_jobs: created Job 606 (Test for host connectivity)
[2017-06-01 18:58:59,205: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 18:58:59,239: INFO/job_scheduler] Job 605: RunJobThread.run
[2017-06-01 18:58:59,248: INFO/job_scheduler] Job 606: RunJobThread.run
[2017-06-01 18:58:59,257: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 18:59:01,194: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:59:01,244: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:59:01,482: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:59:01,516: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:59:02,558: INFO/job_scheduler] Job 603 finished 1 steps successfully
[2017-06-01 18:59:02,609: INFO/job_scheduler] Job 603: complete_job: Updating cache
[2017-06-01 18:59:02,609: INFO/job_scheduler] Job 603 completing (errored=False, cancelled=False)
[2017-06-01 18:59:02,635: INFO/job_scheduler] Job 604 finished 1 steps successfully
[2017-06-01 18:59:02,696: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:59:02,712: INFO/job_scheduler] Job 604: complete_job: Updating cache
[2017-06-01 18:59:02,713: INFO/job_scheduler] Job 604 completing (errored=False, cancelled=False)
[2017-06-01 18:59:02,834: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:59:02,924: INFO/job_scheduler] Job 606 finished 1 steps successfully
[2017-06-01 18:59:02,937: INFO/job_scheduler] Job 605 finished 1 steps successfully
[2017-06-01 18:59:02,945: INFO/job_scheduler] Job 606: complete_job: Updating cache
[2017-06-01 18:59:02,945: INFO/job_scheduler] Job 606 completing (errored=False, cancelled=False)
[2017-06-01 18:59:03,011: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:59:03,028: INFO/job_scheduler] Job 605: complete_job: Updating cache
[2017-06-01 18:59:03,028: INFO/job_scheduler] Job 605 completing (errored=False, cancelled=False)
[2017-06-01 18:59:03,087: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:59:05,527: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:05,527: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:05,554: INFO/command_plan] add_jobs: created Job 607 (Test for host connectivity)
[2017-06-01 18:59:05,605: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:59:05,617: INFO/job_scheduler] Job 607: RunJobThread.run
[2017-06-01 18:59:05,676: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:05,677: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:05,703: INFO/command_plan] add_jobs: created Job 608 (Test for host connectivity)
[2017-06-01 18:59:05,733: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:59:05,741: INFO/job_scheduler] Job 608: RunJobThread.run
[2017-06-01 18:59:05,851: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:05,851: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:05,889: INFO/command_plan] add_jobs: created Job 609 (Test for host connectivity)
[2017-06-01 18:59:05,958: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:05,958: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:06,006: INFO/command_plan] add_jobs: created Job 610 (Test for host connectivity)
[2017-06-01 18:59:06,020: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 18:59:06,024: INFO/job_scheduler] Job 609: RunJobThread.run
[2017-06-01 18:59:06,043: INFO/job_scheduler] Job 610: RunJobThread.run
[2017-06-01 18:59:06,058: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 18:59:07,957: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:59:08,058: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:59:08,337: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:59:08,425: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:59:09,308: INFO/job_scheduler] Job 607 finished 1 steps successfully
[2017-06-01 18:59:09,315: INFO/job_scheduler] Job 608 finished 1 steps successfully
[2017-06-01 18:59:09,327: INFO/job_scheduler] Job 607: complete_job: Updating cache
[2017-06-01 18:59:09,327: INFO/job_scheduler] Job 607 completing (errored=False, cancelled=False)
[2017-06-01 18:59:09,394: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:59:09,462: INFO/job_scheduler] Job 608: complete_job: Updating cache
[2017-06-01 18:59:09,462: INFO/job_scheduler] Job 608 completing (errored=False, cancelled=False)
[2017-06-01 18:59:09,544: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:59:09,634: INFO/job_scheduler] Job 610 finished 1 steps successfully
[2017-06-01 18:59:09,677: INFO/job_scheduler] Job 610: complete_job: Updating cache
[2017-06-01 18:59:09,677: INFO/job_scheduler] Job 610 completing (errored=False, cancelled=False)
[2017-06-01 18:59:09,735: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:59:09,885: INFO/job_scheduler] Job 609 finished 1 steps successfully
[2017-06-01 18:59:09,952: INFO/job_scheduler] Job 609: complete_job: Updating cache
[2017-06-01 18:59:09,952: INFO/job_scheduler] Job 609 completing (errored=False, cancelled=False)
[2017-06-01 18:59:09,986: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:59:11,668: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:11,682: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:11,709: INFO/command_plan] add_jobs: created Job 611 (Test for host connectivity)
[2017-06-01 18:59:11,755: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:11,756: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:11,795: INFO/command_plan] add_jobs: created Job 612 (Test for host connectivity)
[2017-06-01 18:59:11,894: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:59:11,899: INFO/job_scheduler] Job 611: RunJobThread.run
[2017-06-01 18:59:11,900: INFO/job_scheduler] Job 612: RunJobThread.run
[2017-06-01 18:59:11,919: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:59:11,972: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:11,972: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:12,021: INFO/command_plan] add_jobs: created Job 613 (Test for host connectivity)
[2017-06-01 18:59:12,129: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:59:12,137: INFO/job_scheduler] Job 613: RunJobThread.run
[2017-06-01 18:59:12,169: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:12,169: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:12,221: INFO/command_plan] add_jobs: created Job 614 (Test for host connectivity)
[2017-06-01 18:59:12,244: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:59:12,257: INFO/job_scheduler] Job 614: RunJobThread.run
[2017-06-01 18:59:14,126: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:59:14,210: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:59:14,482: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:59:14,564: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:59:15,477: INFO/job_scheduler] Job 612 finished 1 steps successfully
[2017-06-01 18:59:15,510: INFO/job_scheduler] Job 612: complete_job: Updating cache
[2017-06-01 18:59:15,510: INFO/job_scheduler] Job 612 completing (errored=False, cancelled=False)
[2017-06-01 18:59:15,568: INFO/job_scheduler] Job 611 finished 1 steps successfully
[2017-06-01 18:59:15,576: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:59:15,684: INFO/job_scheduler] Job 611: complete_job: Updating cache
[2017-06-01 18:59:15,687: INFO/job_scheduler] Job 611 completing (errored=False, cancelled=False)
[2017-06-01 18:59:15,743: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:59:15,931: INFO/job_scheduler] Job 614 finished 1 steps successfully
[2017-06-01 18:59:15,972: INFO/job_scheduler] Job 613 finished 1 steps successfully
[2017-06-01 18:59:15,975: INFO/job_scheduler] Job 614: complete_job: Updating cache
[2017-06-01 18:59:15,975: INFO/job_scheduler] Job 614 completing (errored=False, cancelled=False)
[2017-06-01 18:59:16,054: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:59:16,069: INFO/job_scheduler] Job 613: complete_job: Updating cache
[2017-06-01 18:59:16,069: INFO/job_scheduler] Job 613 completing (errored=False, cancelled=False)
[2017-06-01 18:59:16,150: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:59:17,889: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:17,889: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:17,901: INFO/command_plan] add_jobs: created Job 615 (Test for host connectivity)
[2017-06-01 18:59:17,969: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:59:17,972: INFO/job_scheduler] Job 615: RunJobThread.run
[2017-06-01 18:59:18,010: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:18,014: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:18,034: INFO/command_plan] add_jobs: created Job 616 (Test for host connectivity)
[2017-06-01 18:59:18,102: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:18,102: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:18,175: INFO/command_plan] add_jobs: created Job 617 (Test for host connectivity)
[2017-06-01 18:59:18,192: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 18:59:18,216: INFO/job_scheduler] Job 616: RunJobThread.run
[2017-06-01 18:59:18,234: INFO/job_scheduler] Job 617: RunJobThread.run
[2017-06-01 18:59:18,259: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:18,260: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:18,276: INFO/command_plan] add_jobs: created Job 618 (Test for host connectivity)
[2017-06-01 18:59:18,290: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:59:18,324: INFO/job_scheduler] Job 618: RunJobThread.run
[2017-06-01 18:59:18,428: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 18:59:20,174: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:59:20,417: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:59:20,512: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:59:20,735: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:59:21,534: INFO/job_scheduler] Job 615 finished 1 steps successfully
[2017-06-01 18:59:21,599: INFO/job_scheduler] Job 615: complete_job: Updating cache
[2017-06-01 18:59:21,600: INFO/job_scheduler] Job 615 completing (errored=False, cancelled=False)
[2017-06-01 18:59:21,643: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:59:21,716: INFO/job_scheduler] Job 616 finished 1 steps successfully
[2017-06-01 18:59:21,751: INFO/job_scheduler] Job 616: complete_job: Updating cache
[2017-06-01 18:59:21,751: INFO/job_scheduler] Job 616 completing (errored=False, cancelled=False)
[2017-06-01 18:59:21,818: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:59:21,845: INFO/job_scheduler] Job 617 finished 1 steps successfully
[2017-06-01 18:59:21,896: INFO/job_scheduler] Job 617: complete_job: Updating cache
[2017-06-01 18:59:21,896: INFO/job_scheduler] Job 617 completing (errored=False, cancelled=False)
[2017-06-01 18:59:21,952: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:59:22,147: INFO/job_scheduler] Job 618 finished 1 steps successfully
[2017-06-01 18:59:22,191: INFO/job_scheduler] Job 618: complete_job: Updating cache
[2017-06-01 18:59:22,191: INFO/job_scheduler] Job 618 completing (errored=False, cancelled=False)
[2017-06-01 18:59:22,224: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:59:23,932: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:23,932: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:23,949: INFO/command_plan] add_jobs: created Job 619 (Test for host connectivity)
[2017-06-01 18:59:23,987: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:23,987: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:24,017: INFO/command_plan] add_jobs: created Job 620 (Test for host connectivity)
[2017-06-01 18:59:24,044: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:59:24,057: INFO/job_scheduler] Job 619: RunJobThread.run
[2017-06-01 18:59:24,086: INFO/job_scheduler] Job 620: RunJobThread.run
[2017-06-01 18:59:24,100: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:59:24,136: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:24,136: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:24,175: INFO/command_plan] add_jobs: created Job 621 (Test for host connectivity)
[2017-06-01 18:59:24,251: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:59:24,254: INFO/job_scheduler] Job 621: RunJobThread.run
[2017-06-01 18:59:24,315: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:24,315: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:24,324: INFO/command_plan] add_jobs: created Job 622 (Test for host connectivity)
[2017-06-01 18:59:24,407: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:59:24,419: INFO/job_scheduler] Job 622: RunJobThread.run
[2017-06-01 18:59:26,311: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:59:26,319: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:59:26,475: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:59:26,577: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:59:27,673: INFO/job_scheduler] Job 620 finished 1 steps successfully
[2017-06-01 18:59:27,678: INFO/job_scheduler] Job 619 finished 1 steps successfully
[2017-06-01 18:59:27,731: INFO/job_scheduler] Job 620: complete_job: Updating cache
[2017-06-01 18:59:27,732: INFO/job_scheduler] Job 620 completing (errored=False, cancelled=False)
[2017-06-01 18:59:27,774: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:59:27,791: INFO/job_scheduler] Job 619: complete_job: Updating cache
[2017-06-01 18:59:27,793: INFO/job_scheduler] Job 619 completing (errored=False, cancelled=False)
[2017-06-01 18:59:27,829: INFO/job_scheduler] Job 621 finished 1 steps successfully
[2017-06-01 18:59:27,907: INFO/job_scheduler] Job 622 finished 1 steps successfully
[2017-06-01 18:59:27,908: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:59:27,941: INFO/job_scheduler] Job 621: complete_job: Updating cache
[2017-06-01 18:59:27,942: INFO/job_scheduler] Job 621 completing (errored=False, cancelled=False)
[2017-06-01 18:59:28,073: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:59:28,092: INFO/job_scheduler] Job 622: complete_job: Updating cache
[2017-06-01 18:59:28,093: INFO/job_scheduler] Job 622 completing (errored=False, cancelled=False)
[2017-06-01 18:59:28,158: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:59:29,986: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:29,986: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:30,007: INFO/command_plan] add_jobs: created Job 623 (Test for host connectivity)
[2017-06-01 18:59:30,028: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:59:30,029: INFO/job_scheduler] Job 623: RunJobThread.run
[2017-06-01 18:59:30,079: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:30,079: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:30,092: INFO/command_plan] add_jobs: created Job 624 (Test for host connectivity)
[2017-06-01 18:59:30,135: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:59:30,166: INFO/job_scheduler] Job 624: RunJobThread.run
[2017-06-01 18:59:30,204: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:30,204: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:30,240: INFO/command_plan] add_jobs: created Job 625 (Test for host connectivity)
[2017-06-01 18:59:30,279: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:59:30,312: INFO/job_scheduler] Job 625: RunJobThread.run
[2017-06-01 18:59:30,331: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:30,331: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:30,378: INFO/command_plan] add_jobs: created Job 626 (Test for host connectivity)
[2017-06-01 18:59:30,413: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:59:30,421: INFO/job_scheduler] Job 626: RunJobThread.run
[2017-06-01 18:59:32,378: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:59:32,514: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:59:32,666: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:59:32,727: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:59:33,715: INFO/job_scheduler] Job 623 finished 1 steps successfully
[2017-06-01 18:59:33,761: INFO/job_scheduler] Job 623: complete_job: Updating cache
[2017-06-01 18:59:33,761: INFO/job_scheduler] Job 623 completing (errored=False, cancelled=False)
[2017-06-01 18:59:33,828: INFO/job_scheduler] Job 624 finished 1 steps successfully
[2017-06-01 18:59:33,865: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:59:33,933: INFO/job_scheduler] Job 624: complete_job: Updating cache
[2017-06-01 18:59:33,934: INFO/job_scheduler] Job 624 completing (errored=False, cancelled=False)
[2017-06-01 18:59:33,997: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:59:34,030: INFO/job_scheduler] Job 626 finished 1 steps successfully
[2017-06-01 18:59:34,055: INFO/job_scheduler] Job 626: complete_job: Updating cache
[2017-06-01 18:59:34,055: INFO/job_scheduler] Job 626 completing (errored=False, cancelled=False)
[2017-06-01 18:59:34,108: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:59:34,126: INFO/job_scheduler] Job 625 finished 1 steps successfully
[2017-06-01 18:59:34,139: INFO/job_scheduler] Job 625: complete_job: Updating cache
[2017-06-01 18:59:34,139: INFO/job_scheduler] Job 625 completing (errored=False, cancelled=False)
[2017-06-01 18:59:34,238: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:59:36,094: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:36,095: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:36,106: INFO/command_plan] add_jobs: created Job 627 (Test for host connectivity)
[2017-06-01 18:59:36,171: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:36,171: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:36,211: INFO/command_plan] add_jobs: created Job 628 (Test for host connectivity)
[2017-06-01 18:59:36,229: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:59:36,240: INFO/job_scheduler] Job 627: RunJobThread.run
[2017-06-01 18:59:36,241: INFO/job_scheduler] Job 628: RunJobThread.run
[2017-06-01 18:59:36,249: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:59:36,403: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:36,404: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:36,449: INFO/command_plan] add_jobs: created Job 629 (Test for host connectivity)
[2017-06-01 18:59:36,483: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:36,483: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:36,532: INFO/command_plan] add_jobs: created Job 630 (Test for host connectivity)
[2017-06-01 18:59:36,542: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 18:59:36,559: INFO/job_scheduler] Job 629: RunJobThread.run
[2017-06-01 18:59:36,560: INFO/job_scheduler] Job 630: RunJobThread.run
[2017-06-01 18:59:36,566: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 18:59:38,440: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:59:38,665: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:59:38,836: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:59:38,873: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:59:39,768: INFO/job_scheduler] Job 627 finished 1 steps successfully
[2017-06-01 18:59:39,807: INFO/job_scheduler] Job 627: complete_job: Updating cache
[2017-06-01 18:59:39,807: INFO/job_scheduler] Job 627 completing (errored=False, cancelled=False)
[2017-06-01 18:59:39,848: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:59:40,012: INFO/job_scheduler] Job 628 finished 1 steps successfully
[2017-06-01 18:59:40,049: INFO/job_scheduler] Job 628: complete_job: Updating cache
[2017-06-01 18:59:40,050: INFO/job_scheduler] Job 628 completing (errored=False, cancelled=False)
[2017-06-01 18:59:40,108: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:59:40,331: INFO/job_scheduler] Job 630 finished 1 steps successfully
[2017-06-01 18:59:40,370: INFO/job_scheduler] Job 629 finished 1 steps successfully
[2017-06-01 18:59:40,373: INFO/job_scheduler] Job 630: complete_job: Updating cache
[2017-06-01 18:59:40,373: INFO/job_scheduler] Job 630 completing (errored=False, cancelled=False)
[2017-06-01 18:59:40,424: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:59:40,438: INFO/job_scheduler] Job 629: complete_job: Updating cache
[2017-06-01 18:59:40,439: INFO/job_scheduler] Job 629 completing (errored=False, cancelled=False)
[2017-06-01 18:59:40,501: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:59:42,190: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:42,191: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:42,217: INFO/command_plan] add_jobs: created Job 631 (Test for host connectivity)
[2017-06-01 18:59:42,295: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:42,295: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:42,335: INFO/command_plan] add_jobs: created Job 632 (Test for host connectivity)
[2017-06-01 18:59:42,350: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:59:42,393: INFO/job_scheduler] Job 632: RunJobThread.run
[2017-06-01 18:59:42,395: INFO/job_scheduler] Job 631: RunJobThread.run
[2017-06-01 18:59:42,410: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:59:42,466: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:42,466: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:42,485: INFO/command_plan] add_jobs: created Job 633 (Test for host connectivity)
[2017-06-01 18:59:42,562: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:59:42,574: INFO/job_scheduler] Job 633: RunJobThread.run
[2017-06-01 18:59:42,602: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:42,603: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:42,631: INFO/command_plan] add_jobs: created Job 634 (Test for host connectivity)
[2017-06-01 18:59:42,642: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:59:42,646: INFO/job_scheduler] Job 634: RunJobThread.run
[2017-06-01 18:59:44,605: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:59:44,739: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:59:44,864: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:59:44,888: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:59:45,895: INFO/job_scheduler] Job 631 finished 1 steps successfully
[2017-06-01 18:59:45,956: INFO/job_scheduler] Job 631: complete_job: Updating cache
[2017-06-01 18:59:45,956: INFO/job_scheduler] Job 631 completing (errored=False, cancelled=False)
[2017-06-01 18:59:46,011: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:59:46,083: INFO/job_scheduler] Job 632 finished 1 steps successfully
[2017-06-01 18:59:46,123: INFO/job_scheduler] Job 632: complete_job: Updating cache
[2017-06-01 18:59:46,123: INFO/job_scheduler] Job 632 completing (errored=False, cancelled=False)
[2017-06-01 18:59:46,166: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:59:46,196: INFO/job_scheduler] Job 634 finished 1 steps successfully
[2017-06-01 18:59:46,240: INFO/job_scheduler] Job 634: complete_job: Updating cache
[2017-06-01 18:59:46,240: INFO/job_scheduler] Job 634 completing (errored=False, cancelled=False)
[2017-06-01 18:59:46,270: INFO/job_scheduler] Job 633 finished 1 steps successfully
[2017-06-01 18:59:46,284: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:59:46,331: INFO/job_scheduler] Job 633: complete_job: Updating cache
[2017-06-01 18:59:46,331: INFO/job_scheduler] Job 633 completing (errored=False, cancelled=False)
[2017-06-01 18:59:46,405: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:59:48,324: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:48,324: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:48,364: INFO/command_plan] add_jobs: created Job 635 (Test for host connectivity)
[2017-06-01 18:59:48,423: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:48,424: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:48,468: INFO/command_plan] add_jobs: created Job 636 (Test for host connectivity)
[2017-06-01 18:59:48,492: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 18:59:48,502: INFO/job_scheduler] Job 635: RunJobThread.run
[2017-06-01 18:59:48,515: INFO/job_scheduler] Job 636: RunJobThread.run
[2017-06-01 18:59:48,530: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:59:48,593: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:48,594: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:48,614: INFO/command_plan] add_jobs: created Job 637 (Test for host connectivity)
[2017-06-01 18:59:48,690: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:59:48,727: INFO/job_scheduler] Job 637: RunJobThread.run
[2017-06-01 18:59:48,774: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:48,775: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:48,791: INFO/command_plan] add_jobs: created Job 638 (Test for host connectivity)
[2017-06-01 18:59:48,838: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:59:48,842: INFO/job_scheduler] Job 638: RunJobThread.run
[2017-06-01 18:59:50,840: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:59:50,842: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:59:50,986: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:59:51,073: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:59:52,225: INFO/job_scheduler] Job 635 finished 1 steps successfully
[2017-06-01 18:59:52,249: INFO/job_scheduler] Job 636 finished 1 steps successfully
[2017-06-01 18:59:52,289: INFO/job_scheduler] Job 635: complete_job: Updating cache
[2017-06-01 18:59:52,289: INFO/job_scheduler] Job 635 completing (errored=False, cancelled=False)
[2017-06-01 18:59:52,333: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:59:52,367: INFO/job_scheduler] Job 636: complete_job: Updating cache
[2017-06-01 18:59:52,369: INFO/job_scheduler] Job 636 completing (errored=False, cancelled=False)
[2017-06-01 18:59:52,427: INFO/job_scheduler] Job 637 finished 1 steps successfully
[2017-06-01 18:59:52,442: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:59:52,487: INFO/job_scheduler] Job 638 finished 1 steps successfully
[2017-06-01 18:59:52,494: INFO/job_scheduler] Job 637: complete_job: Updating cache
[2017-06-01 18:59:52,495: INFO/job_scheduler] Job 637 completing (errored=False, cancelled=False)
[2017-06-01 18:59:52,606: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:59:52,622: INFO/job_scheduler] Job 638: complete_job: Updating cache
[2017-06-01 18:59:52,622: INFO/job_scheduler] Job 638 completing (errored=False, cancelled=False)
[2017-06-01 18:59:52,683: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 18:59:54,445: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:54,448: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:54,456: INFO/command_plan] add_jobs: created Job 639 (Test for host connectivity)
[2017-06-01 18:59:54,502: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 18:59:54,520: INFO/job_scheduler] Job 639: RunJobThread.run
[2017-06-01 18:59:54,561: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:54,562: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:54,585: INFO/command_plan] add_jobs: created Job 640 (Test for host connectivity)
[2017-06-01 18:59:54,625: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 18:59:54,639: INFO/job_scheduler] Job 640: RunJobThread.run
[2017-06-01 18:59:54,681: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:54,681: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:54,715: INFO/command_plan] add_jobs: created Job 641 (Test for host connectivity)
[2017-06-01 18:59:54,756: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 18:59:54,766: INFO/job_scheduler] Job 641: RunJobThread.run
[2017-06-01 18:59:54,888: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 18:59:54,888: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 18:59:54,912: INFO/command_plan] add_jobs: created Job 642 (Test for host connectivity)
[2017-06-01 18:59:54,987: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 18:59:54,993: INFO/job_scheduler] Job 642: RunJobThread.run
[2017-06-01 18:59:56,856: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 18:59:56,947: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:59:57,031: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:59:57,167: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 18:59:58,156: INFO/job_scheduler] Job 639 finished 1 steps successfully
[2017-06-01 18:59:58,182: INFO/job_scheduler] Job 639: complete_job: Updating cache
[2017-06-01 18:59:58,182: INFO/job_scheduler] Job 639 completing (errored=False, cancelled=False)
[2017-06-01 18:59:58,261: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 18:59:58,317: INFO/job_scheduler] Job 640 finished 1 steps successfully
[2017-06-01 18:59:58,350: INFO/job_scheduler] Job 640: complete_job: Updating cache
[2017-06-01 18:59:58,350: INFO/job_scheduler] Job 640 completing (errored=False, cancelled=False)
[2017-06-01 18:59:58,402: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 18:59:58,447: INFO/job_scheduler] Job 641 finished 1 steps successfully
[2017-06-01 18:59:58,480: INFO/job_scheduler] Job 641: complete_job: Updating cache
[2017-06-01 18:59:58,480: INFO/job_scheduler] Job 641 completing (errored=False, cancelled=False)
[2017-06-01 18:59:58,531: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 18:59:58,552: INFO/job_scheduler] Job 642 finished 1 steps successfully
[2017-06-01 18:59:58,597: INFO/job_scheduler] Job 642: complete_job: Updating cache
[2017-06-01 18:59:58,597: INFO/job_scheduler] Job 642 completing (errored=False, cancelled=False)
[2017-06-01 18:59:58,633: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:00:00,566: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:00,567: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:00,588: INFO/command_plan] add_jobs: created Job 643 (Test for host connectivity)
[2017-06-01 19:00:00,605: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:00:00,609: INFO/job_scheduler] Job 643: RunJobThread.run
[2017-06-01 19:00:00,673: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:00,673: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:00,688: INFO/command_plan] add_jobs: created Job 644 (Test for host connectivity)
[2017-06-01 19:00:00,780: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:00,780: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:00,805: INFO/command_plan] add_jobs: created Job 645 (Test for host connectivity)
[2017-06-01 19:00:00,930: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:00:00,933: INFO/job_scheduler] Job 644: RunJobThread.run
[2017-06-01 19:00:00,934: INFO/job_scheduler] Job 645: RunJobThread.run
[2017-06-01 19:00:00,949: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:00:00,983: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:00,985: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:00,998: INFO/command_plan] add_jobs: created Job 646 (Test for host connectivity)
[2017-06-01 19:00:01,054: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:00:01,111: INFO/job_scheduler] Job 646: RunJobThread.run
[2017-06-01 19:00:02,891: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:00:03,259: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:00:03,309: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:00:03,399: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:00:04,230: INFO/job_scheduler] Job 643 finished 1 steps successfully
[2017-06-01 19:00:04,287: INFO/job_scheduler] Job 643: complete_job: Updating cache
[2017-06-01 19:00:04,287: INFO/job_scheduler] Job 643 completing (errored=False, cancelled=False)
[2017-06-01 19:00:04,330: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:00:04,790: INFO/job_scheduler] Job 644 finished 1 steps successfully
[2017-06-01 19:00:04,804: INFO/job_scheduler] Job 645 finished 1 steps successfully
[2017-06-01 19:00:04,828: INFO/job_scheduler] Job 644: complete_job: Updating cache
[2017-06-01 19:00:04,829: INFO/job_scheduler] Job 644 completing (errored=False, cancelled=False)
[2017-06-01 19:00:04,832: INFO/job_scheduler] Job 646 finished 1 steps successfully
[2017-06-01 19:00:04,872: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:00:04,892: INFO/job_scheduler] Job 645: complete_job: Updating cache
[2017-06-01 19:00:04,892: INFO/job_scheduler] Job 645 completing (errored=False, cancelled=False)
[2017-06-01 19:00:04,971: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:00:05,013: INFO/job_scheduler] Job 646: complete_job: Updating cache
[2017-06-01 19:00:05,013: INFO/job_scheduler] Job 646 completing (errored=False, cancelled=False)
[2017-06-01 19:00:05,079: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:00:06,751: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:06,751: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:06,779: INFO/command_plan] add_jobs: created Job 647 (Test for host connectivity)
[2017-06-01 19:00:06,825: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:00:06,832: INFO/job_scheduler] Job 647: RunJobThread.run
[2017-06-01 19:00:06,892: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:06,893: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:06,914: INFO/command_plan] add_jobs: created Job 648 (Test for host connectivity)
[2017-06-01 19:00:06,976: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:00:06,978: INFO/job_scheduler] Job 648: RunJobThread.run
[2017-06-01 19:00:07,031: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:07,033: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:07,153: INFO/command_plan] add_jobs: created Job 649 (Test for host connectivity)
[2017-06-01 19:00:07,197: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:00:07,219: INFO/job_scheduler] Job 649: RunJobThread.run
[2017-06-01 19:00:07,244: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:07,245: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:07,257: INFO/command_plan] add_jobs: created Job 650 (Test for host connectivity)
[2017-06-01 19:00:07,305: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:00:07,307: INFO/job_scheduler] Job 650: RunJobThread.run
[2017-06-01 19:00:09,411: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:00:09,509: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:00:09,852: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:00:09,969: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:00:10,767: INFO/job_scheduler] Job 647 finished 1 steps successfully
[2017-06-01 19:00:10,796: INFO/job_scheduler] Job 647: complete_job: Updating cache
[2017-06-01 19:00:10,796: INFO/job_scheduler] Job 647 completing (errored=False, cancelled=False)
[2017-06-01 19:00:10,837: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:00:10,865: INFO/job_scheduler] Job 648 finished 1 steps successfully
[2017-06-01 19:00:10,921: INFO/job_scheduler] Job 648: complete_job: Updating cache
[2017-06-01 19:00:10,921: INFO/job_scheduler] Job 648 completing (errored=False, cancelled=False)
[2017-06-01 19:00:10,996: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:00:11,263: INFO/job_scheduler] Job 649 finished 1 steps successfully
[2017-06-01 19:00:11,311: INFO/job_scheduler] Job 649: complete_job: Updating cache
[2017-06-01 19:00:11,311: INFO/job_scheduler] Job 649 completing (errored=False, cancelled=False)
[2017-06-01 19:00:11,376: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:00:11,427: INFO/job_scheduler] Job 650 finished 1 steps successfully
[2017-06-01 19:00:11,472: INFO/job_scheduler] Job 650: complete_job: Updating cache
[2017-06-01 19:00:11,473: INFO/job_scheduler] Job 650 completing (errored=False, cancelled=False)
[2017-06-01 19:00:11,519: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:00:12,909: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:12,911: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:12,938: INFO/command_plan] add_jobs: created Job 651 (Test for host connectivity)
[2017-06-01 19:00:12,989: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:00:12,996: INFO/job_scheduler] Job 651: RunJobThread.run
[2017-06-01 19:00:13,019: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:13,019: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:13,045: INFO/command_plan] add_jobs: created Job 652 (Test for host connectivity)
[2017-06-01 19:00:13,104: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:00:13,123: INFO/job_scheduler] Job 652: RunJobThread.run
[2017-06-01 19:00:13,185: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:13,187: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:13,205: INFO/command_plan] add_jobs: created Job 653 (Test for host connectivity)
[2017-06-01 19:00:13,269: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:00:13,307: INFO/job_scheduler] Job 653: RunJobThread.run
[2017-06-01 19:00:13,367: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:13,367: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:13,400: INFO/command_plan] add_jobs: created Job 654 (Test for host connectivity)
[2017-06-01 19:00:13,417: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:00:13,452: INFO/job_scheduler] Job 654: RunJobThread.run
[2017-06-01 19:00:15,300: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:00:15,396: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:00:15,560: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:00:15,714: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:00:16,700: INFO/job_scheduler] Job 651 finished 1 steps successfully
[2017-06-01 19:00:16,747: INFO/job_scheduler] Job 652 finished 1 steps successfully
[2017-06-01 19:00:16,776: INFO/job_scheduler] Job 651: complete_job: Updating cache
[2017-06-01 19:00:16,776: INFO/job_scheduler] Job 651 completing (errored=False, cancelled=False)
[2017-06-01 19:00:16,868: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:00:16,900: INFO/job_scheduler] Job 652: complete_job: Updating cache
[2017-06-01 19:00:16,900: INFO/job_scheduler] Job 652 completing (errored=False, cancelled=False)
[2017-06-01 19:00:16,949: INFO/job_scheduler] Job 653 finished 1 steps successfully
[2017-06-01 19:00:17,019: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:00:17,043: INFO/job_scheduler] Job 654 finished 1 steps successfully
[2017-06-01 19:00:17,052: INFO/job_scheduler] Job 653: complete_job: Updating cache
[2017-06-01 19:00:17,052: INFO/job_scheduler] Job 653 completing (errored=False, cancelled=False)
[2017-06-01 19:00:17,094: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:00:17,108: INFO/job_scheduler] Job 654: complete_job: Updating cache
[2017-06-01 19:00:17,108: INFO/job_scheduler] Job 654 completing (errored=False, cancelled=False)
[2017-06-01 19:00:17,191: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:00:19,063: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:19,063: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:19,098: INFO/command_plan] add_jobs: created Job 655 (Test for host connectivity)
[2017-06-01 19:00:19,163: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:00:19,181: INFO/job_scheduler] Job 655: RunJobThread.run
[2017-06-01 19:00:19,253: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:19,253: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:19,293: INFO/command_plan] add_jobs: created Job 656 (Test for host connectivity)
[2017-06-01 19:00:19,336: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:00:19,341: INFO/job_scheduler] Job 656: RunJobThread.run
[2017-06-01 19:00:19,386: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:19,386: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:19,461: INFO/command_plan] add_jobs: created Job 657 (Test for host connectivity)
[2017-06-01 19:00:19,533: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:00:19,537: INFO/job_scheduler] Job 657: RunJobThread.run
[2017-06-01 19:00:19,582: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:19,583: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:19,618: INFO/command_plan] add_jobs: created Job 658 (Test for host connectivity)
[2017-06-01 19:00:19,644: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:00:19,674: INFO/job_scheduler] Job 658: RunJobThread.run
[2017-06-01 19:00:21,483: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:00:21,623: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:00:21,792: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:00:21,847: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:00:22,813: INFO/job_scheduler] Job 655 finished 1 steps successfully
[2017-06-01 19:00:22,850: INFO/job_scheduler] Job 655: complete_job: Updating cache
[2017-06-01 19:00:22,850: INFO/job_scheduler] Job 655 completing (errored=False, cancelled=False)
[2017-06-01 19:00:22,953: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:00:22,968: INFO/job_scheduler] Job 656 finished 1 steps successfully
[2017-06-01 19:00:23,034: INFO/job_scheduler] Job 656: complete_job: Updating cache
[2017-06-01 19:00:23,034: INFO/job_scheduler] Job 656 completing (errored=False, cancelled=False)
[2017-06-01 19:00:23,108: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:00:23,243: INFO/job_scheduler] Job 658 finished 1 steps successfully
[2017-06-01 19:00:23,246: INFO/job_scheduler] Job 657 finished 1 steps successfully
[2017-06-01 19:00:23,275: INFO/job_scheduler] Job 658: complete_job: Updating cache
[2017-06-01 19:00:23,275: INFO/job_scheduler] Job 658 completing (errored=False, cancelled=False)
[2017-06-01 19:00:23,318: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:00:23,345: INFO/job_scheduler] Job 657: complete_job: Updating cache
[2017-06-01 19:00:23,345: INFO/job_scheduler] Job 657 completing (errored=False, cancelled=False)
[2017-06-01 19:00:23,428: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:00:25,250: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:25,250: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:25,258: INFO/command_plan] add_jobs: created Job 659 (Test for host connectivity)
[2017-06-01 19:00:25,408: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:25,408: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:25,434: INFO/command_plan] add_jobs: created Job 660 (Test for host connectivity)
[2017-06-01 19:00:25,478: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:00:25,492: INFO/job_scheduler] Job 659: RunJobThread.run
[2017-06-01 19:00:25,521: INFO/job_scheduler] Job 660: RunJobThread.run
[2017-06-01 19:00:25,567: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:25,567: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:25,578: INFO/command_plan] add_jobs: created Job 661 (Test for host connectivity)
[2017-06-01 19:00:25,660: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:00:25,680: INFO/job_scheduler] Job 661: RunJobThread.run
[2017-06-01 19:00:25,747: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:25,747: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:25,758: INFO/command_plan] add_jobs: created Job 662 (Test for host connectivity)
[2017-06-01 19:00:25,792: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:00:25,819: INFO/job_scheduler] Job 662: RunJobThread.run
[2017-06-01 19:00:25,835: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:00:27,706: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:00:27,820: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:00:27,912: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:00:28,082: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:00:29,031: INFO/job_scheduler] Job 659 finished 1 steps successfully
[2017-06-01 19:00:29,052: INFO/job_scheduler] Job 659: complete_job: Updating cache
[2017-06-01 19:00:29,052: INFO/job_scheduler] Job 659 completing (errored=False, cancelled=False)
[2017-06-01 19:00:29,100: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:00:29,127: INFO/job_scheduler] Job 660 finished 1 steps successfully
[2017-06-01 19:00:29,151: INFO/job_scheduler] Job 660: complete_job: Updating cache
[2017-06-01 19:00:29,151: INFO/job_scheduler] Job 660 completing (errored=False, cancelled=False)
[2017-06-01 19:00:29,256: INFO/job_scheduler] Job 661 finished 1 steps successfully
[2017-06-01 19:00:29,291: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:00:29,301: INFO/job_scheduler] Job 661: complete_job: Updating cache
[2017-06-01 19:00:29,301: INFO/job_scheduler] Job 661 completing (errored=False, cancelled=False)
[2017-06-01 19:00:29,366: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:00:29,435: INFO/job_scheduler] Job 662 finished 1 steps successfully
[2017-06-01 19:00:29,475: INFO/job_scheduler] Job 662: complete_job: Updating cache
[2017-06-01 19:00:29,476: INFO/job_scheduler] Job 662 completing (errored=False, cancelled=False)
[2017-06-01 19:00:29,516: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:00:32,101: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:32,101: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:32,156: INFO/command_plan] add_jobs: created Job 663 (Test for host connectivity)
[2017-06-01 19:00:32,197: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:00:32,211: INFO/job_scheduler] Job 663: RunJobThread.run
[2017-06-01 19:00:32,285: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:32,286: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:32,298: INFO/command_plan] add_jobs: created Job 664 (Test for host connectivity)
[2017-06-01 19:00:32,312: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:00:32,324: INFO/job_scheduler] Job 664: RunJobThread.run
[2017-06-01 19:00:32,412: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:32,412: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:32,515: INFO/command_plan] add_jobs: created Job 665 (Test for host connectivity)
[2017-06-01 19:00:32,588: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:32,588: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:32,627: INFO/command_plan] add_jobs: created Job 666 (Test for host connectivity)
[2017-06-01 19:00:32,655: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:00:32,686: INFO/job_scheduler] Job 665: RunJobThread.run
[2017-06-01 19:00:32,705: INFO/job_scheduler] Job 666: RunJobThread.run
[2017-06-01 19:00:32,733: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:00:34,598: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:00:34,604: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:00:34,980: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:00:35,047: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:00:35,934: INFO/job_scheduler] Job 663 finished 1 steps successfully
[2017-06-01 19:00:36,006: INFO/job_scheduler] Job 663: complete_job: Updating cache
[2017-06-01 19:00:36,006: INFO/job_scheduler] Job 663 completing (errored=False, cancelled=False)
[2017-06-01 19:00:36,035: INFO/job_scheduler] Job 664 finished 1 steps successfully
[2017-06-01 19:00:36,104: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:00:36,117: INFO/job_scheduler] Job 664: complete_job: Updating cache
[2017-06-01 19:00:36,122: INFO/job_scheduler] Job 664 completing (errored=False, cancelled=False)
[2017-06-01 19:00:36,182: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:00:36,324: INFO/job_scheduler] Job 665 finished 1 steps successfully
[2017-06-01 19:00:36,347: INFO/job_scheduler] Job 665: complete_job: Updating cache
[2017-06-01 19:00:36,348: INFO/job_scheduler] Job 665 completing (errored=False, cancelled=False)
[2017-06-01 19:00:36,400: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:00:36,561: INFO/job_scheduler] Job 666 finished 1 steps successfully
[2017-06-01 19:00:36,606: INFO/job_scheduler] Job 666: complete_job: Updating cache
[2017-06-01 19:00:36,606: INFO/job_scheduler] Job 666 completing (errored=False, cancelled=False)
[2017-06-01 19:00:36,657: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:00:38,394: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:38,395: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:38,407: INFO/command_plan] add_jobs: created Job 667 (Test for host connectivity)
[2017-06-01 19:00:38,432: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:00:38,441: INFO/job_scheduler] Job 667: RunJobThread.run
[2017-06-01 19:00:38,487: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:38,487: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:38,546: INFO/command_plan] add_jobs: created Job 668 (Test for host connectivity)
[2017-06-01 19:00:38,622: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:00:38,624: INFO/job_scheduler] Job 668: RunJobThread.run
[2017-06-01 19:00:38,650: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:38,651: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:38,665: INFO/command_plan] add_jobs: created Job 669 (Test for host connectivity)
[2017-06-01 19:00:38,676: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:00:38,690: INFO/job_scheduler] Job 669: RunJobThread.run
[2017-06-01 19:00:38,783: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:38,783: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:38,832: INFO/command_plan] add_jobs: created Job 670 (Test for host connectivity)
[2017-06-01 19:00:38,886: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:00:38,901: INFO/job_scheduler] Job 670: RunJobThread.run
[2017-06-01 19:00:40,571: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:00:40,837: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:00:40,985: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:00:41,237: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:00:41,895: INFO/job_scheduler] Job 667 finished 1 steps successfully
[2017-06-01 19:00:41,938: INFO/job_scheduler] Job 667: complete_job: Updating cache
[2017-06-01 19:00:41,938: INFO/job_scheduler] Job 667 completing (errored=False, cancelled=False)
[2017-06-01 19:00:42,005: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:00:42,207: INFO/job_scheduler] Job 668 finished 1 steps successfully
[2017-06-01 19:00:42,271: INFO/job_scheduler] Job 668: complete_job: Updating cache
[2017-06-01 19:00:42,271: INFO/job_scheduler] Job 668 completing (errored=False, cancelled=False)
[2017-06-01 19:00:42,325: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:00:42,449: INFO/job_scheduler] Job 669 finished 1 steps successfully
[2017-06-01 19:00:42,463: INFO/job_scheduler] Job 669: complete_job: Updating cache
[2017-06-01 19:00:42,463: INFO/job_scheduler] Job 669 completing (errored=False, cancelled=False)
[2017-06-01 19:00:42,549: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:00:42,633: INFO/job_scheduler] Job 670 finished 1 steps successfully
[2017-06-01 19:00:42,655: INFO/job_scheduler] Job 670: complete_job: Updating cache
[2017-06-01 19:00:42,655: INFO/job_scheduler] Job 670 completing (errored=False, cancelled=False)
[2017-06-01 19:00:42,724: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:00:43,719: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:43,719: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:43,762: INFO/command_plan] add_jobs: created Job 671 (Test for host connectivity)
[2017-06-01 19:00:43,782: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:00:43,787: INFO/job_scheduler] Job 671: RunJobThread.run
[2017-06-01 19:00:43,815: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:43,815: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:43,929: INFO/command_plan] add_jobs: created Job 672 (Test for host connectivity)
[2017-06-01 19:00:43,982: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:00:43,992: INFO/job_scheduler] Job 672: RunJobThread.run
[2017-06-01 19:00:44,054: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:44,054: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:44,081: INFO/command_plan] add_jobs: created Job 673 (Test for host connectivity)
[2017-06-01 19:00:44,159: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:00:44,165: INFO/job_scheduler] Job 673: RunJobThread.run
[2017-06-01 19:00:44,197: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:44,198: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:44,216: INFO/command_plan] add_jobs: created Job 674 (Test for host connectivity)
[2017-06-01 19:00:44,260: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:00:44,274: INFO/job_scheduler] Job 674: RunJobThread.run
[2017-06-01 19:00:46,265: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:00:46,394: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:00:46,507: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:00:46,638: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:00:47,608: INFO/job_scheduler] Job 671 finished 1 steps successfully
[2017-06-01 19:00:47,663: INFO/job_scheduler] Job 671: complete_job: Updating cache
[2017-06-01 19:00:47,663: INFO/job_scheduler] Job 671 completing (errored=False, cancelled=False)
[2017-06-01 19:00:47,766: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:00:47,796: INFO/job_scheduler] Job 672 finished 1 steps successfully
[2017-06-01 19:00:47,817: INFO/job_scheduler] Job 672: complete_job: Updating cache
[2017-06-01 19:00:47,817: INFO/job_scheduler] Job 672 completing (errored=False, cancelled=False)
[2017-06-01 19:00:47,877: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:00:47,969: INFO/job_scheduler] Job 673 finished 1 steps successfully
[2017-06-01 19:00:47,995: INFO/job_scheduler] Job 673: complete_job: Updating cache
[2017-06-01 19:00:47,995: INFO/job_scheduler] Job 673 completing (errored=False, cancelled=False)
[2017-06-01 19:00:48,037: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:00:48,107: INFO/job_scheduler] Job 674 finished 1 steps successfully
[2017-06-01 19:00:48,130: INFO/job_scheduler] Job 674: complete_job: Updating cache
[2017-06-01 19:00:48,130: INFO/job_scheduler] Job 674 completing (errored=False, cancelled=False)
[2017-06-01 19:00:48,195: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:00:49,867: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:49,869: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:49,876: INFO/command_plan] add_jobs: created Job 675 (Test for host connectivity)
[2017-06-01 19:00:49,922: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:00:49,939: INFO/job_scheduler] Job 675: RunJobThread.run
[2017-06-01 19:00:49,977: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:49,978: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:49,992: INFO/command_plan] add_jobs: created Job 676 (Test for host connectivity)
[2017-06-01 19:00:50,032: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:00:50,061: INFO/job_scheduler] Job 676: RunJobThread.run
[2017-06-01 19:00:50,122: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:50,123: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:50,229: INFO/command_plan] add_jobs: created Job 677 (Test for host connectivity)
[2017-06-01 19:00:50,278: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:00:50,284: INFO/job_scheduler] Job 677: RunJobThread.run
[2017-06-01 19:00:50,322: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:50,323: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:50,350: INFO/command_plan] add_jobs: created Job 678 (Test for host connectivity)
[2017-06-01 19:00:50,391: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:00:50,409: INFO/job_scheduler] Job 678: RunJobThread.run
[2017-06-01 19:00:52,243: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:00:52,444: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:00:52,638: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:00:52,750: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:00:53,576: INFO/job_scheduler] Job 675 finished 1 steps successfully
[2017-06-01 19:00:53,607: INFO/job_scheduler] Job 675: complete_job: Updating cache
[2017-06-01 19:00:53,608: INFO/job_scheduler] Job 675 completing (errored=False, cancelled=False)
[2017-06-01 19:00:53,666: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:00:53,670: INFO/job_scheduler] Job 676 finished 1 steps successfully
[2017-06-01 19:00:53,682: INFO/job_scheduler] Job 676: complete_job: Updating cache
[2017-06-01 19:00:53,684: INFO/job_scheduler] Job 676 completing (errored=False, cancelled=False)
[2017-06-01 19:00:53,837: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:00:54,171: INFO/job_scheduler] Job 677 finished 1 steps successfully
[2017-06-01 19:00:54,190: INFO/job_scheduler] Job 677: complete_job: Updating cache
[2017-06-01 19:00:54,191: INFO/job_scheduler] Job 677 completing (errored=False, cancelled=False)
[2017-06-01 19:00:54,224: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:00:54,255: INFO/job_scheduler] Job 678 finished 1 steps successfully
[2017-06-01 19:00:54,277: INFO/job_scheduler] Job 678: complete_job: Updating cache
[2017-06-01 19:00:54,277: INFO/job_scheduler] Job 678 completing (errored=False, cancelled=False)
[2017-06-01 19:00:54,329: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:00:56,006: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:56,007: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:56,123: INFO/command_plan] add_jobs: created Job 679 (Test for host connectivity)
[2017-06-01 19:00:56,148: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:00:56,173: INFO/job_scheduler] Job 679: RunJobThread.run
[2017-06-01 19:00:56,238: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:56,240: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:56,266: INFO/command_plan] add_jobs: created Job 680 (Test for host connectivity)
[2017-06-01 19:00:56,287: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:00:56,326: INFO/job_scheduler] Job 680: RunJobThread.run
[2017-06-01 19:00:56,390: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:56,392: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:56,400: INFO/command_plan] add_jobs: created Job 681 (Test for host connectivity)
[2017-06-01 19:00:56,450: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:00:56,477: INFO/job_scheduler] Job 681: RunJobThread.run
[2017-06-01 19:00:56,542: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:00:56,542: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:00:56,574: INFO/command_plan] add_jobs: created Job 682 (Test for host connectivity)
[2017-06-01 19:00:56,662: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:00:56,692: INFO/job_scheduler] Job 682: RunJobThread.run
[2017-06-01 19:00:58,414: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:00:58,726: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:00:58,781: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:00:58,859: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:00:59,761: INFO/job_scheduler] Job 679 finished 1 steps successfully
[2017-06-01 19:00:59,790: INFO/job_scheduler] Job 679: complete_job: Updating cache
[2017-06-01 19:00:59,790: INFO/job_scheduler] Job 679 completing (errored=False, cancelled=False)
[2017-06-01 19:00:59,885: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:01:00,082: INFO/job_scheduler] Job 680 finished 1 steps successfully
[2017-06-01 19:01:00,123: INFO/job_scheduler] Job 680: complete_job: Updating cache
[2017-06-01 19:01:00,123: INFO/job_scheduler] Job 680 completing (errored=False, cancelled=False)
[2017-06-01 19:01:00,183: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:01:00,422: INFO/job_scheduler] Job 682 finished 1 steps successfully
[2017-06-01 19:01:00,429: INFO/job_scheduler] Job 681 finished 1 steps successfully
[2017-06-01 19:01:00,468: INFO/job_scheduler] Job 682: complete_job: Updating cache
[2017-06-01 19:01:00,468: INFO/job_scheduler] Job 682 completing (errored=False, cancelled=False)
[2017-06-01 19:01:00,519: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:01:00,549: INFO/job_scheduler] Job 681: complete_job: Updating cache
[2017-06-01 19:01:00,550: INFO/job_scheduler] Job 681 completing (errored=False, cancelled=False)
[2017-06-01 19:01:00,626: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:01:02,254: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:02,255: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:02,265: INFO/command_plan] add_jobs: created Job 683 (Test for host connectivity)
[2017-06-01 19:01:02,296: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:01:02,305: INFO/job_scheduler] Job 683: RunJobThread.run
[2017-06-01 19:01:02,339: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:02,340: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:02,359: INFO/command_plan] add_jobs: created Job 684 (Test for host connectivity)
[2017-06-01 19:01:02,389: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:01:02,397: INFO/job_scheduler] Job 684: RunJobThread.run
[2017-06-01 19:01:02,502: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:02,502: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:02,541: INFO/command_plan] add_jobs: created Job 685 (Test for host connectivity)
[2017-06-01 19:01:02,599: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:01:02,631: INFO/job_scheduler] Job 685: RunJobThread.run
[2017-06-01 19:01:02,693: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:02,694: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:02,709: INFO/command_plan] add_jobs: created Job 686 (Test for host connectivity)
[2017-06-01 19:01:02,817: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:01:02,821: INFO/job_scheduler] Job 686: RunJobThread.run
[2017-06-01 19:01:04,532: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:01:04,675: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:01:04,919: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:01:05,150: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:01:05,852: INFO/job_scheduler] Job 683 finished 1 steps successfully
[2017-06-01 19:01:05,881: INFO/job_scheduler] Job 683: complete_job: Updating cache
[2017-06-01 19:01:05,883: INFO/job_scheduler] Job 683 completing (errored=False, cancelled=False)
[2017-06-01 19:01:06,009: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:01:06,124: INFO/job_scheduler] Job 684 finished 1 steps successfully
[2017-06-01 19:01:06,189: INFO/job_scheduler] Job 684: complete_job: Updating cache
[2017-06-01 19:01:06,191: INFO/job_scheduler] Job 684 completing (errored=False, cancelled=False)
[2017-06-01 19:01:06,233: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:01:06,450: INFO/job_scheduler] Job 685 finished 1 steps successfully
[2017-06-01 19:01:06,505: INFO/job_scheduler] Job 685: complete_job: Updating cache
[2017-06-01 19:01:06,506: INFO/job_scheduler] Job 685 completing (errored=False, cancelled=False)
[2017-06-01 19:01:06,564: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:01:06,618: INFO/job_scheduler] Job 686 finished 1 steps successfully
[2017-06-01 19:01:06,647: INFO/job_scheduler] Job 686: complete_job: Updating cache
[2017-06-01 19:01:06,647: INFO/job_scheduler] Job 686 completing (errored=False, cancelled=False)
[2017-06-01 19:01:06,718: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:01:08,445: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:08,446: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:08,588: INFO/command_plan] add_jobs: created Job 687 (Test for host connectivity)
[2017-06-01 19:01:08,646: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:01:08,654: INFO/job_scheduler] Job 687: RunJobThread.run
[2017-06-01 19:01:08,686: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:08,686: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:08,714: INFO/command_plan] add_jobs: created Job 688 (Test for host connectivity)
[2017-06-01 19:01:08,754: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:01:08,757: INFO/job_scheduler] Job 688: RunJobThread.run
[2017-06-01 19:01:08,808: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:08,808: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:08,841: INFO/command_plan] add_jobs: created Job 689 (Test for host connectivity)
[2017-06-01 19:01:08,875: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:01:08,877: INFO/job_scheduler] Job 689: RunJobThread.run
[2017-06-01 19:01:08,920: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:08,920: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:08,987: INFO/command_plan] add_jobs: created Job 690 (Test for host connectivity)
[2017-06-01 19:01:09,035: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:01:09,062: INFO/job_scheduler] Job 690: RunJobThread.run
[2017-06-01 19:01:10,845: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:01:11,345: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:01:11,434: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:01:11,490: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:01:12,235: INFO/job_scheduler] Job 687 finished 1 steps successfully
[2017-06-01 19:01:12,271: INFO/job_scheduler] Job 687: complete_job: Updating cache
[2017-06-01 19:01:12,271: INFO/job_scheduler] Job 687 completing (errored=False, cancelled=False)
[2017-06-01 19:01:12,329: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:01:12,723: INFO/job_scheduler] Job 690 finished 1 steps successfully
[2017-06-01 19:01:12,737: INFO/job_scheduler] Job 690: complete_job: Updating cache
[2017-06-01 19:01:12,738: INFO/job_scheduler] Job 690 completing (errored=False, cancelled=False)
[2017-06-01 19:01:12,781: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:01:12,863: INFO/job_scheduler] Job 688 finished 1 steps successfully
[2017-06-01 19:01:12,874: INFO/job_scheduler] Job 689 finished 1 steps successfully
[2017-06-01 19:01:12,887: INFO/job_scheduler] Job 688: complete_job: Updating cache
[2017-06-01 19:01:12,888: INFO/job_scheduler] Job 688 completing (errored=False, cancelled=False)
[2017-06-01 19:01:12,938: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:01:12,954: INFO/job_scheduler] Job 689: complete_job: Updating cache
[2017-06-01 19:01:12,955: INFO/job_scheduler] Job 689 completing (errored=False, cancelled=False)
[2017-06-01 19:01:13,038: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:01:14,675: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:14,675: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:14,689: INFO/command_plan] add_jobs: created Job 691 (Test for host connectivity)
[2017-06-01 19:01:14,726: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:01:14,734: INFO/job_scheduler] Job 691: RunJobThread.run
[2017-06-01 19:01:14,762: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:14,762: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:14,777: INFO/command_plan] add_jobs: created Job 692 (Test for host connectivity)
[2017-06-01 19:01:14,810: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:01:14,825: INFO/job_scheduler] Job 692: RunJobThread.run
[2017-06-01 19:01:14,905: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:14,905: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:14,929: INFO/command_plan] add_jobs: created Job 693 (Test for host connectivity)
[2017-06-01 19:01:14,981: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:01:14,996: INFO/job_scheduler] Job 693: RunJobThread.run
[2017-06-01 19:01:15,078: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:15,081: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:15,097: INFO/command_plan] add_jobs: created Job 694 (Test for host connectivity)
[2017-06-01 19:01:15,176: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:01:15,182: INFO/job_scheduler] Job 694: RunJobThread.run
[2017-06-01 19:01:17,128: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:01:17,220: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:01:17,285: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:01:17,501: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:01:18,459: INFO/job_scheduler] Job 692 finished 1 steps successfully
[2017-06-01 19:01:18,520: INFO/job_scheduler] Job 692: complete_job: Updating cache
[2017-06-01 19:01:18,521: INFO/job_scheduler] Job 692 completing (errored=False, cancelled=False)
[2017-06-01 19:01:18,545: INFO/job_scheduler] Job 691 finished 1 steps successfully
[2017-06-01 19:01:18,639: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:01:18,663: INFO/job_scheduler] Job 691: complete_job: Updating cache
[2017-06-01 19:01:18,663: INFO/job_scheduler] Job 691 completing (errored=False, cancelled=False)
[2017-06-01 19:01:18,691: INFO/job_scheduler] Job 693 finished 1 steps successfully
[2017-06-01 19:01:18,771: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:01:18,801: INFO/job_scheduler] Job 693: complete_job: Updating cache
[2017-06-01 19:01:18,801: INFO/job_scheduler] Job 693 completing (errored=False, cancelled=False)
[2017-06-01 19:01:18,845: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:01:18,877: INFO/job_scheduler] Job 694 finished 1 steps successfully
[2017-06-01 19:01:18,912: INFO/job_scheduler] Job 694: complete_job: Updating cache
[2017-06-01 19:01:18,912: INFO/job_scheduler] Job 694 completing (errored=False, cancelled=False)
[2017-06-01 19:01:18,961: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:01:20,778: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:20,780: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:20,794: INFO/command_plan] add_jobs: created Job 695 (Test for host connectivity)
[2017-06-01 19:01:20,840: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:01:20,848: INFO/job_scheduler] Job 695: RunJobThread.run
[2017-06-01 19:01:20,912: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:20,912: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:21,001: INFO/command_plan] add_jobs: created Job 696 (Test for host connectivity)
[2017-06-01 19:01:21,071: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:21,071: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:21,108: INFO/command_plan] add_jobs: created Job 697 (Test for host connectivity)
[2017-06-01 19:01:21,123: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:01:21,146: INFO/job_scheduler] Job 696: RunJobThread.run
[2017-06-01 19:01:21,149: INFO/job_scheduler] Job 697: RunJobThread.run
[2017-06-01 19:01:21,154: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:01:21,191: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:21,191: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:21,245: INFO/command_plan] add_jobs: created Job 698 (Test for host connectivity)
[2017-06-01 19:01:21,273: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:01:21,285: INFO/job_scheduler] Job 698: RunJobThread.run
[2017-06-01 19:01:23,147: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:01:23,464: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:01:23,527: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:01:23,553: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:01:24,495: INFO/job_scheduler] Job 695 finished 1 steps successfully
[2017-06-01 19:01:24,552: INFO/job_scheduler] Job 695: complete_job: Updating cache
[2017-06-01 19:01:24,552: INFO/job_scheduler] Job 695 completing (errored=False, cancelled=False)
[2017-06-01 19:01:24,628: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:01:24,863: INFO/job_scheduler] Job 696 finished 1 steps successfully
[2017-06-01 19:01:24,902: INFO/job_scheduler] Job 696: complete_job: Updating cache
[2017-06-01 19:01:24,902: INFO/job_scheduler] Job 696 completing (errored=False, cancelled=False)
[2017-06-01 19:01:24,954: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:01:25,004: INFO/job_scheduler] Job 697 finished 1 steps successfully
[2017-06-01 19:01:25,013: INFO/job_scheduler] Job 697: complete_job: Updating cache
[2017-06-01 19:01:25,013: INFO/job_scheduler] Job 697 completing (errored=False, cancelled=False)
[2017-06-01 19:01:25,054: INFO/job_scheduler] Job 698 finished 1 steps successfully
[2017-06-01 19:01:25,089: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:01:25,123: INFO/job_scheduler] Job 698: complete_job: Updating cache
[2017-06-01 19:01:25,123: INFO/job_scheduler] Job 698 completing (errored=False, cancelled=False)
[2017-06-01 19:01:25,219: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:01:26,901: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:26,903: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:26,911: INFO/command_plan] add_jobs: created Job 699 (Test for host connectivity)
[2017-06-01 19:01:26,993: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:01:26,996: INFO/job_scheduler] Job 699: RunJobThread.run
[2017-06-01 19:01:27,028: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:27,028: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:27,057: INFO/command_plan] add_jobs: created Job 700 (Test for host connectivity)
[2017-06-01 19:01:27,153: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:27,154: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:27,174: INFO/command_plan] add_jobs: created Job 701 (Test for host connectivity)
[2017-06-01 19:01:27,208: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:01:27,236: INFO/job_scheduler] Job 700: RunJobThread.run
[2017-06-01 19:01:27,242: INFO/job_scheduler] Job 701: RunJobThread.run
[2017-06-01 19:01:27,250: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:01:27,306: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:27,312: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:27,341: INFO/command_plan] add_jobs: created Job 702 (Test for host connectivity)
[2017-06-01 19:01:27,417: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:01:27,420: INFO/job_scheduler] Job 702: RunJobThread.run
[2017-06-01 19:01:29,224: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:01:29,469: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:01:29,524: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:01:29,806: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:01:30,717: INFO/job_scheduler] Job 699 finished 1 steps successfully
[2017-06-01 19:01:30,849: INFO/job_scheduler] Job 699: complete_job: Updating cache
[2017-06-01 19:01:30,849: INFO/job_scheduler] Job 699 completing (errored=False, cancelled=False)
[2017-06-01 19:01:30,851: INFO/job_scheduler] Job 700 finished 1 steps successfully
[2017-06-01 19:01:30,862: INFO/job_scheduler] Job 701 finished 1 steps successfully
[2017-06-01 19:01:30,915: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:01:30,932: INFO/job_scheduler] Job 700: complete_job: Updating cache
[2017-06-01 19:01:30,933: INFO/job_scheduler] Job 700 completing (errored=False, cancelled=False)
[2017-06-01 19:01:31,023: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:01:31,075: INFO/job_scheduler] Job 701: complete_job: Updating cache
[2017-06-01 19:01:31,075: INFO/job_scheduler] Job 701 completing (errored=False, cancelled=False)
[2017-06-01 19:01:31,149: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:01:31,191: INFO/job_scheduler] Job 702 finished 1 steps successfully
[2017-06-01 19:01:31,230: INFO/job_scheduler] Job 702: complete_job: Updating cache
[2017-06-01 19:01:31,231: INFO/job_scheduler] Job 702 completing (errored=False, cancelled=False)
[2017-06-01 19:01:31,272: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:01:33,068: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:33,069: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:33,081: INFO/command_plan] add_jobs: created Job 703 (Test for host connectivity)
[2017-06-01 19:01:33,146: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:01:33,147: INFO/job_scheduler] Job 703: RunJobThread.run
[2017-06-01 19:01:33,298: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:33,299: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:33,334: INFO/command_plan] add_jobs: created Job 704 (Test for host connectivity)
[2017-06-01 19:01:33,358: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:01:33,377: INFO/job_scheduler] Job 704: RunJobThread.run
[2017-06-01 19:01:33,415: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:33,416: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:33,431: INFO/command_plan] add_jobs: created Job 705 (Test for host connectivity)
[2017-06-01 19:01:33,492: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:01:33,522: INFO/job_scheduler] Job 705: RunJobThread.run
[2017-06-01 19:01:33,592: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:33,594: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:33,623: INFO/command_plan] add_jobs: created Job 706 (Test for host connectivity)
[2017-06-01 19:01:33,677: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:01:33,706: INFO/job_scheduler] Job 706: RunJobThread.run
[2017-06-01 19:01:35,415: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:01:35,501: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:01:35,798: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:01:35,900: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:01:36,738: INFO/job_scheduler] Job 703 finished 1 steps successfully
[2017-06-01 19:01:36,763: INFO/job_scheduler] Job 703: complete_job: Updating cache
[2017-06-01 19:01:36,764: INFO/job_scheduler] Job 703 completing (errored=False, cancelled=False)
[2017-06-01 19:01:36,814: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:01:36,838: INFO/job_scheduler] Job 704 finished 1 steps successfully
[2017-06-01 19:01:36,883: INFO/job_scheduler] Job 704: complete_job: Updating cache
[2017-06-01 19:01:36,884: INFO/job_scheduler] Job 704 completing (errored=False, cancelled=False)
[2017-06-01 19:01:36,942: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:01:37,278: INFO/job_scheduler] Job 705 finished 1 steps successfully
[2017-06-01 19:01:37,330: INFO/job_scheduler] Job 705: complete_job: Updating cache
[2017-06-01 19:01:37,330: INFO/job_scheduler] Job 705 completing (errored=False, cancelled=False)
[2017-06-01 19:01:37,373: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:01:37,464: INFO/job_scheduler] Job 706 finished 1 steps successfully
[2017-06-01 19:01:37,521: INFO/job_scheduler] Job 706: complete_job: Updating cache
[2017-06-01 19:01:37,521: INFO/job_scheduler] Job 706 completing (errored=False, cancelled=False)
[2017-06-01 19:01:37,564: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:01:39,307: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:39,309: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:39,322: INFO/command_plan] add_jobs: created Job 707 (Test for host connectivity)
[2017-06-01 19:01:39,379: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:39,380: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:39,412: INFO/command_plan] add_jobs: created Job 708 (Test for host connectivity)
[2017-06-01 19:01:39,430: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:01:39,441: INFO/job_scheduler] Job 707: RunJobThread.run
[2017-06-01 19:01:39,468: INFO/job_scheduler] Job 708: RunJobThread.run
[2017-06-01 19:01:39,486: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:01:39,527: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:39,528: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:39,562: INFO/command_plan] add_jobs: created Job 709 (Test for host connectivity)
[2017-06-01 19:01:39,574: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:01:39,606: INFO/job_scheduler] Job 709: RunJobThread.run
[2017-06-01 19:01:39,725: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:39,726: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:39,871: INFO/command_plan] add_jobs: created Job 710 (Test for host connectivity)
[2017-06-01 19:01:39,931: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:01:39,943: INFO/job_scheduler] Job 710: RunJobThread.run
[2017-06-01 19:01:41,796: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:01:41,885: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:01:41,969: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:01:42,170: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:01:43,181: INFO/job_scheduler] Job 707 finished 1 steps successfully
[2017-06-01 19:01:43,193: INFO/job_scheduler] Job 708 finished 1 steps successfully
[2017-06-01 19:01:43,212: INFO/job_scheduler] Job 707: complete_job: Updating cache
[2017-06-01 19:01:43,212: INFO/job_scheduler] Job 707 completing (errored=False, cancelled=False)
[2017-06-01 19:01:43,247: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:01:43,262: INFO/job_scheduler] Job 708: complete_job: Updating cache
[2017-06-01 19:01:43,263: INFO/job_scheduler] Job 708 completing (errored=False, cancelled=False)
[2017-06-01 19:01:43,343: INFO/job_scheduler] Job 709 finished 1 steps successfully
[2017-06-01 19:01:43,371: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:01:43,379: INFO/job_scheduler] Job 709: complete_job: Updating cache
[2017-06-01 19:01:43,379: INFO/job_scheduler] Job 709 completing (errored=False, cancelled=False)
[2017-06-01 19:01:43,463: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:01:43,465: INFO/job_scheduler] Job 710 finished 1 steps successfully
[2017-06-01 19:01:43,506: INFO/job_scheduler] Job 710: complete_job: Updating cache
[2017-06-01 19:01:43,506: INFO/job_scheduler] Job 710 completing (errored=False, cancelled=False)
[2017-06-01 19:01:43,580: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:01:45,628: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:45,629: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:45,673: INFO/command_plan] add_jobs: created Job 711 (Test for host connectivity)
[2017-06-01 19:01:45,720: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:01:45,736: INFO/job_scheduler] Job 711: RunJobThread.run
[2017-06-01 19:01:45,799: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:45,801: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:45,816: INFO/command_plan] add_jobs: created Job 712 (Test for host connectivity)
[2017-06-01 19:01:45,881: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:01:45,886: INFO/job_scheduler] Job 712: RunJobThread.run
[2017-06-01 19:01:45,913: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:45,914: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:45,989: INFO/command_plan] add_jobs: created Job 713 (Test for host connectivity)
[2017-06-01 19:01:46,058: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:01:46,062: INFO/job_scheduler] Job 713: RunJobThread.run
[2017-06-01 19:01:46,223: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:46,224: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:46,264: INFO/command_plan] add_jobs: created Job 714 (Test for host connectivity)
[2017-06-01 19:01:46,288: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:01:46,305: INFO/job_scheduler] Job 714: RunJobThread.run
[2017-06-01 19:01:48,001: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:01:48,234: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:01:48,334: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:01:48,534: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:01:49,352: INFO/job_scheduler] Job 711 finished 1 steps successfully
[2017-06-01 19:01:49,379: INFO/job_scheduler] Job 711: complete_job: Updating cache
[2017-06-01 19:01:49,379: INFO/job_scheduler] Job 711 completing (errored=False, cancelled=False)
[2017-06-01 19:01:49,453: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:01:49,558: INFO/job_scheduler] Job 712 finished 1 steps successfully
[2017-06-01 19:01:49,586: INFO/job_scheduler] Job 712: complete_job: Updating cache
[2017-06-01 19:01:49,586: INFO/job_scheduler] Job 712 completing (errored=False, cancelled=False)
[2017-06-01 19:01:49,633: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:01:49,685: INFO/job_scheduler] Job 713 finished 1 steps successfully
[2017-06-01 19:01:49,728: INFO/job_scheduler] Job 713: complete_job: Updating cache
[2017-06-01 19:01:49,728: INFO/job_scheduler] Job 713 completing (errored=False, cancelled=False)
[2017-06-01 19:01:49,780: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:01:50,021: INFO/job_scheduler] Job 714 finished 1 steps successfully
[2017-06-01 19:01:50,070: INFO/job_scheduler] Job 714: complete_job: Updating cache
[2017-06-01 19:01:50,070: INFO/job_scheduler] Job 714 completing (errored=False, cancelled=False)
[2017-06-01 19:01:50,129: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:01:51,939: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:51,940: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:51,963: INFO/command_plan] add_jobs: created Job 715 (Test for host connectivity)
[2017-06-01 19:01:52,002: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:01:52,007: INFO/job_scheduler] Job 715: RunJobThread.run
[2017-06-01 19:01:52,036: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:52,036: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:52,063: INFO/command_plan] add_jobs: created Job 716 (Test for host connectivity)
[2017-06-01 19:01:52,098: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:01:52,116: INFO/job_scheduler] Job 716: RunJobThread.run
[2017-06-01 19:01:52,204: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:52,205: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:52,228: INFO/command_plan] add_jobs: created Job 717 (Test for host connectivity)
[2017-06-01 19:01:52,309: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:01:52,320: INFO/job_scheduler] Job 717: RunJobThread.run
[2017-06-01 19:01:52,385: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:52,385: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:52,415: INFO/command_plan] add_jobs: created Job 718 (Test for host connectivity)
[2017-06-01 19:01:52,446: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:01:52,448: INFO/job_scheduler] Job 718: RunJobThread.run
[2017-06-01 19:01:54,388: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:01:54,486: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:01:54,552: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:01:54,683: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:01:55,770: INFO/job_scheduler] Job 715 finished 1 steps successfully
[2017-06-01 19:01:55,853: INFO/job_scheduler] Job 715: complete_job: Updating cache
[2017-06-01 19:01:55,853: INFO/job_scheduler] Job 715 completing (errored=False, cancelled=False)
[2017-06-01 19:01:55,878: INFO/job_scheduler] Job 716 finished 1 steps successfully
[2017-06-01 19:01:55,920: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:01:55,936: INFO/job_scheduler] Job 716: complete_job: Updating cache
[2017-06-01 19:01:55,936: INFO/job_scheduler] Job 716 completing (errored=False, cancelled=False)
[2017-06-01 19:01:56,016: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:01:56,023: INFO/job_scheduler] Job 717 finished 1 steps successfully
[2017-06-01 19:01:56,048: INFO/job_scheduler] Job 717: complete_job: Updating cache
[2017-06-01 19:01:56,048: INFO/job_scheduler] Job 717 completing (errored=False, cancelled=False)
[2017-06-01 19:01:56,051: INFO/job_scheduler] Job 718 finished 1 steps successfully
[2017-06-01 19:01:56,118: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:01:56,144: INFO/job_scheduler] Job 718: complete_job: Updating cache
[2017-06-01 19:01:56,144: INFO/job_scheduler] Job 718 completing (errored=False, cancelled=False)
[2017-06-01 19:01:56,220: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:01:58,097: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:58,097: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:58,112: INFO/command_plan] add_jobs: created Job 719 (Test for host connectivity)
[2017-06-01 19:01:58,158: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:01:58,175: INFO/job_scheduler] Job 719: RunJobThread.run
[2017-06-01 19:01:58,216: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:58,222: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:58,229: INFO/command_plan] add_jobs: created Job 720 (Test for host connectivity)
[2017-06-01 19:01:58,295: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:01:58,297: INFO/job_scheduler] Job 720: RunJobThread.run
[2017-06-01 19:01:58,352: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:58,352: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:58,386: INFO/command_plan] add_jobs: created Job 721 (Test for host connectivity)
[2017-06-01 19:01:58,492: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:01:58,503: INFO/job_scheduler] Job 721: RunJobThread.run
[2017-06-01 19:01:58,524: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:01:58,524: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:01:58,563: INFO/command_plan] add_jobs: created Job 722 (Test for host connectivity)
[2017-06-01 19:01:58,622: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:01:58,670: INFO/job_scheduler] Job 722: RunJobThread.run
[2017-06-01 19:02:00,423: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:02:00,557: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:02:00,644: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:02:00,837: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:02:01,767: INFO/job_scheduler] Job 719 finished 1 steps successfully
[2017-06-01 19:02:01,785: INFO/job_scheduler] Job 719: complete_job: Updating cache
[2017-06-01 19:02:01,785: INFO/job_scheduler] Job 719 completing (errored=False, cancelled=False)
[2017-06-01 19:02:01,868: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:02:01,908: INFO/job_scheduler] Job 720 finished 1 steps successfully
[2017-06-01 19:02:01,969: INFO/job_scheduler] Job 720: complete_job: Updating cache
[2017-06-01 19:02:01,969: INFO/job_scheduler] Job 720 completing (errored=False, cancelled=False)
[2017-06-01 19:02:02,035: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:02:02,144: INFO/job_scheduler] Job 721 finished 1 steps successfully
[2017-06-01 19:02:02,211: INFO/job_scheduler] Job 721: complete_job: Updating cache
[2017-06-01 19:02:02,211: INFO/job_scheduler] Job 721 completing (errored=False, cancelled=False)
[2017-06-01 19:02:02,265: INFO/job_scheduler] Job 722 finished 1 steps successfully
[2017-06-01 19:02:02,273: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:02:02,285: INFO/job_scheduler] Job 722: complete_job: Updating cache
[2017-06-01 19:02:02,287: INFO/job_scheduler] Job 722 completing (errored=False, cancelled=False)
[2017-06-01 19:02:02,335: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:02:03,700: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:03,700: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:03,711: INFO/command_plan] add_jobs: created Job 723 (Test for host connectivity)
[2017-06-01 19:02:03,753: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:03,753: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:03,762: INFO/command_plan] add_jobs: created Job 724 (Test for host connectivity)
[2017-06-01 19:02:03,771: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:02:03,809: INFO/job_scheduler] Job 723: RunJobThread.run
[2017-06-01 19:02:03,829: INFO/job_scheduler] Job 724: RunJobThread.run
[2017-06-01 19:02:03,847: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:02:03,971: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:03,973: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:03,988: INFO/command_plan] add_jobs: created Job 725 (Test for host connectivity)
[2017-06-01 19:02:04,089: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:02:04,091: INFO/job_scheduler] Job 725: RunJobThread.run
[2017-06-01 19:02:04,157: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:04,157: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:04,169: INFO/command_plan] add_jobs: created Job 726 (Test for host connectivity)
[2017-06-01 19:02:04,257: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:02:04,261: INFO/job_scheduler] Job 726: RunJobThread.run
[2017-06-01 19:02:06,274: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:02:06,317: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:02:06,428: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:02:06,524: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:02:07,500: INFO/job_scheduler] Job 723 finished 1 steps successfully
[2017-06-01 19:02:07,544: INFO/job_scheduler] Job 723: complete_job: Updating cache
[2017-06-01 19:02:07,544: INFO/job_scheduler] Job 723 completing (errored=False, cancelled=False)
[2017-06-01 19:02:07,653: INFO/job_scheduler] Job 724 finished 1 steps successfully
[2017-06-01 19:02:07,660: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:02:07,740: INFO/job_scheduler] Job 724: complete_job: Updating cache
[2017-06-01 19:02:07,740: INFO/job_scheduler] Job 724 completing (errored=False, cancelled=False)
[2017-06-01 19:02:07,803: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:02:07,887: INFO/job_scheduler] Job 725 finished 1 steps successfully
[2017-06-01 19:02:07,897: INFO/job_scheduler] Job 726 finished 1 steps successfully
[2017-06-01 19:02:07,943: INFO/job_scheduler] Job 725: complete_job: Updating cache
[2017-06-01 19:02:07,943: INFO/job_scheduler] Job 725 completing (errored=False, cancelled=False)
[2017-06-01 19:02:08,019: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:02:08,026: INFO/job_scheduler] Job 726: complete_job: Updating cache
[2017-06-01 19:02:08,027: INFO/job_scheduler] Job 726 completing (errored=False, cancelled=False)
[2017-06-01 19:02:08,168: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:02:09,922: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:09,924: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:09,935: INFO/command_plan] add_jobs: created Job 727 (Test for host connectivity)
[2017-06-01 19:02:10,007: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:10,007: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:10,029: INFO/command_plan] add_jobs: created Job 728 (Test for host connectivity)
[2017-06-01 19:02:10,059: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:02:10,065: INFO/job_scheduler] Job 728: RunJobThread.run
[2017-06-01 19:02:10,066: INFO/job_scheduler] Job 727: RunJobThread.run
[2017-06-01 19:02:10,080: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:02:10,148: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:10,148: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:10,160: INFO/command_plan] add_jobs: created Job 729 (Test for host connectivity)
[2017-06-01 19:02:10,293: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:10,294: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:10,339: INFO/command_plan] add_jobs: created Job 730 (Test for host connectivity)
[2017-06-01 19:02:10,386: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:02:10,409: INFO/job_scheduler] Job 729: RunJobThread.run
[2017-06-01 19:02:10,439: INFO/job_scheduler] Job 730: RunJobThread.run
[2017-06-01 19:02:10,457: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:02:12,385: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:02:12,543: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:02:12,738: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:02:12,853: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:02:13,746: INFO/job_scheduler] Job 727 finished 1 steps successfully
[2017-06-01 19:02:13,775: INFO/job_scheduler] Job 727: complete_job: Updating cache
[2017-06-01 19:02:13,776: INFO/job_scheduler] Job 727 completing (errored=False, cancelled=False)
[2017-06-01 19:02:13,840: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:02:14,092: INFO/job_scheduler] Job 728 finished 1 steps successfully
[2017-06-01 19:02:14,142: INFO/job_scheduler] Job 728: complete_job: Updating cache
[2017-06-01 19:02:14,142: INFO/job_scheduler] Job 728 completing (errored=False, cancelled=False)
[2017-06-01 19:02:14,177: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:02:14,195: INFO/job_scheduler] Job 730 finished 1 steps successfully
[2017-06-01 19:02:14,251: INFO/job_scheduler] Job 730: complete_job: Updating cache
[2017-06-01 19:02:14,251: INFO/job_scheduler] Job 730 completing (errored=False, cancelled=False)
[2017-06-01 19:02:14,318: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:02:14,417: INFO/job_scheduler] Job 729 finished 1 steps successfully
[2017-06-01 19:02:14,450: INFO/job_scheduler] Job 729: complete_job: Updating cache
[2017-06-01 19:02:14,451: INFO/job_scheduler] Job 729 completing (errored=False, cancelled=False)
[2017-06-01 19:02:14,509: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:02:16,058: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:16,059: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:16,094: INFO/command_plan] add_jobs: created Job 731 (Test for host connectivity)
[2017-06-01 19:02:16,132: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:02:16,145: INFO/job_scheduler] Job 731: RunJobThread.run
[2017-06-01 19:02:16,179: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:16,179: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:16,224: INFO/command_plan] add_jobs: created Job 732 (Test for host connectivity)
[2017-06-01 19:02:16,295: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:02:16,336: INFO/job_scheduler] Job 732: RunJobThread.run
[2017-06-01 19:02:16,374: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:16,385: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:16,416: INFO/command_plan] add_jobs: created Job 733 (Test for host connectivity)
[2017-06-01 19:02:16,501: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:16,501: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:16,565: INFO/command_plan] add_jobs: created Job 734 (Test for host connectivity)
[2017-06-01 19:02:16,577: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:02:16,583: INFO/job_scheduler] Job 733: RunJobThread.run
[2017-06-01 19:02:16,584: INFO/job_scheduler] Job 734: RunJobThread.run
[2017-06-01 19:02:16,594: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:02:18,520: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:02:18,611: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:02:18,809: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:02:18,938: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:02:19,855: INFO/job_scheduler] Job 731 finished 1 steps successfully
[2017-06-01 19:02:19,917: INFO/job_scheduler] Job 731: complete_job: Updating cache
[2017-06-01 19:02:19,917: INFO/job_scheduler] Job 731 completing (errored=False, cancelled=False)
[2017-06-01 19:02:19,967: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:02:19,988: INFO/job_scheduler] Job 732 finished 1 steps successfully
[2017-06-01 19:02:20,027: INFO/job_scheduler] Job 732: complete_job: Updating cache
[2017-06-01 19:02:20,027: INFO/job_scheduler] Job 732 completing (errored=False, cancelled=False)
[2017-06-01 19:02:20,117: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:02:20,127: INFO/job_scheduler] Job 734 finished 1 steps successfully
[2017-06-01 19:02:20,158: INFO/job_scheduler] Job 734: complete_job: Updating cache
[2017-06-01 19:02:20,158: INFO/job_scheduler] Job 734 completing (errored=False, cancelled=False)
[2017-06-01 19:02:20,244: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:02:20,378: INFO/job_scheduler] Job 733 finished 1 steps successfully
[2017-06-01 19:02:20,400: INFO/job_scheduler] Job 733: complete_job: Updating cache
[2017-06-01 19:02:20,400: INFO/job_scheduler] Job 733 completing (errored=False, cancelled=False)
[2017-06-01 19:02:20,442: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:02:22,275: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:22,275: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:22,292: INFO/command_plan] add_jobs: created Job 735 (Test for host connectivity)
[2017-06-01 19:02:22,320: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:02:22,343: INFO/job_scheduler] Job 735: RunJobThread.run
[2017-06-01 19:02:22,397: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:22,400: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:22,494: INFO/command_plan] add_jobs: created Job 736 (Test for host connectivity)
[2017-06-01 19:02:22,571: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:22,571: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:22,606: INFO/command_plan] add_jobs: created Job 737 (Test for host connectivity)
[2017-06-01 19:02:22,624: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:02:22,655: INFO/job_scheduler] Job 736: RunJobThread.run
[2017-06-01 19:02:22,655: INFO/job_scheduler] Job 737: RunJobThread.run
[2017-06-01 19:02:22,694: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:22,695: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:22,734: INFO/command_plan] add_jobs: created Job 738 (Test for host connectivity)
[2017-06-01 19:02:22,794: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:02:22,884: INFO/job_scheduler] Job 738: RunJobThread.run
[2017-06-01 19:02:22,921: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:02:24,587: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:02:25,014: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:02:25,085: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:02:25,100: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:02:25,912: INFO/job_scheduler] Job 735 finished 1 steps successfully
[2017-06-01 19:02:25,928: INFO/job_scheduler] Job 735: complete_job: Updating cache
[2017-06-01 19:02:25,928: INFO/job_scheduler] Job 735 completing (errored=False, cancelled=False)
[2017-06-01 19:02:25,971: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:02:26,417: INFO/job_scheduler] Job 736 finished 1 steps successfully
[2017-06-01 19:02:26,495: INFO/job_scheduler] Job 736: complete_job: Updating cache
[2017-06-01 19:02:26,495: INFO/job_scheduler] Job 736 completing (errored=False, cancelled=False)
[2017-06-01 19:02:26,528: INFO/job_scheduler] Job 738 finished 1 steps successfully
[2017-06-01 19:02:26,537: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:02:26,553: INFO/job_scheduler] Job 738: complete_job: Updating cache
[2017-06-01 19:02:26,560: INFO/job_scheduler] Job 738 completing (errored=False, cancelled=False)
[2017-06-01 19:02:26,578: INFO/job_scheduler] Job 737 finished 1 steps successfully
[2017-06-01 19:02:26,663: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:02:26,712: INFO/job_scheduler] Job 737: complete_job: Updating cache
[2017-06-01 19:02:26,712: INFO/job_scheduler] Job 737 completing (errored=False, cancelled=False)
[2017-06-01 19:02:26,762: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:02:28,511: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:28,512: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:28,522: INFO/command_plan] add_jobs: created Job 739 (Test for host connectivity)
[2017-06-01 19:02:28,586: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:28,587: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:28,616: INFO/command_plan] add_jobs: created Job 740 (Test for host connectivity)
[2017-06-01 19:02:28,631: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:02:28,639: INFO/job_scheduler] Job 739: RunJobThread.run
[2017-06-01 19:02:28,640: INFO/job_scheduler] Job 740: RunJobThread.run
[2017-06-01 19:02:28,653: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:02:28,768: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:28,769: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:28,824: INFO/command_plan] add_jobs: created Job 741 (Test for host connectivity)
[2017-06-01 19:02:28,901: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:02:28,920: INFO/job_scheduler] Job 741: RunJobThread.run
[2017-06-01 19:02:28,966: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:28,971: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:28,986: INFO/command_plan] add_jobs: created Job 742 (Test for host connectivity)
[2017-06-01 19:02:29,020: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:02:29,052: INFO/job_scheduler] Job 742: RunJobThread.run
[2017-06-01 19:02:30,926: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:02:30,981: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:02:31,185: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:02:31,223: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:02:32,329: INFO/job_scheduler] Job 740 finished 1 steps successfully
[2017-06-01 19:02:32,371: INFO/job_scheduler] Job 739 finished 1 steps successfully
[2017-06-01 19:02:32,410: INFO/job_scheduler] Job 740: complete_job: Updating cache
[2017-06-01 19:02:32,410: INFO/job_scheduler] Job 740 completing (errored=False, cancelled=False)
[2017-06-01 19:02:32,519: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:02:32,530: INFO/job_scheduler] Job 739: complete_job: Updating cache
[2017-06-01 19:02:32,530: INFO/job_scheduler] Job 739 completing (errored=False, cancelled=False)
[2017-06-01 19:02:32,568: INFO/job_scheduler] Job 742 finished 1 steps successfully
[2017-06-01 19:02:32,625: INFO/job_scheduler] Job 741 finished 1 steps successfully
[2017-06-01 19:02:32,652: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:02:32,666: INFO/job_scheduler] Job 742: complete_job: Updating cache
[2017-06-01 19:02:32,666: INFO/job_scheduler] Job 742 completing (errored=False, cancelled=False)
[2017-06-01 19:02:32,694: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:02:32,704: INFO/job_scheduler] Job 741: complete_job: Updating cache
[2017-06-01 19:02:32,704: INFO/job_scheduler] Job 741 completing (errored=False, cancelled=False)
[2017-06-01 19:02:32,768: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:02:34,717: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:34,719: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:34,742: INFO/command_plan] add_jobs: created Job 743 (Test for host connectivity)
[2017-06-01 19:02:34,782: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:02:34,866: INFO/job_scheduler] Job 743: RunJobThread.run
[2017-06-01 19:02:34,897: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:34,897: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:34,921: INFO/command_plan] add_jobs: created Job 744 (Test for host connectivity)
[2017-06-01 19:02:34,955: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:02:34,969: INFO/job_scheduler] Job 744: RunJobThread.run
[2017-06-01 19:02:34,990: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:34,991: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:35,110: INFO/command_plan] add_jobs: created Job 745 (Test for host connectivity)
[2017-06-01 19:02:35,185: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:35,185: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:35,270: INFO/command_plan] add_jobs: created Job 746 (Test for host connectivity)
[2017-06-01 19:02:35,285: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:02:35,311: INFO/job_scheduler] Job 745: RunJobThread.run
[2017-06-01 19:02:35,323: INFO/job_scheduler] Job 746: RunJobThread.run
[2017-06-01 19:02:35,336: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:02:37,236: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:02:37,325: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:02:37,526: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:02:37,748: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:02:38,588: INFO/job_scheduler] Job 743 finished 1 steps successfully
[2017-06-01 19:02:38,617: INFO/job_scheduler] Job 743: complete_job: Updating cache
[2017-06-01 19:02:38,617: INFO/job_scheduler] Job 743 completing (errored=False, cancelled=False)
[2017-06-01 19:02:38,653: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:02:38,672: INFO/job_scheduler] Job 744 finished 1 steps successfully
[2017-06-01 19:02:38,691: INFO/job_scheduler] Job 744: complete_job: Updating cache
[2017-06-01 19:02:38,691: INFO/job_scheduler] Job 744 completing (errored=False, cancelled=False)
[2017-06-01 19:02:38,802: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:02:38,854: INFO/job_scheduler] Job 746 finished 1 steps successfully
[2017-06-01 19:02:38,888: INFO/job_scheduler] Job 746: complete_job: Updating cache
[2017-06-01 19:02:38,888: INFO/job_scheduler] Job 746 completing (errored=False, cancelled=False)
[2017-06-01 19:02:38,954: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:02:39,244: INFO/job_scheduler] Job 745 finished 1 steps successfully
[2017-06-01 19:02:39,275: INFO/job_scheduler] Job 745: complete_job: Updating cache
[2017-06-01 19:02:39,276: INFO/job_scheduler] Job 745 completing (errored=False, cancelled=False)
[2017-06-01 19:02:39,343: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:02:41,019: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:41,021: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:41,062: INFO/command_plan] add_jobs: created Job 747 (Test for host connectivity)
[2017-06-01 19:02:41,102: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:41,103: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:41,126: INFO/command_plan] add_jobs: created Job 748 (Test for host connectivity)
[2017-06-01 19:02:41,135: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:02:41,188: INFO/job_scheduler] Job 747: RunJobThread.run
[2017-06-01 19:02:41,189: INFO/job_scheduler] Job 748: RunJobThread.run
[2017-06-01 19:02:41,239: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:41,255: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:41,287: INFO/command_plan] add_jobs: created Job 749 (Test for host connectivity)
[2017-06-01 19:02:41,355: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:02:41,388: INFO/job_scheduler] Job 749: RunJobThread.run
[2017-06-01 19:02:41,442: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:41,444: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:41,453: INFO/command_plan] add_jobs: created Job 750 (Test for host connectivity)
[2017-06-01 19:02:41,481: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:02:41,487: INFO/job_scheduler] Job 750: RunJobThread.run
[2017-06-01 19:02:41,565: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:02:43,351: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:02:43,494: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:02:43,610: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:02:43,835: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:02:44,680: INFO/job_scheduler] Job 748 finished 1 steps successfully
[2017-06-01 19:02:44,708: INFO/job_scheduler] Job 748: complete_job: Updating cache
[2017-06-01 19:02:44,708: INFO/job_scheduler] Job 748 completing (errored=False, cancelled=False)
[2017-06-01 19:02:44,779: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:02:44,792: INFO/job_scheduler] Job 747 finished 1 steps successfully
[2017-06-01 19:02:44,836: INFO/job_scheduler] Job 747: complete_job: Updating cache
[2017-06-01 19:02:44,837: INFO/job_scheduler] Job 747 completing (errored=False, cancelled=False)
[2017-06-01 19:02:44,876: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:02:45,048: INFO/job_scheduler] Job 749 finished 1 steps successfully
[2017-06-01 19:02:45,074: INFO/job_scheduler] Job 749: complete_job: Updating cache
[2017-06-01 19:02:45,074: INFO/job_scheduler] Job 749 completing (errored=False, cancelled=False)
[2017-06-01 19:02:45,110: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:02:45,272: INFO/job_scheduler] Job 750 finished 1 steps successfully
[2017-06-01 19:02:45,316: INFO/job_scheduler] Job 750: complete_job: Updating cache
[2017-06-01 19:02:45,316: INFO/job_scheduler] Job 750 completing (errored=False, cancelled=False)
[2017-06-01 19:02:45,350: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:02:47,193: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:47,194: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:47,209: INFO/command_plan] add_jobs: created Job 751 (Test for host connectivity)
[2017-06-01 19:02:47,227: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:02:47,244: INFO/job_scheduler] Job 751: RunJobThread.run
[2017-06-01 19:02:47,326: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:47,327: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:47,340: INFO/command_plan] add_jobs: created Job 752 (Test for host connectivity)
[2017-06-01 19:02:47,410: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:02:47,412: INFO/job_scheduler] Job 752: RunJobThread.run
[2017-06-01 19:02:47,475: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:47,477: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:47,501: INFO/command_plan] add_jobs: created Job 753 (Test for host connectivity)
[2017-06-01 19:02:47,614: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:47,615: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:47,633: INFO/command_plan] add_jobs: created Job 754 (Test for host connectivity)
[2017-06-01 19:02:47,683: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:02:47,706: INFO/job_scheduler] Job 753: RunJobThread.run
[2017-06-01 19:02:47,720: INFO/job_scheduler] Job 754: RunJobThread.run
[2017-06-01 19:02:47,735: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:02:49,687: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:02:49,743: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:02:49,957: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:02:50,045: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:02:51,024: INFO/job_scheduler] Job 751 finished 1 steps successfully
[2017-06-01 19:02:51,085: INFO/job_scheduler] Job 751: complete_job: Updating cache
[2017-06-01 19:02:51,085: INFO/job_scheduler] Job 751 completing (errored=False, cancelled=False)
[2017-06-01 19:02:51,089: INFO/job_scheduler] Job 752 finished 1 steps successfully
[2017-06-01 19:02:51,128: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:02:51,149: INFO/job_scheduler] Job 752: complete_job: Updating cache
[2017-06-01 19:02:51,153: INFO/job_scheduler] Job 752 completing (errored=False, cancelled=False)
[2017-06-01 19:02:51,231: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:02:51,274: INFO/job_scheduler] Job 754 finished 1 steps successfully
[2017-06-01 19:02:51,298: INFO/job_scheduler] Job 754: complete_job: Updating cache
[2017-06-01 19:02:51,299: INFO/job_scheduler] Job 754 completing (errored=False, cancelled=False)
[2017-06-01 19:02:51,375: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:02:51,384: INFO/job_scheduler] Job 753 finished 1 steps successfully
[2017-06-01 19:02:51,418: INFO/job_scheduler] Job 753: complete_job: Updating cache
[2017-06-01 19:02:51,418: INFO/job_scheduler] Job 753 completing (errored=False, cancelled=False)
[2017-06-01 19:02:51,481: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:02:53,512: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:53,514: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:53,532: INFO/command_plan] add_jobs: created Job 755 (Test for host connectivity)
[2017-06-01 19:02:53,561: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:02:53,564: INFO/job_scheduler] Job 755: RunJobThread.run
[2017-06-01 19:02:53,602: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:53,603: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:53,656: INFO/command_plan] add_jobs: created Job 756 (Test for host connectivity)
[2017-06-01 19:02:53,694: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:02:53,712: INFO/job_scheduler] Job 756: RunJobThread.run
[2017-06-01 19:02:53,783: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:53,785: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:53,939: INFO/command_plan] add_jobs: created Job 757 (Test for host connectivity)
[2017-06-01 19:02:53,978: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:53,978: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:53,992: INFO/command_plan] add_jobs: created Job 758 (Test for host connectivity)
[2017-06-01 19:02:54,008: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:02:54,020: INFO/job_scheduler] Job 757: RunJobThread.run
[2017-06-01 19:02:54,053: INFO/job_scheduler] Job 758: RunJobThread.run
[2017-06-01 19:02:54,066: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:02:55,882: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:02:56,078: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:02:56,473: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:02:56,490: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:02:57,281: INFO/job_scheduler] Job 755 finished 1 steps successfully
[2017-06-01 19:02:57,330: INFO/job_scheduler] Job 755: complete_job: Updating cache
[2017-06-01 19:02:57,330: INFO/job_scheduler] Job 755 completing (errored=False, cancelled=False)
[2017-06-01 19:02:57,365: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:02:57,508: INFO/job_scheduler] Job 756 finished 1 steps successfully
[2017-06-01 19:02:57,563: INFO/job_scheduler] Job 756: complete_job: Updating cache
[2017-06-01 19:02:57,563: INFO/job_scheduler] Job 756 completing (errored=False, cancelled=False)
[2017-06-01 19:02:57,772: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:02:57,890: INFO/job_scheduler] Job 758 finished 1 steps successfully
[2017-06-01 19:02:57,905: INFO/job_scheduler] Job 758: complete_job: Updating cache
[2017-06-01 19:02:57,906: INFO/job_scheduler] Job 758 completing (errored=False, cancelled=False)
[2017-06-01 19:02:57,946: INFO/job_scheduler] Job 757 finished 1 steps successfully
[2017-06-01 19:02:57,947: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:02:57,972: INFO/job_scheduler] Job 757: complete_job: Updating cache
[2017-06-01 19:02:57,972: INFO/job_scheduler] Job 757 completing (errored=False, cancelled=False)
[2017-06-01 19:02:58,041: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:02:59,735: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:59,736: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:59,747: INFO/command_plan] add_jobs: created Job 759 (Test for host connectivity)
[2017-06-01 19:02:59,759: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:02:59,776: INFO/job_scheduler] Job 759: RunJobThread.run
[2017-06-01 19:02:59,850: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:02:59,850: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:02:59,901: INFO/command_plan] add_jobs: created Job 760 (Test for host connectivity)
[2017-06-01 19:02:59,935: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:02:59,971: INFO/job_scheduler] Job 760: RunJobThread.run
[2017-06-01 19:03:00,031: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:00,031: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:00,078: INFO/command_plan] add_jobs: created Job 761 (Test for host connectivity)
[2017-06-01 19:03:00,228: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:00,229: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:00,248: INFO/command_plan] add_jobs: created Job 762 (Test for host connectivity)
[2017-06-01 19:03:00,280: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:03:00,283: INFO/job_scheduler] Job 761: RunJobThread.run
[2017-06-01 19:03:00,284: INFO/job_scheduler] Job 762: RunJobThread.run
[2017-06-01 19:03:00,289: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:03:02,199: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:03:02,293: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:03:02,388: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:03:02,618: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:03:03,544: INFO/job_scheduler] Job 759 finished 1 steps successfully
[2017-06-01 19:03:03,570: INFO/job_scheduler] Job 759: complete_job: Updating cache
[2017-06-01 19:03:03,571: INFO/job_scheduler] Job 759 completing (errored=False, cancelled=False)
[2017-06-01 19:03:03,640: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:03:03,696: INFO/job_scheduler] Job 760 finished 1 steps successfully
[2017-06-01 19:03:03,741: INFO/job_scheduler] Job 761 finished 1 steps successfully
[2017-06-01 19:03:03,762: INFO/job_scheduler] Job 760: complete_job: Updating cache
[2017-06-01 19:03:03,762: INFO/job_scheduler] Job 760 completing (errored=False, cancelled=False)
[2017-06-01 19:03:03,861: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:03:03,898: INFO/job_scheduler] Job 761: complete_job: Updating cache
[2017-06-01 19:03:03,899: INFO/job_scheduler] Job 761 completing (errored=False, cancelled=False)
[2017-06-01 19:03:03,955: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:03:04,029: INFO/job_scheduler] Job 762 finished 1 steps successfully
[2017-06-01 19:03:04,070: INFO/job_scheduler] Job 762: complete_job: Updating cache
[2017-06-01 19:03:04,071: INFO/job_scheduler] Job 762 completing (errored=False, cancelled=False)
[2017-06-01 19:03:04,122: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:03:05,959: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:05,959: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:05,981: INFO/command_plan] add_jobs: created Job 763 (Test for host connectivity)
[2017-06-01 19:03:06,037: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:06,037: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:06,054: INFO/command_plan] add_jobs: created Job 764 (Test for host connectivity)
[2017-06-01 19:03:06,063: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:03:06,120: INFO/job_scheduler] Job 763: RunJobThread.run
[2017-06-01 19:03:06,135: INFO/job_scheduler] Job 764: RunJobThread.run
[2017-06-01 19:03:06,175: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:06,175: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:06,206: INFO/command_plan] add_jobs: created Job 765 (Test for host connectivity)
[2017-06-01 19:03:06,238: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:03:06,272: INFO/job_scheduler] Job 765: RunJobThread.run
[2017-06-01 19:03:06,416: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:06,416: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:06,444: INFO/command_plan] add_jobs: created Job 766 (Test for host connectivity)
[2017-06-01 19:03:06,509: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:03:06,535: INFO/job_scheduler] Job 766: RunJobThread.run
[2017-06-01 19:03:06,554: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:03:08,371: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:03:08,470: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:03:08,572: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:03:08,723: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:03:09,710: INFO/job_scheduler] Job 764 finished 1 steps successfully
[2017-06-01 19:03:09,744: INFO/job_scheduler] Job 764: complete_job: Updating cache
[2017-06-01 19:03:09,744: INFO/job_scheduler] Job 764 completing (errored=False, cancelled=False)
[2017-06-01 19:03:09,819: INFO/job_scheduler] Job 763 finished 1 steps successfully
[2017-06-01 19:03:09,845: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:03:09,945: INFO/job_scheduler] Job 763: complete_job: Updating cache
[2017-06-01 19:03:09,945: INFO/job_scheduler] Job 763 completing (errored=False, cancelled=False)
[2017-06-01 19:03:09,996: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:03:10,089: INFO/job_scheduler] Job 765 finished 1 steps successfully
[2017-06-01 19:03:10,091: INFO/job_scheduler] Job 766 finished 1 steps successfully
[2017-06-01 19:03:10,111: INFO/job_scheduler] Job 765: complete_job: Updating cache
[2017-06-01 19:03:10,111: INFO/job_scheduler] Job 765 completing (errored=False, cancelled=False)
[2017-06-01 19:03:10,148: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:03:10,179: INFO/job_scheduler] Job 766: complete_job: Updating cache
[2017-06-01 19:03:10,179: INFO/job_scheduler] Job 766 completing (errored=False, cancelled=False)
[2017-06-01 19:03:10,253: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:03:12,194: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:12,195: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:12,218: INFO/command_plan] add_jobs: created Job 767 (Test for host connectivity)
[2017-06-01 19:03:12,259: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:03:12,270: INFO/job_scheduler] Job 767: RunJobThread.run
[2017-06-01 19:03:12,301: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:12,302: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:12,336: INFO/command_plan] add_jobs: created Job 768 (Test for host connectivity)
[2017-06-01 19:03:12,393: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:03:12,398: INFO/job_scheduler] Job 768: RunJobThread.run
[2017-06-01 19:03:12,464: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:12,466: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:12,477: INFO/command_plan] add_jobs: created Job 769 (Test for host connectivity)
[2017-06-01 19:03:12,538: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:03:12,558: INFO/job_scheduler] Job 769: RunJobThread.run
[2017-06-01 19:03:12,614: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:12,615: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:12,652: INFO/command_plan] add_jobs: created Job 770 (Test for host connectivity)
[2017-06-01 19:03:12,690: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:03:12,691: INFO/job_scheduler] Job 770: RunJobThread.run
[2017-06-01 19:03:14,488: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:03:14,642: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:03:14,737: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:03:14,836: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:03:15,795: INFO/job_scheduler] Job 767 finished 1 steps successfully
[2017-06-01 19:03:15,823: INFO/job_scheduler] Job 767: complete_job: Updating cache
[2017-06-01 19:03:15,824: INFO/job_scheduler] Job 767 completing (errored=False, cancelled=False)
[2017-06-01 19:03:15,893: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:03:15,995: INFO/job_scheduler] Job 768 finished 1 steps successfully
[2017-06-01 19:03:16,056: INFO/job_scheduler] Job 769 finished 1 steps successfully
[2017-06-01 19:03:16,060: INFO/job_scheduler] Job 768: complete_job: Updating cache
[2017-06-01 19:03:16,060: INFO/job_scheduler] Job 768 completing (errored=False, cancelled=False)
[2017-06-01 19:03:16,136: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:03:16,154: INFO/job_scheduler] Job 769: complete_job: Updating cache
[2017-06-01 19:03:16,154: INFO/job_scheduler] Job 769 completing (errored=False, cancelled=False)
[2017-06-01 19:03:16,203: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:03:16,250: INFO/job_scheduler] Job 770 finished 1 steps successfully
[2017-06-01 19:03:16,285: INFO/job_scheduler] Job 770: complete_job: Updating cache
[2017-06-01 19:03:16,285: INFO/job_scheduler] Job 770 completing (errored=False, cancelled=False)
[2017-06-01 19:03:16,345: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:03:18,359: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:18,359: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:18,383: INFO/command_plan] add_jobs: created Job 771 (Test for host connectivity)
[2017-06-01 19:03:18,445: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:03:18,469: INFO/job_scheduler] Job 771: RunJobThread.run
[2017-06-01 19:03:18,507: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:18,508: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:18,518: INFO/command_plan] add_jobs: created Job 772 (Test for host connectivity)
[2017-06-01 19:03:18,575: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:03:18,606: INFO/job_scheduler] Job 772: RunJobThread.run
[2017-06-01 19:03:18,647: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:18,649: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:18,684: INFO/command_plan] add_jobs: created Job 773 (Test for host connectivity)
[2017-06-01 19:03:18,737: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:03:18,740: INFO/job_scheduler] Job 773: RunJobThread.run
[2017-06-01 19:03:18,776: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:18,777: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:18,802: INFO/command_plan] add_jobs: created Job 774 (Test for host connectivity)
[2017-06-01 19:03:18,866: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:03:18,884: INFO/job_scheduler] Job 774: RunJobThread.run
[2017-06-01 19:03:20,723: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:03:20,875: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:03:21,079: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:03:21,103: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:03:22,091: INFO/job_scheduler] Job 771 finished 1 steps successfully
[2017-06-01 19:03:22,190: INFO/job_scheduler] Job 771: complete_job: Updating cache
[2017-06-01 19:03:22,190: INFO/job_scheduler] Job 771 completing (errored=False, cancelled=False)
[2017-06-01 19:03:22,202: INFO/job_scheduler] Job 772 finished 1 steps successfully
[2017-06-01 19:03:22,232: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:03:22,365: INFO/job_scheduler] Job 772: complete_job: Updating cache
[2017-06-01 19:03:22,365: INFO/job_scheduler] Job 772 completing (errored=False, cancelled=False)
[2017-06-01 19:03:22,416: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:03:22,456: INFO/job_scheduler] Job 774 finished 1 steps successfully
[2017-06-01 19:03:22,464: INFO/job_scheduler] Job 773 finished 1 steps successfully
[2017-06-01 19:03:22,517: INFO/job_scheduler] Job 774: complete_job: Updating cache
[2017-06-01 19:03:22,517: INFO/job_scheduler] Job 774 completing (errored=False, cancelled=False)
[2017-06-01 19:03:22,583: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:03:22,611: INFO/job_scheduler] Job 773: complete_job: Updating cache
[2017-06-01 19:03:22,611: INFO/job_scheduler] Job 773 completing (errored=False, cancelled=False)
[2017-06-01 19:03:22,701: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:03:24,549: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:24,549: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:24,557: INFO/command_plan] add_jobs: created Job 775 (Test for host connectivity)
[2017-06-01 19:03:24,577: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:03:24,593: INFO/job_scheduler] Job 775: RunJobThread.run
[2017-06-01 19:03:24,647: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:24,647: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:24,673: INFO/command_plan] add_jobs: created Job 776 (Test for host connectivity)
[2017-06-01 19:03:24,693: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:03:24,709: INFO/job_scheduler] Job 776: RunJobThread.run
[2017-06-01 19:03:24,768: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:24,769: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:24,804: INFO/command_plan] add_jobs: created Job 777 (Test for host connectivity)
[2017-06-01 19:03:24,868: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:03:24,897: INFO/job_scheduler] Job 777: RunJobThread.run
[2017-06-01 19:03:24,963: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:24,966: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:24,984: INFO/command_plan] add_jobs: created Job 778 (Test for host connectivity)
[2017-06-01 19:03:25,006: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:03:25,022: INFO/job_scheduler] Job 778: RunJobThread.run
[2017-06-01 19:03:26,995: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:03:27,109: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:03:27,255: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:03:27,299: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:03:28,306: INFO/job_scheduler] Job 775 finished 1 steps successfully
[2017-06-01 19:03:28,363: INFO/job_scheduler] Job 775: complete_job: Updating cache
[2017-06-01 19:03:28,364: INFO/job_scheduler] Job 775 completing (errored=False, cancelled=False)
[2017-06-01 19:03:28,414: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:03:28,460: INFO/job_scheduler] Job 776 finished 1 steps successfully
[2017-06-01 19:03:28,497: INFO/job_scheduler] Job 776: complete_job: Updating cache
[2017-06-01 19:03:28,497: INFO/job_scheduler] Job 776 completing (errored=False, cancelled=False)
[2017-06-01 19:03:28,521: INFO/job_scheduler] Job 777 finished 1 steps successfully
[2017-06-01 19:03:28,581: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:03:28,588: INFO/job_scheduler] Job 777: complete_job: Updating cache
[2017-06-01 19:03:28,588: INFO/job_scheduler] Job 777 completing (errored=False, cancelled=False)
[2017-06-01 19:03:28,650: INFO/job_scheduler] Job 778 finished 1 steps successfully
[2017-06-01 19:03:28,651: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:03:28,664: INFO/job_scheduler] Job 778: complete_job: Updating cache
[2017-06-01 19:03:28,664: INFO/job_scheduler] Job 778 completing (errored=False, cancelled=False)
[2017-06-01 19:03:28,732: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:03:31,672: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:31,673: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:31,717: INFO/command_plan] add_jobs: created Job 779 (Test for host connectivity)
[2017-06-01 19:03:31,734: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:03:31,736: INFO/job_scheduler] Job 779: RunJobThread.run
[2017-06-01 19:03:31,815: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:31,816: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:31,855: INFO/command_plan] add_jobs: created Job 780 (Test for host connectivity)
[2017-06-01 19:03:31,942: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:03:31,950: INFO/job_scheduler] Job 780: RunJobThread.run
[2017-06-01 19:03:31,990: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:31,990: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:32,005: INFO/command_plan] add_jobs: created Job 781 (Test for host connectivity)
[2017-06-01 19:03:32,042: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:03:32,101: INFO/job_scheduler] Job 781: RunJobThread.run
[2017-06-01 19:03:32,175: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:32,176: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:32,213: INFO/command_plan] add_jobs: created Job 782 (Test for host connectivity)
[2017-06-01 19:03:32,277: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:03:32,298: INFO/job_scheduler] Job 782: RunJobThread.run
[2017-06-01 19:03:33,921: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:03:34,246: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:03:34,326: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:03:34,513: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:03:35,247: INFO/job_scheduler] Job 779 finished 1 steps successfully
[2017-06-01 19:03:35,387: INFO/job_scheduler] Job 779: complete_job: Updating cache
[2017-06-01 19:03:35,387: INFO/job_scheduler] Job 779 completing (errored=False, cancelled=False)
[2017-06-01 19:03:35,465: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:03:35,760: INFO/job_scheduler] Job 780 finished 1 steps successfully
[2017-06-01 19:03:35,803: INFO/job_scheduler] Job 780: complete_job: Updating cache
[2017-06-01 19:03:35,804: INFO/job_scheduler] Job 780 completing (errored=False, cancelled=False)
[2017-06-01 19:03:35,870: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:03:35,872: INFO/job_scheduler] Job 782 finished 1 steps successfully
[2017-06-01 19:03:35,905: INFO/job_scheduler] Job 782: complete_job: Updating cache
[2017-06-01 19:03:35,905: INFO/job_scheduler] Job 782 completing (errored=False, cancelled=False)
[2017-06-01 19:03:35,909: INFO/job_scheduler] Job 781 finished 1 steps successfully
[2017-06-01 19:03:36,013: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:03:36,054: INFO/job_scheduler] Job 781: complete_job: Updating cache
[2017-06-01 19:03:36,054: INFO/job_scheduler] Job 781 completing (errored=False, cancelled=False)
[2017-06-01 19:03:36,106: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:03:37,924: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:37,924: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:37,975: INFO/command_plan] add_jobs: created Job 783 (Test for host connectivity)
[2017-06-01 19:03:38,043: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:03:38,046: INFO/job_scheduler] Job 783: RunJobThread.run
[2017-06-01 19:03:38,088: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:38,089: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:38,115: INFO/command_plan] add_jobs: created Job 784 (Test for host connectivity)
[2017-06-01 19:03:38,149: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:03:38,170: INFO/job_scheduler] Job 784: RunJobThread.run
[2017-06-01 19:03:38,231: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:38,232: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:38,273: INFO/command_plan] add_jobs: created Job 785 (Test for host connectivity)
[2017-06-01 19:03:38,337: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:03:38,363: INFO/job_scheduler] Job 785: RunJobThread.run
[2017-06-01 19:03:38,437: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:38,437: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:38,462: INFO/command_plan] add_jobs: created Job 786 (Test for host connectivity)
[2017-06-01 19:03:38,533: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:03:38,537: INFO/job_scheduler] Job 786: RunJobThread.run
[2017-06-01 19:03:40,337: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:03:40,512: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:03:40,737: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:03:40,746: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:03:41,730: INFO/job_scheduler] Job 783 finished 1 steps successfully
[2017-06-01 19:03:41,765: INFO/job_scheduler] Job 783: complete_job: Updating cache
[2017-06-01 19:03:41,765: INFO/job_scheduler] Job 783 completing (errored=False, cancelled=False)
[2017-06-01 19:03:41,867: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:03:41,869: INFO/job_scheduler] Job 784 finished 1 steps successfully
[2017-06-01 19:03:41,972: INFO/job_scheduler] Job 784: complete_job: Updating cache
[2017-06-01 19:03:41,972: INFO/job_scheduler] Job 784 completing (errored=False, cancelled=False)
[2017-06-01 19:03:42,130: INFO/job_scheduler] Job 785 finished 1 steps successfully
[2017-06-01 19:03:42,202: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:03:42,214: INFO/job_scheduler] Job 786 finished 1 steps successfully
[2017-06-01 19:03:42,261: INFO/job_scheduler] Job 785: complete_job: Updating cache
[2017-06-01 19:03:42,261: INFO/job_scheduler] Job 785 completing (errored=False, cancelled=False)
[2017-06-01 19:03:42,302: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:03:42,318: INFO/job_scheduler] Job 786: complete_job: Updating cache
[2017-06-01 19:03:42,319: INFO/job_scheduler] Job 786 completing (errored=False, cancelled=False)
[2017-06-01 19:03:42,403: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:03:44,200: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:44,200: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:44,219: INFO/command_plan] add_jobs: created Job 787 (Test for host connectivity)
[2017-06-01 19:03:44,286: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:44,286: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:44,294: INFO/command_plan] add_jobs: created Job 788 (Test for host connectivity)
[2017-06-01 19:03:44,339: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:03:44,344: INFO/job_scheduler] Job 787: RunJobThread.run
[2017-06-01 19:03:44,355: INFO/job_scheduler] Job 788: RunJobThread.run
[2017-06-01 19:03:44,362: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:03:44,421: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:44,421: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:44,472: INFO/command_plan] add_jobs: created Job 789 (Test for host connectivity)
[2017-06-01 19:03:44,533: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:03:44,562: INFO/job_scheduler] Job 789: RunJobThread.run
[2017-06-01 19:03:44,623: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:44,625: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:44,638: INFO/command_plan] add_jobs: created Job 790 (Test for host connectivity)
[2017-06-01 19:03:44,701: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:03:44,717: INFO/job_scheduler] Job 790: RunJobThread.run
[2017-06-01 19:03:46,611: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:03:46,618: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:03:46,869: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:03:46,953: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:03:47,961: INFO/job_scheduler] Job 788 finished 1 steps successfully
[2017-06-01 19:03:47,974: INFO/job_scheduler] Job 787 finished 1 steps successfully
[2017-06-01 19:03:47,993: INFO/job_scheduler] Job 788: complete_job: Updating cache
[2017-06-01 19:03:47,993: INFO/job_scheduler] Job 788 completing (errored=False, cancelled=False)
[2017-06-01 19:03:48,087: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:03:48,110: INFO/job_scheduler] Job 787: complete_job: Updating cache
[2017-06-01 19:03:48,110: INFO/job_scheduler] Job 787 completing (errored=False, cancelled=False)
[2017-06-01 19:03:48,234: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:03:48,330: INFO/job_scheduler] Job 790 finished 1 steps successfully
[2017-06-01 19:03:48,351: INFO/job_scheduler] Job 790: complete_job: Updating cache
[2017-06-01 19:03:48,351: INFO/job_scheduler] Job 790 completing (errored=False, cancelled=False)
[2017-06-01 19:03:48,363: INFO/job_scheduler] Job 789 finished 1 steps successfully
[2017-06-01 19:03:48,385: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:03:48,416: INFO/job_scheduler] Job 789: complete_job: Updating cache
[2017-06-01 19:03:48,416: INFO/job_scheduler] Job 789 completing (errored=False, cancelled=False)
[2017-06-01 19:03:48,502: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:03:50,380: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:50,382: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:50,402: INFO/command_plan] add_jobs: created Job 791 (Test for host connectivity)
[2017-06-01 19:03:50,435: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:03:50,448: INFO/job_scheduler] Job 791: RunJobThread.run
[2017-06-01 19:03:50,507: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:50,509: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:50,526: INFO/command_plan] add_jobs: created Job 792 (Test for host connectivity)
[2017-06-01 19:03:50,579: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:03:50,601: INFO/job_scheduler] Job 792: RunJobThread.run
[2017-06-01 19:03:50,657: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:50,660: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:50,676: INFO/command_plan] add_jobs: created Job 793 (Test for host connectivity)
[2017-06-01 19:03:50,694: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:03:50,720: INFO/job_scheduler] Job 793: RunJobThread.run
[2017-06-01 19:03:50,787: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:50,788: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:50,814: INFO/command_plan] add_jobs: created Job 794 (Test for host connectivity)
[2017-06-01 19:03:50,899: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:03:50,915: INFO/job_scheduler] Job 794: RunJobThread.run
[2017-06-01 19:03:52,700: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:03:52,852: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:03:53,045: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:03:53,069: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:03:54,009: INFO/job_scheduler] Job 791 finished 1 steps successfully
[2017-06-01 19:03:54,042: INFO/job_scheduler] Job 791: complete_job: Updating cache
[2017-06-01 19:03:54,042: INFO/job_scheduler] Job 791 completing (errored=False, cancelled=False)
[2017-06-01 19:03:54,135: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:03:54,192: INFO/job_scheduler] Job 792 finished 1 steps successfully
[2017-06-01 19:03:54,244: INFO/job_scheduler] Job 792: complete_job: Updating cache
[2017-06-01 19:03:54,244: INFO/job_scheduler] Job 792 completing (errored=False, cancelled=False)
[2017-06-01 19:03:54,295: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:03:54,429: INFO/job_scheduler] Job 794 finished 1 steps successfully
[2017-06-01 19:03:54,475: INFO/job_scheduler] Job 794: complete_job: Updating cache
[2017-06-01 19:03:54,475: INFO/job_scheduler] Job 794 completing (errored=False, cancelled=False)
[2017-06-01 19:03:54,526: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:03:54,637: INFO/job_scheduler] Job 793 finished 1 steps successfully
[2017-06-01 19:03:54,675: INFO/job_scheduler] Job 793: complete_job: Updating cache
[2017-06-01 19:03:54,675: INFO/job_scheduler] Job 793 completing (errored=False, cancelled=False)
[2017-06-01 19:03:54,717: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:03:56,536: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:56,537: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:56,549: INFO/command_plan] add_jobs: created Job 795 (Test for host connectivity)
[2017-06-01 19:03:56,598: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:03:56,621: INFO/job_scheduler] Job 795: RunJobThread.run
[2017-06-01 19:03:56,641: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:56,642: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:56,782: INFO/command_plan] add_jobs: created Job 796 (Test for host connectivity)
[2017-06-01 19:03:56,841: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:56,842: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:56,866: INFO/command_plan] add_jobs: created Job 797 (Test for host connectivity)
[2017-06-01 19:03:56,903: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:03:56,916: INFO/job_scheduler] Job 796: RunJobThread.run
[2017-06-01 19:03:56,949: INFO/job_scheduler] Job 797: RunJobThread.run
[2017-06-01 19:03:56,963: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:03:57,026: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:03:57,027: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:03:57,033: INFO/command_plan] add_jobs: created Job 798 (Test for host connectivity)
[2017-06-01 19:03:57,200: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:03:57,203: INFO/job_scheduler] Job 798: RunJobThread.run
[2017-06-01 19:03:58,897: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:03:59,144: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:03:59,241: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:03:59,305: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:04:00,225: INFO/job_scheduler] Job 795 finished 1 steps successfully
[2017-06-01 19:04:00,254: INFO/job_scheduler] Job 795: complete_job: Updating cache
[2017-06-01 19:04:00,254: INFO/job_scheduler] Job 795 completing (errored=False, cancelled=False)
[2017-06-01 19:04:00,314: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:04:00,553: INFO/job_scheduler] Job 796 finished 1 steps successfully
[2017-06-01 19:04:00,615: INFO/job_scheduler] Job 796: complete_job: Updating cache
[2017-06-01 19:04:00,615: INFO/job_scheduler] Job 796 completing (errored=False, cancelled=False)
[2017-06-01 19:04:00,676: INFO/job_scheduler] Job 797 finished 1 steps successfully
[2017-06-01 19:04:00,676: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:04:00,704: INFO/job_scheduler] Job 797: complete_job: Updating cache
[2017-06-01 19:04:00,704: INFO/job_scheduler] Job 797 completing (errored=False, cancelled=False)
[2017-06-01 19:04:00,798: INFO/job_scheduler] Job 798 finished 1 steps successfully
[2017-06-01 19:04:00,815: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:04:00,838: INFO/job_scheduler] Job 798: complete_job: Updating cache
[2017-06-01 19:04:00,838: INFO/job_scheduler] Job 798 completing (errored=False, cancelled=False)
[2017-06-01 19:04:00,924: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:04:02,862: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:02,863: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:02,882: INFO/command_plan] add_jobs: created Job 799 (Test for host connectivity)
[2017-06-01 19:04:02,928: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:04:02,930: INFO/job_scheduler] Job 799: RunJobThread.run
[2017-06-01 19:04:02,958: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:02,958: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:02,982: INFO/command_plan] add_jobs: created Job 800 (Test for host connectivity)
[2017-06-01 19:04:03,002: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:04:03,007: INFO/job_scheduler] Job 800: RunJobThread.run
[2017-06-01 19:04:03,113: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:03,115: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:03,132: INFO/command_plan] add_jobs: created Job 801 (Test for host connectivity)
[2017-06-01 19:04:03,169: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:04:03,195: INFO/job_scheduler] Job 801: RunJobThread.run
[2017-06-01 19:04:03,251: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:03,272: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:03,323: INFO/command_plan] add_jobs: created Job 802 (Test for host connectivity)
[2017-06-01 19:04:03,375: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:04:03,403: INFO/job_scheduler] Job 802: RunJobThread.run
[2017-06-01 19:04:05,201: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:04:05,340: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:04:05,571: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:04:05,659: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:04:06,521: INFO/job_scheduler] Job 799 finished 1 steps successfully
[2017-06-01 19:04:06,589: INFO/job_scheduler] Job 799: complete_job: Updating cache
[2017-06-01 19:04:06,589: INFO/job_scheduler] Job 799 completing (errored=False, cancelled=False)
[2017-06-01 19:04:06,632: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:04:06,642: INFO/job_scheduler] Job 800 finished 1 steps successfully
[2017-06-01 19:04:06,667: INFO/job_scheduler] Job 800: complete_job: Updating cache
[2017-06-01 19:04:06,691: INFO/job_scheduler] Job 800 completing (errored=False, cancelled=False)
[2017-06-01 19:04:06,789: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:04:07,017: INFO/job_scheduler] Job 801 finished 1 steps successfully
[2017-06-01 19:04:07,064: INFO/job_scheduler] Job 801: complete_job: Updating cache
[2017-06-01 19:04:07,064: INFO/job_scheduler] Job 801 completing (errored=False, cancelled=False)
[2017-06-01 19:04:07,111: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:04:07,220: INFO/job_scheduler] Job 802 finished 1 steps successfully
[2017-06-01 19:04:07,322: INFO/job_scheduler] Job 802: complete_job: Updating cache
[2017-06-01 19:04:07,323: INFO/job_scheduler] Job 802 completing (errored=False, cancelled=False)
[2017-06-01 19:04:07,381: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:04:09,052: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:09,054: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:09,073: INFO/command_plan] add_jobs: created Job 803 (Test for host connectivity)
[2017-06-01 19:04:09,162: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:09,163: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:09,182: INFO/command_plan] add_jobs: created Job 804 (Test for host connectivity)
[2017-06-01 19:04:09,217: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:04:09,243: INFO/job_scheduler] Job 803: RunJobThread.run
[2017-06-01 19:04:09,244: INFO/job_scheduler] Job 804: RunJobThread.run
[2017-06-01 19:04:09,257: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:04:09,303: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:09,303: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:09,336: INFO/command_plan] add_jobs: created Job 805 (Test for host connectivity)
[2017-06-01 19:04:09,450: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:09,450: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:09,480: INFO/command_plan] add_jobs: created Job 806 (Test for host connectivity)
[2017-06-01 19:04:09,505: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:04:09,512: INFO/job_scheduler] Job 805: RunJobThread.run
[2017-06-01 19:04:09,535: INFO/job_scheduler] Job 806: RunJobThread.run
[2017-06-01 19:04:09,548: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:04:11,518: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:04:11,567: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:04:11,723: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:04:11,839: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:04:12,902: INFO/job_scheduler] Job 804 finished 1 steps successfully
[2017-06-01 19:04:12,927: INFO/job_scheduler] Job 803 finished 1 steps successfully
[2017-06-01 19:04:12,988: INFO/job_scheduler] Job 804: complete_job: Updating cache
[2017-06-01 19:04:12,988: INFO/job_scheduler] Job 804 completing (errored=False, cancelled=False)
[2017-06-01 19:04:13,041: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:04:13,075: INFO/job_scheduler] Job 803: complete_job: Updating cache
[2017-06-01 19:04:13,075: INFO/job_scheduler] Job 803 completing (errored=False, cancelled=False)
[2017-06-01 19:04:13,120: INFO/job_scheduler] Job 806 finished 1 steps successfully
[2017-06-01 19:04:13,194: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:04:13,237: INFO/job_scheduler] Job 806: complete_job: Updating cache
[2017-06-01 19:04:13,237: INFO/job_scheduler] Job 806 completing (errored=False, cancelled=False)
[2017-06-01 19:04:13,242: INFO/job_scheduler] Job 805 finished 1 steps successfully
[2017-06-01 19:04:13,355: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:04:13,384: INFO/job_scheduler] Job 805: complete_job: Updating cache
[2017-06-01 19:04:13,384: INFO/job_scheduler] Job 805 completing (errored=False, cancelled=False)
[2017-06-01 19:04:13,423: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:04:15,213: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:15,215: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:15,221: INFO/command_plan] add_jobs: created Job 807 (Test for host connectivity)
[2017-06-01 19:04:15,242: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:04:15,247: INFO/job_scheduler] Job 807: RunJobThread.run
[2017-06-01 19:04:15,335: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:15,336: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:15,372: INFO/command_plan] add_jobs: created Job 808 (Test for host connectivity)
[2017-06-01 19:04:15,418: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:04:15,473: INFO/job_scheduler] Job 808: RunJobThread.run
[2017-06-01 19:04:15,532: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:15,534: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:15,557: INFO/command_plan] add_jobs: created Job 809 (Test for host connectivity)
[2017-06-01 19:04:15,601: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:04:15,635: INFO/job_scheduler] Job 809: RunJobThread.run
[2017-06-01 19:04:15,692: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:15,692: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:15,723: INFO/command_plan] add_jobs: created Job 810 (Test for host connectivity)
[2017-06-01 19:04:15,794: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:04:15,799: INFO/job_scheduler] Job 810: RunJobThread.run
[2017-06-01 19:04:17,615: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:04:17,724: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:04:17,955: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:04:18,056: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:04:18,976: INFO/job_scheduler] Job 807 finished 1 steps successfully
[2017-06-01 19:04:19,030: INFO/job_scheduler] Job 807: complete_job: Updating cache
[2017-06-01 19:04:19,030: INFO/job_scheduler] Job 807 completing (errored=False, cancelled=False)
[2017-06-01 19:04:19,065: INFO/job_scheduler] Job 808 finished 1 steps successfully
[2017-06-01 19:04:19,123: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:04:19,163: INFO/job_scheduler] Job 808: complete_job: Updating cache
[2017-06-01 19:04:19,164: INFO/job_scheduler] Job 808 completing (errored=False, cancelled=False)
[2017-06-01 19:04:19,273: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:04:19,390: INFO/job_scheduler] Job 809 finished 1 steps successfully
[2017-06-01 19:04:19,430: INFO/job_scheduler] Job 809: complete_job: Updating cache
[2017-06-01 19:04:19,430: INFO/job_scheduler] Job 809 completing (errored=False, cancelled=False)
[2017-06-01 19:04:19,461: INFO/job_scheduler] Job 810 finished 1 steps successfully
[2017-06-01 19:04:19,474: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:04:19,510: INFO/job_scheduler] Job 810: complete_job: Updating cache
[2017-06-01 19:04:19,510: INFO/job_scheduler] Job 810 completing (errored=False, cancelled=False)
[2017-06-01 19:04:19,586: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:04:21,441: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:21,441: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:21,455: INFO/command_plan] add_jobs: created Job 811 (Test for host connectivity)
[2017-06-01 19:04:21,514: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:04:21,515: INFO/job_scheduler] Job 811: RunJobThread.run
[2017-06-01 19:04:21,654: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:21,655: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:21,683: INFO/command_plan] add_jobs: created Job 812 (Test for host connectivity)
[2017-06-01 19:04:21,716: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:04:21,717: INFO/job_scheduler] Job 812: RunJobThread.run
[2017-06-01 19:04:21,773: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:21,773: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:21,821: INFO/command_plan] add_jobs: created Job 813 (Test for host connectivity)
[2017-06-01 19:04:21,875: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:04:21,884: INFO/job_scheduler] Job 813: RunJobThread.run
[2017-06-01 19:04:21,958: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:21,958: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:21,980: INFO/command_plan] add_jobs: created Job 814 (Test for host connectivity)
[2017-06-01 19:04:22,034: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:04:22,055: INFO/job_scheduler] Job 814: RunJobThread.run
[2017-06-01 19:04:23,694: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:04:23,907: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:04:24,218: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:04:24,246: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:04:25,018: INFO/job_scheduler] Job 811 finished 1 steps successfully
[2017-06-01 19:04:25,054: INFO/job_scheduler] Job 811: complete_job: Updating cache
[2017-06-01 19:04:25,054: INFO/job_scheduler] Job 811 completing (errored=False, cancelled=False)
[2017-06-01 19:04:25,097: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:04:25,229: INFO/job_scheduler] Job 812 finished 1 steps successfully
[2017-06-01 19:04:25,262: INFO/job_scheduler] Job 812: complete_job: Updating cache
[2017-06-01 19:04:25,263: INFO/job_scheduler] Job 812 completing (errored=False, cancelled=False)
[2017-06-01 19:04:25,341: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:04:25,624: INFO/job_scheduler] Job 814 finished 1 steps successfully
[2017-06-01 19:04:25,679: INFO/job_scheduler] Job 814: complete_job: Updating cache
[2017-06-01 19:04:25,679: INFO/job_scheduler] Job 814 completing (errored=False, cancelled=False)
[2017-06-01 19:04:25,723: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:04:25,748: INFO/job_scheduler] Job 813 finished 1 steps successfully
[2017-06-01 19:04:25,762: INFO/job_scheduler] Job 813: complete_job: Updating cache
[2017-06-01 19:04:25,762: INFO/job_scheduler] Job 813 completing (errored=False, cancelled=False)
[2017-06-01 19:04:25,830: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:04:27,695: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:27,696: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:27,704: INFO/command_plan] add_jobs: created Job 815 (Test for host connectivity)
[2017-06-01 19:04:27,754: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:04:27,757: INFO/job_scheduler] Job 815: RunJobThread.run
[2017-06-01 19:04:27,793: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:27,794: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:27,842: INFO/command_plan] add_jobs: created Job 816 (Test for host connectivity)
[2017-06-01 19:04:27,870: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:04:27,880: INFO/job_scheduler] Job 816: RunJobThread.run
[2017-06-01 19:04:27,942: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:27,942: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:28,001: INFO/command_plan] add_jobs: created Job 817 (Test for host connectivity)
[2017-06-01 19:04:28,137: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:28,139: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:28,165: INFO/command_plan] add_jobs: created Job 818 (Test for host connectivity)
[2017-06-01 19:04:28,204: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:04:28,213: INFO/job_scheduler] Job 817: RunJobThread.run
[2017-06-01 19:04:28,224: INFO/job_scheduler] Job 818: RunJobThread.run
[2017-06-01 19:04:28,231: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:04:30,067: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:04:30,206: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:04:30,418: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:04:30,556: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:04:31,385: INFO/job_scheduler] Job 815 finished 1 steps successfully
[2017-06-01 19:04:31,440: INFO/job_scheduler] Job 815: complete_job: Updating cache
[2017-06-01 19:04:31,443: INFO/job_scheduler] Job 815 completing (errored=False, cancelled=False)
[2017-06-01 19:04:31,554: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:04:31,684: INFO/job_scheduler] Job 816 finished 1 steps successfully
[2017-06-01 19:04:31,694: INFO/job_scheduler] Job 816: complete_job: Updating cache
[2017-06-01 19:04:31,694: INFO/job_scheduler] Job 816 completing (errored=False, cancelled=False)
[2017-06-01 19:04:31,736: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:04:31,851: INFO/job_scheduler] Job 818 finished 1 steps successfully
[2017-06-01 19:04:31,886: INFO/job_scheduler] Job 818: complete_job: Updating cache
[2017-06-01 19:04:31,886: INFO/job_scheduler] Job 818 completing (errored=False, cancelled=False)
[2017-06-01 19:04:31,919: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:04:31,947: INFO/job_scheduler] Job 817 finished 1 steps successfully
[2017-06-01 19:04:32,002: INFO/job_scheduler] Job 817: complete_job: Updating cache
[2017-06-01 19:04:32,003: INFO/job_scheduler] Job 817 completing (errored=False, cancelled=False)
[2017-06-01 19:04:32,053: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:04:33,956: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:33,956: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:33,978: INFO/command_plan] add_jobs: created Job 819 (Test for host connectivity)
[2017-06-01 19:04:34,022: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:34,022: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:34,038: INFO/command_plan] add_jobs: created Job 820 (Test for host connectivity)
[2017-06-01 19:04:34,057: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:04:34,079: INFO/job_scheduler] Job 819: RunJobThread.run
[2017-06-01 19:04:34,106: INFO/job_scheduler] Job 820: RunJobThread.run
[2017-06-01 19:04:34,120: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:04:34,186: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:34,188: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:34,205: INFO/command_plan] add_jobs: created Job 821 (Test for host connectivity)
[2017-06-01 19:04:34,375: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:34,377: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:34,405: INFO/command_plan] add_jobs: created Job 822 (Test for host connectivity)
[2017-06-01 19:04:34,454: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:04:34,456: INFO/job_scheduler] Job 821: RunJobThread.run
[2017-06-01 19:04:34,457: INFO/job_scheduler] Job 822: RunJobThread.run
[2017-06-01 19:04:34,470: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:04:36,356: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:04:36,554: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:04:36,776: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:04:36,812: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:04:37,653: INFO/job_scheduler] Job 819 finished 1 steps successfully
[2017-06-01 19:04:37,710: INFO/job_scheduler] Job 819: complete_job: Updating cache
[2017-06-01 19:04:37,710: INFO/job_scheduler] Job 819 completing (errored=False, cancelled=False)
[2017-06-01 19:04:37,779: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:04:37,931: INFO/job_scheduler] Job 820 finished 1 steps successfully
[2017-06-01 19:04:37,997: INFO/job_scheduler] Job 820: complete_job: Updating cache
[2017-06-01 19:04:37,997: INFO/job_scheduler] Job 820 completing (errored=False, cancelled=False)
[2017-06-01 19:04:38,067: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:04:38,151: INFO/job_scheduler] Job 822 finished 1 steps successfully
[2017-06-01 19:04:38,168: INFO/job_scheduler] Job 822: complete_job: Updating cache
[2017-06-01 19:04:38,168: INFO/job_scheduler] Job 822 completing (errored=False, cancelled=False)
[2017-06-01 19:04:38,210: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:04:38,315: INFO/job_scheduler] Job 821 finished 1 steps successfully
[2017-06-01 19:04:38,376: INFO/job_scheduler] Job 821: complete_job: Updating cache
[2017-06-01 19:04:38,376: INFO/job_scheduler] Job 821 completing (errored=False, cancelled=False)
[2017-06-01 19:04:38,436: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:04:40,184: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:40,184: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:40,211: INFO/command_plan] add_jobs: created Job 823 (Test for host connectivity)
[2017-06-01 19:04:40,257: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:40,257: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:40,278: INFO/command_plan] add_jobs: created Job 824 (Test for host connectivity)
[2017-06-01 19:04:40,328: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:04:40,333: INFO/job_scheduler] Job 824: RunJobThread.run
[2017-06-01 19:04:40,333: INFO/job_scheduler] Job 823: RunJobThread.run
[2017-06-01 19:04:40,346: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:04:40,388: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:40,388: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:40,423: INFO/command_plan] add_jobs: created Job 825 (Test for host connectivity)
[2017-06-01 19:04:40,458: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:04:40,492: INFO/job_scheduler] Job 825: RunJobThread.run
[2017-06-01 19:04:40,578: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:40,580: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:40,625: INFO/command_plan] add_jobs: created Job 826 (Test for host connectivity)
[2017-06-01 19:04:40,672: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:04:40,732: INFO/job_scheduler] Job 826: RunJobThread.run
[2017-06-01 19:04:42,619: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:04:42,670: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:04:42,826: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:04:42,971: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:04:44,022: INFO/job_scheduler] Job 824 finished 1 steps successfully
[2017-06-01 19:04:44,038: INFO/job_scheduler] Job 823 finished 1 steps successfully
[2017-06-01 19:04:44,092: INFO/job_scheduler] Job 824: complete_job: Updating cache
[2017-06-01 19:04:44,093: INFO/job_scheduler] Job 824 completing (errored=False, cancelled=False)
[2017-06-01 19:04:44,142: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:04:44,175: INFO/job_scheduler] Job 823: complete_job: Updating cache
[2017-06-01 19:04:44,175: INFO/job_scheduler] Job 823 completing (errored=False, cancelled=False)
[2017-06-01 19:04:44,230: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:04:44,247: INFO/job_scheduler] Job 825 finished 1 steps successfully
[2017-06-01 19:04:44,285: INFO/job_scheduler] Job 826 finished 1 steps successfully
[2017-06-01 19:04:44,323: INFO/job_scheduler] Job 825: complete_job: Updating cache
[2017-06-01 19:04:44,332: INFO/job_scheduler] Job 825 completing (errored=False, cancelled=False)
[2017-06-01 19:04:44,383: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:04:44,402: INFO/job_scheduler] Job 826: complete_job: Updating cache
[2017-06-01 19:04:44,402: INFO/job_scheduler] Job 826 completing (errored=False, cancelled=False)
[2017-06-01 19:04:44,466: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:04:46,319: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:46,320: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:46,343: INFO/command_plan] add_jobs: created Job 827 (Test for host connectivity)
[2017-06-01 19:04:46,406: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:04:46,411: INFO/job_scheduler] Job 827: RunJobThread.run
[2017-06-01 19:04:46,436: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:46,437: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:46,466: INFO/command_plan] add_jobs: created Job 828 (Test for host connectivity)
[2017-06-01 19:04:46,528: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:04:46,540: INFO/job_scheduler] Job 828: RunJobThread.run
[2017-06-01 19:04:46,581: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:46,587: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:46,627: INFO/command_plan] add_jobs: created Job 829 (Test for host connectivity)
[2017-06-01 19:04:46,666: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:04:46,705: INFO/job_scheduler] Job 829: RunJobThread.run
[2017-06-01 19:04:46,773: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:46,778: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:46,812: INFO/command_plan] add_jobs: created Job 830 (Test for host connectivity)
[2017-06-01 19:04:46,878: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:04:46,885: INFO/job_scheduler] Job 830: RunJobThread.run
[2017-06-01 19:04:48,793: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:04:48,835: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:04:48,958: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:04:49,092: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:04:50,143: INFO/job_scheduler] Job 827 finished 1 steps successfully
[2017-06-01 19:04:50,199: INFO/job_scheduler] Job 827: complete_job: Updating cache
[2017-06-01 19:04:50,199: INFO/job_scheduler] Job 827 completing (errored=False, cancelled=False)
[2017-06-01 19:04:50,247: INFO/job_scheduler] Job 829 finished 1 steps successfully
[2017-06-01 19:04:50,257: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:04:50,272: INFO/job_scheduler] Job 828 finished 1 steps successfully
[2017-06-01 19:04:50,335: INFO/job_scheduler] Job 829: complete_job: Updating cache
[2017-06-01 19:04:50,335: INFO/job_scheduler] Job 829 completing (errored=False, cancelled=False)
[2017-06-01 19:04:50,412: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:04:50,464: INFO/job_scheduler] Job 830 finished 1 steps successfully
[2017-06-01 19:04:50,480: INFO/job_scheduler] Job 828: complete_job: Updating cache
[2017-06-01 19:04:50,480: INFO/job_scheduler] Job 828 completing (errored=False, cancelled=False)
[2017-06-01 19:04:50,540: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:04:50,565: INFO/job_scheduler] Job 830: complete_job: Updating cache
[2017-06-01 19:04:50,566: INFO/job_scheduler] Job 830 completing (errored=False, cancelled=False)
[2017-06-01 19:04:50,616: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:04:52,531: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:52,531: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:52,540: INFO/command_plan] add_jobs: created Job 831 (Test for host connectivity)
[2017-06-01 19:04:52,594: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:52,594: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:52,682: INFO/command_plan] add_jobs: created Job 832 (Test for host connectivity)
[2017-06-01 19:04:52,708: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:04:52,721: INFO/job_scheduler] Job 832: RunJobThread.run
[2017-06-01 19:04:52,722: INFO/job_scheduler] Job 831: RunJobThread.run
[2017-06-01 19:04:52,733: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:04:52,814: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:52,816: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:52,842: INFO/command_plan] add_jobs: created Job 833 (Test for host connectivity)
[2017-06-01 19:04:52,941: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:04:52,964: INFO/job_scheduler] Job 833: RunJobThread.run
[2017-06-01 19:04:53,026: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:53,034: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:53,055: INFO/command_plan] add_jobs: created Job 834 (Test for host connectivity)
[2017-06-01 19:04:53,139: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:04:53,149: INFO/job_scheduler] Job 834: RunJobThread.run
[2017-06-01 19:04:55,077: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:04:55,264: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:04:55,308: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:04:55,349: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:04:56,422: INFO/job_scheduler] Job 831 finished 1 steps successfully
[2017-06-01 19:04:56,480: INFO/job_scheduler] Job 831: complete_job: Updating cache
[2017-06-01 19:04:56,480: INFO/job_scheduler] Job 831 completing (errored=False, cancelled=False)
[2017-06-01 19:04:56,533: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:04:56,587: INFO/job_scheduler] Job 832 finished 1 steps successfully
[2017-06-01 19:04:56,621: INFO/job_scheduler] Job 832: complete_job: Updating cache
[2017-06-01 19:04:56,621: INFO/job_scheduler] Job 832 completing (errored=False, cancelled=False)
[2017-06-01 19:04:56,667: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:04:56,785: INFO/job_scheduler] Job 833 finished 1 steps successfully
[2017-06-01 19:04:56,813: INFO/job_scheduler] Job 833: complete_job: Updating cache
[2017-06-01 19:04:56,813: INFO/job_scheduler] Job 833 completing (errored=False, cancelled=False)
[2017-06-01 19:04:56,815: INFO/job_scheduler] Job 834 finished 1 steps successfully
[2017-06-01 19:04:56,871: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:04:56,904: INFO/job_scheduler] Job 834: complete_job: Updating cache
[2017-06-01 19:04:56,917: INFO/job_scheduler] Job 834 completing (errored=False, cancelled=False)
[2017-06-01 19:04:56,962: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:04:58,812: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:58,813: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:58,822: INFO/command_plan] add_jobs: created Job 835 (Test for host connectivity)
[2017-06-01 19:04:58,893: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:58,893: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:58,921: INFO/command_plan] add_jobs: created Job 836 (Test for host connectivity)
[2017-06-01 19:04:58,932: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:04:58,945: INFO/job_scheduler] Job 835: RunJobThread.run
[2017-06-01 19:04:58,998: INFO/job_scheduler] Job 836: RunJobThread.run
[2017-06-01 19:04:59,034: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:04:59,126: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:59,126: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:59,174: INFO/command_plan] add_jobs: created Job 837 (Test for host connectivity)
[2017-06-01 19:04:59,227: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:04:59,236: INFO/job_scheduler] Job 837: RunJobThread.run
[2017-06-01 19:04:59,320: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:04:59,320: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:04:59,357: INFO/command_plan] add_jobs: created Job 838 (Test for host connectivity)
[2017-06-01 19:04:59,410: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:04:59,415: INFO/job_scheduler] Job 838: RunJobThread.run
[2017-06-01 19:05:01,246: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:05:01,310: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:05:01,457: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:05:01,647: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:05:02,595: INFO/job_scheduler] Job 835 finished 1 steps successfully
[2017-06-01 19:05:02,679: INFO/job_scheduler] Job 835: complete_job: Updating cache
[2017-06-01 19:05:02,680: INFO/job_scheduler] Job 835 completing (errored=False, cancelled=False)
[2017-06-01 19:05:02,724: INFO/job_scheduler] Job 836 finished 1 steps successfully
[2017-06-01 19:05:02,754: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:05:02,763: INFO/job_scheduler] Job 836: complete_job: Updating cache
[2017-06-01 19:05:02,763: INFO/job_scheduler] Job 836 completing (errored=False, cancelled=False)
[2017-06-01 19:05:02,887: INFO/job_scheduler] Job 837 finished 1 steps successfully
[2017-06-01 19:05:02,987: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:05:03,004: INFO/job_scheduler] Job 837: complete_job: Updating cache
[2017-06-01 19:05:03,004: INFO/job_scheduler] Job 837 completing (errored=False, cancelled=False)
[2017-06-01 19:05:03,074: INFO/job_scheduler] Job 838 finished 1 steps successfully
[2017-06-01 19:05:03,106: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:05:03,130: INFO/job_scheduler] Job 838: complete_job: Updating cache
[2017-06-01 19:05:03,130: INFO/job_scheduler] Job 838 completing (errored=False, cancelled=False)
[2017-06-01 19:05:03,213: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:05:04,578: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:04,580: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:04,587: INFO/command_plan] add_jobs: created Job 839 (Test for host connectivity)
[2017-06-01 19:05:04,633: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:05:04,654: INFO/job_scheduler] Job 839: RunJobThread.run
[2017-06-01 19:05:04,688: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:04,690: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:04,705: INFO/command_plan] add_jobs: created Job 840 (Test for host connectivity)
[2017-06-01 19:05:04,822: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:04,822: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:04,829: INFO/command_plan] add_jobs: created Job 841 (Test for host connectivity)
[2017-06-01 19:05:04,877: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:05:04,886: INFO/job_scheduler] Job 840: RunJobThread.run
[2017-06-01 19:05:04,894: INFO/job_scheduler] Job 841: RunJobThread.run
[2017-06-01 19:05:04,905: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:05:04,969: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:04,969: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:05,030: INFO/command_plan] add_jobs: created Job 842 (Test for host connectivity)
[2017-06-01 19:05:05,070: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:05:05,089: INFO/job_scheduler] Job 842: RunJobThread.run
[2017-06-01 19:05:06,973: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:05:07,107: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:05:07,168: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:05:07,303: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:05:08,352: INFO/job_scheduler] Job 839 finished 1 steps successfully
[2017-06-01 19:05:08,420: INFO/job_scheduler] Job 839: complete_job: Updating cache
[2017-06-01 19:05:08,421: INFO/job_scheduler] Job 839 completing (errored=False, cancelled=False)
[2017-06-01 19:05:08,458: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:05:08,516: INFO/job_scheduler] Job 840 finished 1 steps successfully
[2017-06-01 19:05:08,540: INFO/job_scheduler] Job 840: complete_job: Updating cache
[2017-06-01 19:05:08,541: INFO/job_scheduler] Job 840 completing (errored=False, cancelled=False)
[2017-06-01 19:05:08,647: INFO/job_scheduler] Job 841 finished 1 steps successfully
[2017-06-01 19:05:08,660: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:05:08,667: INFO/job_scheduler] Job 842 finished 1 steps successfully
[2017-06-01 19:05:08,713: INFO/job_scheduler] Job 841: complete_job: Updating cache
[2017-06-01 19:05:08,713: INFO/job_scheduler] Job 841 completing (errored=False, cancelled=False)
[2017-06-01 19:05:08,764: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:05:08,794: INFO/job_scheduler] Job 842: complete_job: Updating cache
[2017-06-01 19:05:08,794: INFO/job_scheduler] Job 842 completing (errored=False, cancelled=False)
[2017-06-01 19:05:08,889: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:05:10,801: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:10,804: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:10,812: INFO/command_plan] add_jobs: created Job 843 (Test for host connectivity)
[2017-06-01 19:05:10,859: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:05:10,876: INFO/job_scheduler] Job 843: RunJobThread.run
[2017-06-01 19:05:10,942: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:10,942: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:10,963: INFO/command_plan] add_jobs: created Job 844 (Test for host connectivity)
[2017-06-01 19:05:11,012: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:05:11,044: INFO/job_scheduler] Job 844: RunJobThread.run
[2017-06-01 19:05:11,098: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:11,105: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:11,140: INFO/command_plan] add_jobs: created Job 845 (Test for host connectivity)
[2017-06-01 19:05:11,177: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:05:11,179: INFO/job_scheduler] Job 845: RunJobThread.run
[2017-06-01 19:05:11,250: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:11,250: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:11,370: INFO/command_plan] add_jobs: created Job 846 (Test for host connectivity)
[2017-06-01 19:05:11,467: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:05:11,474: INFO/job_scheduler] Job 846: RunJobThread.run
[2017-06-01 19:05:13,067: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:05:13,377: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:05:13,526: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:05:13,639: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:05:14,388: INFO/job_scheduler] Job 843 finished 1 steps successfully
[2017-06-01 19:05:14,437: INFO/job_scheduler] Job 843: complete_job: Updating cache
[2017-06-01 19:05:14,437: INFO/job_scheduler] Job 843 completing (errored=False, cancelled=False)
[2017-06-01 19:05:14,496: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:05:14,730: INFO/job_scheduler] Job 844 finished 1 steps successfully
[2017-06-01 19:05:14,778: INFO/job_scheduler] Job 844: complete_job: Updating cache
[2017-06-01 19:05:14,778: INFO/job_scheduler] Job 844 completing (errored=False, cancelled=False)
[2017-06-01 19:05:14,821: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:05:14,906: INFO/job_scheduler] Job 845 finished 1 steps successfully
[2017-06-01 19:05:14,961: INFO/job_scheduler] Job 845: complete_job: Updating cache
[2017-06-01 19:05:14,962: INFO/job_scheduler] Job 845 completing (errored=False, cancelled=False)
[2017-06-01 19:05:14,995: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:05:15,020: INFO/job_scheduler] Job 846 finished 1 steps successfully
[2017-06-01 19:05:15,043: INFO/job_scheduler] Job 846: complete_job: Updating cache
[2017-06-01 19:05:15,043: INFO/job_scheduler] Job 846 completing (errored=False, cancelled=False)
[2017-06-01 19:05:15,104: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:05:17,109: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:17,110: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:17,120: INFO/command_plan] add_jobs: created Job 847 (Test for host connectivity)
[2017-06-01 19:05:17,155: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:05:17,175: INFO/job_scheduler] Job 847: RunJobThread.run
[2017-06-01 19:05:17,187: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:17,188: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:17,212: INFO/command_plan] add_jobs: created Job 848 (Test for host connectivity)
[2017-06-01 19:05:17,242: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:05:17,288: INFO/job_scheduler] Job 848: RunJobThread.run
[2017-06-01 19:05:17,334: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:17,335: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:17,359: INFO/command_plan] add_jobs: created Job 849 (Test for host connectivity)
[2017-06-01 19:05:17,413: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:05:17,456: INFO/job_scheduler] Job 849: RunJobThread.run
[2017-06-01 19:05:17,541: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:17,542: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:17,563: INFO/command_plan] add_jobs: created Job 850 (Test for host connectivity)
[2017-06-01 19:05:17,649: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:05:17,663: INFO/job_scheduler] Job 850: RunJobThread.run
[2017-06-01 19:05:19,538: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:05:19,674: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:05:19,723: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:05:19,942: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:05:20,846: INFO/job_scheduler] Job 847 finished 1 steps successfully
[2017-06-01 19:05:20,877: INFO/job_scheduler] Job 847: complete_job: Updating cache
[2017-06-01 19:05:20,877: INFO/job_scheduler] Job 847 completing (errored=False, cancelled=False)
[2017-06-01 19:05:20,969: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:05:21,004: INFO/job_scheduler] Job 848 finished 1 steps successfully
[2017-06-01 19:05:21,047: INFO/job_scheduler] Job 849 finished 1 steps successfully
[2017-06-01 19:05:21,052: INFO/job_scheduler] Job 848: complete_job: Updating cache
[2017-06-01 19:05:21,052: INFO/job_scheduler] Job 848 completing (errored=False, cancelled=False)
[2017-06-01 19:05:21,171: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:05:21,212: INFO/job_scheduler] Job 849: complete_job: Updating cache
[2017-06-01 19:05:21,221: INFO/job_scheduler] Job 849 completing (errored=False, cancelled=False)
[2017-06-01 19:05:21,271: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:05:21,412: INFO/job_scheduler] Job 850 finished 1 steps successfully
[2017-06-01 19:05:21,435: INFO/job_scheduler] Job 850: complete_job: Updating cache
[2017-06-01 19:05:21,436: INFO/job_scheduler] Job 850 completing (errored=False, cancelled=False)
[2017-06-01 19:05:21,477: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:05:23,366: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:23,366: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:23,379: INFO/command_plan] add_jobs: created Job 851 (Test for host connectivity)
[2017-06-01 19:05:23,467: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:23,469: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:23,477: INFO/command_plan] add_jobs: created Job 852 (Test for host connectivity)
[2017-06-01 19:05:23,495: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:05:23,540: INFO/job_scheduler] Job 851: RunJobThread.run
[2017-06-01 19:05:23,554: INFO/job_scheduler] Job 852: RunJobThread.run
[2017-06-01 19:05:23,612: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:23,612: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:23,650: INFO/command_plan] add_jobs: created Job 853 (Test for host connectivity)
[2017-06-01 19:05:23,689: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:05:23,752: INFO/job_scheduler] Job 853: RunJobThread.run
[2017-06-01 19:05:23,799: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:23,801: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:23,835: INFO/command_plan] add_jobs: created Job 854 (Test for host connectivity)
[2017-06-01 19:05:23,863: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:05:23,889: INFO/job_scheduler] Job 854: RunJobThread.run
[2017-06-01 19:05:23,915: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:05:25,837: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:05:25,984: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:05:26,096: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:05:26,212: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:05:27,119: INFO/job_scheduler] Job 851 finished 1 steps successfully
[2017-06-01 19:05:27,176: INFO/job_scheduler] Job 851: complete_job: Updating cache
[2017-06-01 19:05:27,176: INFO/job_scheduler] Job 851 completing (errored=False, cancelled=False)
[2017-06-01 19:05:27,259: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:05:27,472: INFO/job_scheduler] Job 853 finished 1 steps successfully
[2017-06-01 19:05:27,482: INFO/job_scheduler] Job 852 finished 1 steps successfully
[2017-06-01 19:05:27,501: INFO/job_scheduler] Job 853: complete_job: Updating cache
[2017-06-01 19:05:27,501: INFO/job_scheduler] Job 853 completing (errored=False, cancelled=False)
[2017-06-01 19:05:27,592: INFO/job_scheduler] Job 854 finished 1 steps successfully
[2017-06-01 19:05:27,626: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:05:27,634: INFO/job_scheduler] Job 852: complete_job: Updating cache
[2017-06-01 19:05:27,635: INFO/job_scheduler] Job 852 completing (errored=False, cancelled=False)
[2017-06-01 19:05:27,685: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:05:27,735: INFO/job_scheduler] Job 854: complete_job: Updating cache
[2017-06-01 19:05:27,735: INFO/job_scheduler] Job 854 completing (errored=False, cancelled=False)
[2017-06-01 19:05:27,798: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:05:29,668: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:29,668: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:29,685: INFO/command_plan] add_jobs: created Job 855 (Test for host connectivity)
[2017-06-01 19:05:29,777: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:05:29,778: INFO/job_scheduler] Job 855: RunJobThread.run
[2017-06-01 19:05:29,814: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:29,814: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:29,854: INFO/command_plan] add_jobs: created Job 856 (Test for host connectivity)
[2017-06-01 19:05:29,947: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:05:29,959: INFO/job_scheduler] Job 856: RunJobThread.run
[2017-06-01 19:05:29,991: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:29,992: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:30,001: INFO/command_plan] add_jobs: created Job 857 (Test for host connectivity)
[2017-06-01 19:05:30,071: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:05:30,117: INFO/job_scheduler] Job 857: RunJobThread.run
[2017-06-01 19:05:30,211: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:30,212: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:30,240: INFO/command_plan] add_jobs: created Job 858 (Test for host connectivity)
[2017-06-01 19:05:30,333: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:05:30,353: INFO/job_scheduler] Job 858: RunJobThread.run
[2017-06-01 19:05:31,978: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:05:32,254: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:05:32,553: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:05:32,580: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:05:33,318: INFO/job_scheduler] Job 855 finished 1 steps successfully
[2017-06-01 19:05:33,350: INFO/job_scheduler] Job 855: complete_job: Updating cache
[2017-06-01 19:05:33,350: INFO/job_scheduler] Job 855 completing (errored=False, cancelled=False)
[2017-06-01 19:05:33,416: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:05:33,584: INFO/job_scheduler] Job 856 finished 1 steps successfully
[2017-06-01 19:05:33,635: INFO/job_scheduler] Job 856: complete_job: Updating cache
[2017-06-01 19:05:33,635: INFO/job_scheduler] Job 856 completing (errored=False, cancelled=False)
[2017-06-01 19:05:33,666: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:05:33,816: INFO/job_scheduler] Job 857 finished 1 steps successfully
[2017-06-01 19:05:33,874: INFO/job_scheduler] Job 857: complete_job: Updating cache
[2017-06-01 19:05:33,875: INFO/job_scheduler] Job 857 completing (errored=False, cancelled=False)
[2017-06-01 19:05:33,909: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:05:34,083: INFO/job_scheduler] Job 858 finished 1 steps successfully
[2017-06-01 19:05:34,107: INFO/job_scheduler] Job 858: complete_job: Updating cache
[2017-06-01 19:05:34,108: INFO/job_scheduler] Job 858 completing (errored=False, cancelled=False)
[2017-06-01 19:05:34,158: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:05:35,975: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:35,976: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:35,991: INFO/command_plan] add_jobs: created Job 859 (Test for host connectivity)
[2017-06-01 19:05:36,029: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:05:36,036: INFO/job_scheduler] Job 859: RunJobThread.run
[2017-06-01 19:05:36,076: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:36,076: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:36,232: INFO/command_plan] add_jobs: created Job 860 (Test for host connectivity)
[2017-06-01 19:05:36,294: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:05:36,298: INFO/job_scheduler] Job 860: RunJobThread.run
[2017-06-01 19:05:36,335: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:36,335: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:36,366: INFO/command_plan] add_jobs: created Job 861 (Test for host connectivity)
[2017-06-01 19:05:36,460: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:05:36,474: INFO/job_scheduler] Job 861: RunJobThread.run
[2017-06-01 19:05:36,519: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:36,520: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:36,542: INFO/command_plan] add_jobs: created Job 862 (Test for host connectivity)
[2017-06-01 19:05:36,634: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:05:36,671: INFO/job_scheduler] Job 862: RunJobThread.run
[2017-06-01 19:05:38,379: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:05:38,575: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:05:38,732: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:05:38,848: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:05:39,734: INFO/job_scheduler] Job 859 finished 1 steps successfully
[2017-06-01 19:05:39,756: INFO/job_scheduler] Job 859: complete_job: Updating cache
[2017-06-01 19:05:39,757: INFO/job_scheduler] Job 859 completing (errored=False, cancelled=False)
[2017-06-01 19:05:39,849: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:05:39,880: INFO/job_scheduler] Job 860 finished 1 steps successfully
[2017-06-01 19:05:39,933: INFO/job_scheduler] Job 860: complete_job: Updating cache
[2017-06-01 19:05:39,933: INFO/job_scheduler] Job 860 completing (errored=False, cancelled=False)
[2017-06-01 19:05:40,045: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:05:40,144: INFO/job_scheduler] Job 861 finished 1 steps successfully
[2017-06-01 19:05:40,191: INFO/job_scheduler] Job 861: complete_job: Updating cache
[2017-06-01 19:05:40,192: INFO/job_scheduler] Job 861 completing (errored=False, cancelled=False)
[2017-06-01 19:05:40,231: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:05:40,275: INFO/job_scheduler] Job 862 finished 1 steps successfully
[2017-06-01 19:05:40,315: INFO/job_scheduler] Job 862: complete_job: Updating cache
[2017-06-01 19:05:40,315: INFO/job_scheduler] Job 862 completing (errored=False, cancelled=False)
[2017-06-01 19:05:40,365: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:05:42,267: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:42,267: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:42,289: INFO/command_plan] add_jobs: created Job 863 (Test for host connectivity)
[2017-06-01 19:05:42,336: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:05:42,351: INFO/job_scheduler] Job 863: RunJobThread.run
[2017-06-01 19:05:42,408: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:42,408: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:42,422: INFO/command_plan] add_jobs: created Job 864 (Test for host connectivity)
[2017-06-01 19:05:42,520: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:05:42,538: INFO/job_scheduler] Job 864: RunJobThread.run
[2017-06-01 19:05:42,593: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:42,599: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:42,630: INFO/command_plan] add_jobs: created Job 865 (Test for host connectivity)
[2017-06-01 19:05:42,700: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:05:42,707: INFO/job_scheduler] Job 865: RunJobThread.run
[2017-06-01 19:05:42,743: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:42,743: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:42,788: INFO/command_plan] add_jobs: created Job 866 (Test for host connectivity)
[2017-06-01 19:05:42,857: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:05:42,889: INFO/job_scheduler] Job 866: RunJobThread.run
[2017-06-01 19:05:44,633: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:05:44,795: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:05:45,047: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:05:45,075: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:05:46,110: INFO/job_scheduler] Job 863 finished 1 steps successfully
[2017-06-01 19:05:46,166: INFO/job_scheduler] Job 863: complete_job: Updating cache
[2017-06-01 19:05:46,167: INFO/job_scheduler] Job 863 completing (errored=False, cancelled=False)
[2017-06-01 19:05:46,208: INFO/job_scheduler] Job 864 finished 1 steps successfully
[2017-06-01 19:05:46,214: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:05:46,254: INFO/job_scheduler] Job 864: complete_job: Updating cache
[2017-06-01 19:05:46,255: INFO/job_scheduler] Job 864 completing (errored=False, cancelled=False)
[2017-06-01 19:05:46,353: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:05:46,373: INFO/job_scheduler] Job 866 finished 1 steps successfully
[2017-06-01 19:05:46,470: INFO/job_scheduler] Job 866: complete_job: Updating cache
[2017-06-01 19:05:46,473: INFO/job_scheduler] Job 866 completing (errored=False, cancelled=False)
[2017-06-01 19:05:46,479: INFO/job_scheduler] Job 865 finished 1 steps successfully
[2017-06-01 19:05:46,521: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:05:46,580: INFO/job_scheduler] Job 865: complete_job: Updating cache
[2017-06-01 19:05:46,580: INFO/job_scheduler] Job 865 completing (errored=False, cancelled=False)
[2017-06-01 19:05:46,647: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:05:48,536: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:48,536: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:48,570: INFO/command_plan] add_jobs: created Job 867 (Test for host connectivity)
[2017-06-01 19:05:48,634: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:48,635: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:48,648: INFO/command_plan] add_jobs: created Job 868 (Test for host connectivity)
[2017-06-01 19:05:48,770: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:05:48,774: INFO/job_scheduler] Job 867: RunJobThread.run
[2017-06-01 19:05:48,775: INFO/job_scheduler] Job 868: RunJobThread.run
[2017-06-01 19:05:48,780: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:05:48,813: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:48,822: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:48,838: INFO/command_plan] add_jobs: created Job 869 (Test for host connectivity)
[2017-06-01 19:05:48,889: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:05:48,895: INFO/job_scheduler] Job 869: RunJobThread.run
[2017-06-01 19:05:48,994: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:48,996: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:49,034: INFO/command_plan] add_jobs: created Job 870 (Test for host connectivity)
[2017-06-01 19:05:49,108: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:05:49,110: INFO/job_scheduler] Job 870: RunJobThread.run
[2017-06-01 19:05:50,988: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:05:51,137: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:05:51,264: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:05:51,290: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:05:52,320: INFO/job_scheduler] Job 867 finished 1 steps successfully
[2017-06-01 19:05:52,369: INFO/job_scheduler] Job 867: complete_job: Updating cache
[2017-06-01 19:05:52,369: INFO/job_scheduler] Job 867 completing (errored=False, cancelled=False)
[2017-06-01 19:05:52,460: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:05:52,476: INFO/job_scheduler] Job 868 finished 1 steps successfully
[2017-06-01 19:05:52,503: INFO/job_scheduler] Job 868: complete_job: Updating cache
[2017-06-01 19:05:52,503: INFO/job_scheduler] Job 868 completing (errored=False, cancelled=False)
[2017-06-01 19:05:52,589: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:05:52,661: INFO/job_scheduler] Job 869 finished 1 steps successfully
[2017-06-01 19:05:52,704: INFO/job_scheduler] Job 869: complete_job: Updating cache
[2017-06-01 19:05:52,704: INFO/job_scheduler] Job 869 completing (errored=False, cancelled=False)
[2017-06-01 19:05:52,754: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:05:52,803: INFO/job_scheduler] Job 870 finished 1 steps successfully
[2017-06-01 19:05:52,854: INFO/job_scheduler] Job 870: complete_job: Updating cache
[2017-06-01 19:05:52,854: INFO/job_scheduler] Job 870 completing (errored=False, cancelled=False)
[2017-06-01 19:05:52,887: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:05:54,977: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:54,978: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:55,020: INFO/command_plan] add_jobs: created Job 871 (Test for host connectivity)
[2017-06-01 19:05:55,060: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:05:55,064: INFO/job_scheduler] Job 871: RunJobThread.run
[2017-06-01 19:05:55,107: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:55,107: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:55,126: INFO/command_plan] add_jobs: created Job 872 (Test for host connectivity)
[2017-06-01 19:05:55,273: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:55,274: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:55,301: INFO/command_plan] add_jobs: created Job 873 (Test for host connectivity)
[2017-06-01 19:05:55,314: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:05:55,341: INFO/job_scheduler] Job 872: RunJobThread.run
[2017-06-01 19:05:55,342: INFO/job_scheduler] Job 873: RunJobThread.run
[2017-06-01 19:05:55,388: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:05:55,395: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:05:55,419: INFO/command_plan] add_jobs: created Job 874 (Test for host connectivity)
[2017-06-01 19:05:55,437: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:05:55,484: INFO/job_scheduler] Job 874: RunJobThread.run
[2017-06-01 19:05:55,544: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:05:57,335: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:05:57,527: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:05:57,707: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:05:57,775: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:05:58,657: INFO/job_scheduler] Job 871 finished 1 steps successfully
[2017-06-01 19:05:58,694: INFO/job_scheduler] Job 871: complete_job: Updating cache
[2017-06-01 19:05:58,694: INFO/job_scheduler] Job 871 completing (errored=False, cancelled=False)
[2017-06-01 19:05:58,737: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:05:58,849: INFO/job_scheduler] Job 872 finished 1 steps successfully
[2017-06-01 19:05:58,878: INFO/job_scheduler] Job 872: complete_job: Updating cache
[2017-06-01 19:05:58,878: INFO/job_scheduler] Job 872 completing (errored=False, cancelled=False)
[2017-06-01 19:05:58,935: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:05:59,123: INFO/job_scheduler] Job 874 finished 1 steps successfully
[2017-06-01 19:05:59,177: INFO/job_scheduler] Job 874: complete_job: Updating cache
[2017-06-01 19:05:59,177: INFO/job_scheduler] Job 874 completing (errored=False, cancelled=False)
[2017-06-01 19:05:59,220: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:05:59,292: INFO/job_scheduler] Job 873 finished 1 steps successfully
[2017-06-01 19:05:59,312: INFO/job_scheduler] Job 873: complete_job: Updating cache
[2017-06-01 19:05:59,312: INFO/job_scheduler] Job 873 completing (errored=False, cancelled=False)
[2017-06-01 19:05:59,369: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:06:01,152: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:01,152: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:01,203: INFO/command_plan] add_jobs: created Job 875 (Test for host connectivity)
[2017-06-01 19:06:01,258: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:01,258: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:01,303: INFO/command_plan] add_jobs: created Job 876 (Test for host connectivity)
[2017-06-01 19:06:01,322: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:06:01,326: INFO/job_scheduler] Job 875: RunJobThread.run
[2017-06-01 19:06:01,326: INFO/job_scheduler] Job 876: RunJobThread.run
[2017-06-01 19:06:01,338: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:06:01,441: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:01,444: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:01,456: INFO/command_plan] add_jobs: created Job 877 (Test for host connectivity)
[2017-06-01 19:06:01,513: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:06:01,530: INFO/job_scheduler] Job 877: RunJobThread.run
[2017-06-01 19:06:01,593: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:01,593: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:01,618: INFO/command_plan] add_jobs: created Job 878 (Test for host connectivity)
[2017-06-01 19:06:01,682: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:06:01,699: INFO/job_scheduler] Job 878: RunJobThread.run
[2017-06-01 19:06:03,527: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:06:03,647: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:06:03,761: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:06:04,050: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:06:04,881: INFO/job_scheduler] Job 875 finished 1 steps successfully
[2017-06-01 19:06:04,920: INFO/job_scheduler] Job 875: complete_job: Updating cache
[2017-06-01 19:06:04,920: INFO/job_scheduler] Job 875 completing (errored=False, cancelled=False)
[2017-06-01 19:06:04,980: INFO/job_scheduler] Job 876 finished 1 steps successfully
[2017-06-01 19:06:04,980: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:06:05,005: INFO/job_scheduler] Job 876: complete_job: Updating cache
[2017-06-01 19:06:05,015: INFO/job_scheduler] Job 876 completing (errored=False, cancelled=False)
[2017-06-01 19:06:05,137: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:06:05,214: INFO/job_scheduler] Job 877 finished 1 steps successfully
[2017-06-01 19:06:05,253: INFO/job_scheduler] Job 877: complete_job: Updating cache
[2017-06-01 19:06:05,253: INFO/job_scheduler] Job 877 completing (errored=False, cancelled=False)
[2017-06-01 19:06:05,342: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:06:05,498: INFO/job_scheduler] Job 878 finished 1 steps successfully
[2017-06-01 19:06:05,561: INFO/job_scheduler] Job 878: complete_job: Updating cache
[2017-06-01 19:06:05,562: INFO/job_scheduler] Job 878 completing (errored=False, cancelled=False)
[2017-06-01 19:06:05,612: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:06:07,335: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:07,335: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:07,355: INFO/command_plan] add_jobs: created Job 879 (Test for host connectivity)
[2017-06-01 19:06:07,381: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:06:07,393: INFO/job_scheduler] Job 879: RunJobThread.run
[2017-06-01 19:06:07,482: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:07,483: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:07,503: INFO/command_plan] add_jobs: created Job 880 (Test for host connectivity)
[2017-06-01 19:06:07,573: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:06:07,600: INFO/job_scheduler] Job 880: RunJobThread.run
[2017-06-01 19:06:07,649: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:07,649: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:07,661: INFO/command_plan] add_jobs: created Job 881 (Test for host connectivity)
[2017-06-01 19:06:07,673: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:06:07,706: INFO/job_scheduler] Job 881: RunJobThread.run
[2017-06-01 19:06:07,791: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:07,791: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:07,812: INFO/command_plan] add_jobs: created Job 882 (Test for host connectivity)
[2017-06-01 19:06:07,851: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:06:07,882: INFO/job_scheduler] Job 882: RunJobThread.run
[2017-06-01 19:06:09,738: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:06:09,840: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:06:10,172: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:06:10,182: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:06:11,059: INFO/job_scheduler] Job 879 finished 1 steps successfully
[2017-06-01 19:06:11,118: INFO/job_scheduler] Job 879: complete_job: Updating cache
[2017-06-01 19:06:11,119: INFO/job_scheduler] Job 879 completing (errored=False, cancelled=False)
[2017-06-01 19:06:11,161: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:06:11,214: INFO/job_scheduler] Job 880 finished 1 steps successfully
[2017-06-01 19:06:11,252: INFO/job_scheduler] Job 880: complete_job: Updating cache
[2017-06-01 19:06:11,252: INFO/job_scheduler] Job 880 completing (errored=False, cancelled=False)
[2017-06-01 19:06:11,360: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:06:11,632: INFO/job_scheduler] Job 881 finished 1 steps successfully
[2017-06-01 19:06:11,678: INFO/job_scheduler] Job 881: complete_job: Updating cache
[2017-06-01 19:06:11,678: INFO/job_scheduler] Job 881 completing (errored=False, cancelled=False)
[2017-06-01 19:06:11,716: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:06:11,747: INFO/job_scheduler] Job 882 finished 1 steps successfully
[2017-06-01 19:06:11,786: INFO/job_scheduler] Job 882: complete_job: Updating cache
[2017-06-01 19:06:11,792: INFO/job_scheduler] Job 882 completing (errored=False, cancelled=False)
[2017-06-01 19:06:11,853: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:06:13,542: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:13,543: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:13,551: INFO/command_plan] add_jobs: created Job 883 (Test for host connectivity)
[2017-06-01 19:06:13,583: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:06:13,587: INFO/job_scheduler] Job 883: RunJobThread.run
[2017-06-01 19:06:13,675: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:13,677: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:13,686: INFO/command_plan] add_jobs: created Job 884 (Test for host connectivity)
[2017-06-01 19:06:13,791: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:06:13,798: INFO/job_scheduler] Job 884: RunJobThread.run
[2017-06-01 19:06:13,843: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:13,849: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:13,869: INFO/command_plan] add_jobs: created Job 885 (Test for host connectivity)
[2017-06-01 19:06:13,914: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:06:13,925: INFO/job_scheduler] Job 885: RunJobThread.run
[2017-06-01 19:06:13,960: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:13,961: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:14,100: INFO/command_plan] add_jobs: created Job 886 (Test for host connectivity)
[2017-06-01 19:06:14,184: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:06:14,200: INFO/job_scheduler] Job 886: RunJobThread.run
[2017-06-01 19:06:15,809: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:06:16,044: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:06:16,181: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:06:16,328: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:06:17,141: INFO/job_scheduler] Job 883 finished 1 steps successfully
[2017-06-01 19:06:17,167: INFO/job_scheduler] Job 883: complete_job: Updating cache
[2017-06-01 19:06:17,168: INFO/job_scheduler] Job 883 completing (errored=False, cancelled=False)
[2017-06-01 19:06:17,209: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:06:17,439: INFO/job_scheduler] Job 884 finished 1 steps successfully
[2017-06-01 19:06:17,459: INFO/job_scheduler] Job 884: complete_job: Updating cache
[2017-06-01 19:06:17,459: INFO/job_scheduler] Job 884 completing (errored=False, cancelled=False)
[2017-06-01 19:06:17,527: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:06:17,612: INFO/job_scheduler] Job 885 finished 1 steps successfully
[2017-06-01 19:06:17,651: INFO/job_scheduler] Job 885: complete_job: Updating cache
[2017-06-01 19:06:17,651: INFO/job_scheduler] Job 885 completing (errored=False, cancelled=False)
[2017-06-01 19:06:17,694: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:06:17,727: INFO/job_scheduler] Job 886 finished 1 steps successfully
[2017-06-01 19:06:17,771: INFO/job_scheduler] Job 886: complete_job: Updating cache
[2017-06-01 19:06:17,771: INFO/job_scheduler] Job 886 completing (errored=False, cancelled=False)
[2017-06-01 19:06:17,839: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:06:19,869: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:19,871: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:19,917: INFO/command_plan] add_jobs: created Job 887 (Test for host connectivity)
[2017-06-01 19:06:19,954: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:19,955: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:19,967: INFO/command_plan] add_jobs: created Job 888 (Test for host connectivity)
[2017-06-01 19:06:19,976: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:06:19,991: INFO/job_scheduler] Job 888: RunJobThread.run
[2017-06-01 19:06:19,992: INFO/job_scheduler] Job 887: RunJobThread.run
[2017-06-01 19:06:20,017: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:06:20,075: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:20,075: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:20,093: INFO/command_plan] add_jobs: created Job 889 (Test for host connectivity)
[2017-06-01 19:06:20,256: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:06:20,261: INFO/job_scheduler] Job 889: RunJobThread.run
[2017-06-01 19:06:20,309: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:20,311: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:20,337: INFO/command_plan] add_jobs: created Job 890 (Test for host connectivity)
[2017-06-01 19:06:20,447: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:06:20,455: INFO/job_scheduler] Job 890: RunJobThread.run
[2017-06-01 19:06:22,333: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:06:22,528: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:06:22,554: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:06:22,711: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:06:23,715: INFO/job_scheduler] Job 888 finished 1 steps successfully
[2017-06-01 19:06:23,783: INFO/job_scheduler] Job 888: complete_job: Updating cache
[2017-06-01 19:06:23,783: INFO/job_scheduler] Job 888 completing (errored=False, cancelled=False)
[2017-06-01 19:06:23,866: INFO/job_scheduler] Job 887 finished 1 steps successfully
[2017-06-01 19:06:23,924: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:06:23,941: INFO/job_scheduler] Job 887: complete_job: Updating cache
[2017-06-01 19:06:23,941: INFO/job_scheduler] Job 887 completing (errored=False, cancelled=False)
[2017-06-01 19:06:24,001: INFO/job_scheduler] Job 889 finished 1 steps successfully
[2017-06-01 19:06:24,001: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:06:24,016: INFO/job_scheduler] Job 889: complete_job: Updating cache
[2017-06-01 19:06:24,016: INFO/job_scheduler] Job 889 completing (errored=False, cancelled=False)
[2017-06-01 19:06:24,091: INFO/job_scheduler] Job 890 finished 1 steps successfully
[2017-06-01 19:06:24,108: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:06:24,150: INFO/job_scheduler] Job 890: complete_job: Updating cache
[2017-06-01 19:06:24,150: INFO/job_scheduler] Job 890 completing (errored=False, cancelled=False)
[2017-06-01 19:06:24,225: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:06:26,047: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:26,056: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:26,075: INFO/command_plan] add_jobs: created Job 891 (Test for host connectivity)
[2017-06-01 19:06:26,112: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:06:26,144: INFO/job_scheduler] Job 891: RunJobThread.run
[2017-06-01 19:06:26,194: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:26,196: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:26,218: INFO/command_plan] add_jobs: created Job 892 (Test for host connectivity)
[2017-06-01 19:06:26,305: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:06:26,313: INFO/job_scheduler] Job 892: RunJobThread.run
[2017-06-01 19:06:26,346: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:26,347: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:26,389: INFO/command_plan] add_jobs: created Job 893 (Test for host connectivity)
[2017-06-01 19:06:26,422: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:06:26,429: INFO/job_scheduler] Job 893: RunJobThread.run
[2017-06-01 19:06:26,491: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:26,497: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:26,557: INFO/command_plan] add_jobs: created Job 894 (Test for host connectivity)
[2017-06-01 19:06:26,607: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:06:26,621: INFO/job_scheduler] Job 894: RunJobThread.run
[2017-06-01 19:06:28,428: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:06:28,542: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:06:28,761: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:06:28,793: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:06:29,728: INFO/job_scheduler] Job 891 finished 1 steps successfully
[2017-06-01 19:06:29,788: INFO/job_scheduler] Job 891: complete_job: Updating cache
[2017-06-01 19:06:29,788: INFO/job_scheduler] Job 891 completing (errored=False, cancelled=False)
[2017-06-01 19:06:29,865: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:06:29,891: INFO/job_scheduler] Job 892 finished 1 steps successfully
[2017-06-01 19:06:29,919: INFO/job_scheduler] Job 892: complete_job: Updating cache
[2017-06-01 19:06:29,920: INFO/job_scheduler] Job 892 completing (errored=False, cancelled=False)
[2017-06-01 19:06:30,003: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:06:30,164: INFO/job_scheduler] Job 893 finished 1 steps successfully
[2017-06-01 19:06:30,175: INFO/job_scheduler] Job 894 finished 1 steps successfully
[2017-06-01 19:06:30,182: INFO/job_scheduler] Job 893: complete_job: Updating cache
[2017-06-01 19:06:30,182: INFO/job_scheduler] Job 893 completing (errored=False, cancelled=False)
[2017-06-01 19:06:30,216: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:06:30,250: INFO/job_scheduler] Job 894: complete_job: Updating cache
[2017-06-01 19:06:30,257: INFO/job_scheduler] Job 894 completing (errored=False, cancelled=False)
[2017-06-01 19:06:30,324: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:06:31,685: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:31,687: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:31,707: INFO/command_plan] add_jobs: created Job 895 (Test for host connectivity)
[2017-06-01 19:06:31,760: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:06:31,767: INFO/job_scheduler] Job 895: RunJobThread.run
[2017-06-01 19:06:31,828: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:31,828: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:31,843: INFO/command_plan] add_jobs: created Job 896 (Test for host connectivity)
[2017-06-01 19:06:31,905: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:06:31,965: INFO/job_scheduler] Job 896: RunJobThread.run
[2017-06-01 19:06:32,021: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:32,023: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:32,069: INFO/command_plan] add_jobs: created Job 897 (Test for host connectivity)
[2017-06-01 19:06:32,132: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:32,132: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:32,182: INFO/command_plan] add_jobs: created Job 898 (Test for host connectivity)
[2017-06-01 19:06:32,203: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:06:32,230: INFO/job_scheduler] Job 897: RunJobThread.run
[2017-06-01 19:06:32,237: INFO/job_scheduler] Job 898: RunJobThread.run
[2017-06-01 19:06:32,248: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:06:34,038: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:06:34,210: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:06:34,527: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:06:34,585: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:06:35,351: INFO/job_scheduler] Job 895 finished 1 steps successfully
[2017-06-01 19:06:35,380: INFO/job_scheduler] Job 895: complete_job: Updating cache
[2017-06-01 19:06:35,381: INFO/job_scheduler] Job 895 completing (errored=False, cancelled=False)
[2017-06-01 19:06:35,431: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:06:35,562: INFO/job_scheduler] Job 896 finished 1 steps successfully
[2017-06-01 19:06:35,617: INFO/job_scheduler] Job 896: complete_job: Updating cache
[2017-06-01 19:06:35,617: INFO/job_scheduler] Job 896 completing (errored=False, cancelled=False)
[2017-06-01 19:06:35,647: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:06:35,936: INFO/job_scheduler] Job 898 finished 1 steps successfully
[2017-06-01 19:06:35,972: INFO/job_scheduler] Job 898: complete_job: Updating cache
[2017-06-01 19:06:35,972: INFO/job_scheduler] Job 898 completing (errored=False, cancelled=False)
[2017-06-01 19:06:36,007: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:06:36,138: INFO/job_scheduler] Job 897 finished 1 steps successfully
[2017-06-01 19:06:36,264: INFO/job_scheduler] Job 897: complete_job: Updating cache
[2017-06-01 19:06:36,264: INFO/job_scheduler] Job 897 completing (errored=False, cancelled=False)
[2017-06-01 19:06:36,314: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:06:37,912: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:37,912: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:37,922: INFO/command_plan] add_jobs: created Job 899 (Test for host connectivity)
[2017-06-01 19:06:37,984: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:06:38,002: INFO/job_scheduler] Job 899: RunJobThread.run
[2017-06-01 19:06:38,040: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:38,040: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:38,058: INFO/command_plan] add_jobs: created Job 900 (Test for host connectivity)
[2017-06-01 19:06:38,188: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:38,189: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:38,222: INFO/command_plan] add_jobs: created Job 901 (Test for host connectivity)
[2017-06-01 19:06:38,245: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:06:38,292: INFO/job_scheduler] Job 900: RunJobThread.run
[2017-06-01 19:06:38,301: INFO/job_scheduler] Job 901: RunJobThread.run
[2017-06-01 19:06:38,329: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:38,330: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:38,434: INFO/command_plan] add_jobs: created Job 902 (Test for host connectivity)
[2017-06-01 19:06:38,463: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:06:38,479: INFO/job_scheduler] Job 902: RunJobThread.run
[2017-06-01 19:06:38,680: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:06:40,436: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:06:40,463: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:06:40,520: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:06:40,756: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:06:41,794: INFO/job_scheduler] Job 899 finished 1 steps successfully
[2017-06-01 19:06:41,803: INFO/job_scheduler] Job 900 finished 1 steps successfully
[2017-06-01 19:06:41,821: INFO/job_scheduler] Job 899: complete_job: Updating cache
[2017-06-01 19:06:41,821: INFO/job_scheduler] Job 899 completing (errored=False, cancelled=False)
[2017-06-01 19:06:41,897: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:06:41,926: INFO/job_scheduler] Job 900: complete_job: Updating cache
[2017-06-01 19:06:41,927: INFO/job_scheduler] Job 900 completing (errored=False, cancelled=False)
[2017-06-01 19:06:42,002: INFO/job_scheduler] Job 901 finished 1 steps successfully
[2017-06-01 19:06:42,004: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:06:42,021: INFO/job_scheduler] Job 901: complete_job: Updating cache
[2017-06-01 19:06:42,021: INFO/job_scheduler] Job 901 completing (errored=False, cancelled=False)
[2017-06-01 19:06:42,105: INFO/job_scheduler] Job 902 finished 1 steps successfully
[2017-06-01 19:06:42,132: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:06:42,156: INFO/job_scheduler] Job 902: complete_job: Updating cache
[2017-06-01 19:06:42,156: INFO/job_scheduler] Job 902 completing (errored=False, cancelled=False)
[2017-06-01 19:06:42,247: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:06:44,133: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:44,135: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:44,146: INFO/command_plan] add_jobs: created Job 903 (Test for host connectivity)
[2017-06-01 19:06:44,174: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:06:44,182: INFO/job_scheduler] Job 903: RunJobThread.run
[2017-06-01 19:06:44,246: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:44,246: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:44,269: INFO/command_plan] add_jobs: created Job 904 (Test for host connectivity)
[2017-06-01 19:06:44,329: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:06:44,353: INFO/job_scheduler] Job 904: RunJobThread.run
[2017-06-01 19:06:44,377: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:44,378: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:44,396: INFO/command_plan] add_jobs: created Job 905 (Test for host connectivity)
[2017-06-01 19:06:44,406: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:06:44,409: INFO/job_scheduler] Job 905: RunJobThread.run
[2017-06-01 19:06:44,555: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:44,555: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:44,594: INFO/command_plan] add_jobs: created Job 906 (Test for host connectivity)
[2017-06-01 19:06:44,697: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:06:44,704: INFO/job_scheduler] Job 906: RunJobThread.run
[2017-06-01 19:06:46,564: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:06:46,700: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:06:46,879: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:06:46,897: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:06:47,909: INFO/job_scheduler] Job 903 finished 1 steps successfully
[2017-06-01 19:06:47,970: INFO/job_scheduler] Job 903: complete_job: Updating cache
[2017-06-01 19:06:47,970: INFO/job_scheduler] Job 903 completing (errored=False, cancelled=False)
[2017-06-01 19:06:48,046: INFO/job_scheduler] Job 904 finished 1 steps successfully
[2017-06-01 19:06:48,122: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:06:48,145: INFO/job_scheduler] Job 904: complete_job: Updating cache
[2017-06-01 19:06:48,145: INFO/job_scheduler] Job 904 completing (errored=False, cancelled=False)
[2017-06-01 19:06:48,279: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:06:48,370: INFO/job_scheduler] Job 905 finished 1 steps successfully
[2017-06-01 19:06:48,371: INFO/job_scheduler] Job 906 finished 1 steps successfully
[2017-06-01 19:06:48,395: INFO/job_scheduler] Job 905: complete_job: Updating cache
[2017-06-01 19:06:48,395: INFO/job_scheduler] Job 905 completing (errored=False, cancelled=False)
[2017-06-01 19:06:48,536: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:06:48,544: INFO/job_scheduler] Job 906: complete_job: Updating cache
[2017-06-01 19:06:48,545: INFO/job_scheduler] Job 906 completing (errored=False, cancelled=False)
[2017-06-01 19:06:48,637: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:06:50,363: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:50,363: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:50,371: INFO/command_plan] add_jobs: created Job 907 (Test for host connectivity)
[2017-06-01 19:06:50,412: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:06:50,416: INFO/job_scheduler] Job 907: RunJobThread.run
[2017-06-01 19:06:50,479: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:50,480: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:50,524: INFO/command_plan] add_jobs: created Job 908 (Test for host connectivity)
[2017-06-01 19:06:50,573: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:06:50,576: INFO/job_scheduler] Job 908: RunJobThread.run
[2017-06-01 19:06:50,642: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:50,644: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:50,654: INFO/command_plan] add_jobs: created Job 909 (Test for host connectivity)
[2017-06-01 19:06:50,698: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:06:50,727: INFO/job_scheduler] Job 909: RunJobThread.run
[2017-06-01 19:06:50,800: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:50,801: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:50,828: INFO/command_plan] add_jobs: created Job 910 (Test for host connectivity)
[2017-06-01 19:06:50,917: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:06:50,940: INFO/job_scheduler] Job 910: RunJobThread.run
[2017-06-01 19:06:52,725: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:06:52,889: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:06:53,039: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:06:53,173: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:06:54,041: INFO/job_scheduler] Job 907 finished 1 steps successfully
[2017-06-01 19:06:54,069: INFO/job_scheduler] Job 907: complete_job: Updating cache
[2017-06-01 19:06:54,069: INFO/job_scheduler] Job 907 completing (errored=False, cancelled=False)
[2017-06-01 19:06:54,138: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:06:54,227: INFO/job_scheduler] Job 908 finished 1 steps successfully
[2017-06-01 19:06:54,279: INFO/job_scheduler] Job 908: complete_job: Updating cache
[2017-06-01 19:06:54,279: INFO/job_scheduler] Job 908 completing (errored=False, cancelled=False)
[2017-06-01 19:06:54,352: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:06:54,548: INFO/job_scheduler] Job 909 finished 1 steps successfully
[2017-06-01 19:06:54,602: INFO/job_scheduler] Job 909: complete_job: Updating cache
[2017-06-01 19:06:54,602: INFO/job_scheduler] Job 909 completing (errored=False, cancelled=False)
[2017-06-01 19:06:54,666: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:06:54,713: INFO/job_scheduler] Job 910 finished 1 steps successfully
[2017-06-01 19:06:54,736: INFO/job_scheduler] Job 910: complete_job: Updating cache
[2017-06-01 19:06:54,736: INFO/job_scheduler] Job 910 completing (errored=False, cancelled=False)
[2017-06-01 19:06:54,793: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:06:56,598: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:56,600: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:56,612: INFO/command_plan] add_jobs: created Job 911 (Test for host connectivity)
[2017-06-01 19:06:56,683: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:56,683: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:56,745: INFO/command_plan] add_jobs: created Job 912 (Test for host connectivity)
[2017-06-01 19:06:56,760: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:06:56,773: INFO/job_scheduler] Job 912: RunJobThread.run
[2017-06-01 19:06:56,795: INFO/job_scheduler] Job 911: RunJobThread.run
[2017-06-01 19:06:56,805: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:06:56,837: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:56,840: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:56,884: INFO/command_plan] add_jobs: created Job 913 (Test for host connectivity)
[2017-06-01 19:06:56,931: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:06:56,986: INFO/job_scheduler] Job 913: RunJobThread.run
[2017-06-01 19:06:57,015: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:06:57,015: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:06:57,074: INFO/command_plan] add_jobs: created Job 914 (Test for host connectivity)
[2017-06-01 19:06:57,108: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:06:57,130: INFO/job_scheduler] Job 914: RunJobThread.run
[2017-06-01 19:06:59,125: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:06:59,162: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:06:59,345: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:06:59,461: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:07:00,544: INFO/job_scheduler] Job 911 finished 1 steps successfully
[2017-06-01 19:07:00,544: INFO/job_scheduler] Job 912 finished 1 steps successfully
[2017-06-01 19:07:00,593: INFO/job_scheduler] Job 911: complete_job: Updating cache
[2017-06-01 19:07:00,593: INFO/job_scheduler] Job 911 completing (errored=False, cancelled=False)
[2017-06-01 19:07:00,618: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:07:00,621: INFO/job_scheduler] Job 912: complete_job: Updating cache
[2017-06-01 19:07:00,627: INFO/job_scheduler] Job 912 completing (errored=False, cancelled=False)
[2017-06-01 19:07:00,723: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:07:00,750: INFO/job_scheduler] Job 913 finished 1 steps successfully
[2017-06-01 19:07:00,793: INFO/job_scheduler] Job 913: complete_job: Updating cache
[2017-06-01 19:07:00,793: INFO/job_scheduler] Job 913 completing (errored=False, cancelled=False)
[2017-06-01 19:07:00,803: INFO/job_scheduler] Job 914 finished 1 steps successfully
[2017-06-01 19:07:00,877: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:07:00,902: INFO/job_scheduler] Job 914: complete_job: Updating cache
[2017-06-01 19:07:00,902: INFO/job_scheduler] Job 914 completing (errored=False, cancelled=False)
[2017-06-01 19:07:00,992: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:07:02,777: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:02,779: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:02,786: INFO/command_plan] add_jobs: created Job 915 (Test for host connectivity)
[2017-06-01 19:07:02,822: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:07:02,825: INFO/job_scheduler] Job 915: RunJobThread.run
[2017-06-01 19:07:02,898: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:02,909: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:02,958: INFO/command_plan] add_jobs: created Job 916 (Test for host connectivity)
[2017-06-01 19:07:02,995: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:07:03,025: INFO/job_scheduler] Job 916: RunJobThread.run
[2017-06-01 19:07:03,100: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:03,100: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:03,119: INFO/command_plan] add_jobs: created Job 917 (Test for host connectivity)
[2017-06-01 19:07:03,172: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:07:03,206: INFO/job_scheduler] Job 917: RunJobThread.run
[2017-06-01 19:07:03,276: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:03,277: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:03,330: INFO/command_plan] add_jobs: created Job 918 (Test for host connectivity)
[2017-06-01 19:07:03,426: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:07:03,433: INFO/job_scheduler] Job 918: RunJobThread.run
[2017-06-01 19:07:05,125: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:07:05,232: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:07:05,579: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:07:05,658: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:07:06,574: INFO/job_scheduler] Job 916 finished 1 steps successfully
[2017-06-01 19:07:06,649: INFO/job_scheduler] Job 916: complete_job: Updating cache
[2017-06-01 19:07:06,649: INFO/job_scheduler] Job 916 completing (errored=False, cancelled=False)
[2017-06-01 19:07:06,671: INFO/job_scheduler] Job 915 finished 1 steps successfully
[2017-06-01 19:07:06,701: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:07:06,719: INFO/job_scheduler] Job 915: complete_job: Updating cache
[2017-06-01 19:07:06,720: INFO/job_scheduler] Job 915 completing (errored=False, cancelled=False)
[2017-06-01 19:07:06,810: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:07:06,958: INFO/job_scheduler] Job 917 finished 1 steps successfully
[2017-06-01 19:07:06,992: INFO/job_scheduler] Job 917: complete_job: Updating cache
[2017-06-01 19:07:06,992: INFO/job_scheduler] Job 917 completing (errored=False, cancelled=False)
[2017-06-01 19:07:07,091: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:07:07,189: INFO/job_scheduler] Job 918 finished 1 steps successfully
[2017-06-01 19:07:07,242: INFO/job_scheduler] Job 918: complete_job: Updating cache
[2017-06-01 19:07:07,242: INFO/job_scheduler] Job 918 completing (errored=False, cancelled=False)
[2017-06-01 19:07:07,284: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:07:09,080: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:09,080: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:09,094: INFO/command_plan] add_jobs: created Job 919 (Test for host connectivity)
[2017-06-01 19:07:09,156: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:07:09,159: INFO/job_scheduler] Job 919: RunJobThread.run
[2017-06-01 19:07:09,190: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:09,191: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:09,212: INFO/command_plan] add_jobs: created Job 920 (Test for host connectivity)
[2017-06-01 19:07:09,293: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:07:09,302: INFO/job_scheduler] Job 920: RunJobThread.run
[2017-06-01 19:07:09,374: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:09,374: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:09,391: INFO/command_plan] add_jobs: created Job 921 (Test for host connectivity)
[2017-06-01 19:07:09,515: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:09,515: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:09,551: INFO/command_plan] add_jobs: created Job 922 (Test for host connectivity)
[2017-06-01 19:07:09,600: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:07:09,613: INFO/job_scheduler] Job 921: RunJobThread.run
[2017-06-01 19:07:09,614: INFO/job_scheduler] Job 922: RunJobThread.run
[2017-06-01 19:07:09,632: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:07:11,582: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:07:11,609: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:07:11,826: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:07:11,909: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:07:12,905: INFO/job_scheduler] Job 919 finished 1 steps successfully
[2017-06-01 19:07:12,950: INFO/job_scheduler] Job 920 finished 1 steps successfully
[2017-06-01 19:07:12,966: INFO/job_scheduler] Job 919: complete_job: Updating cache
[2017-06-01 19:07:12,967: INFO/job_scheduler] Job 919 completing (errored=False, cancelled=False)
[2017-06-01 19:07:13,019: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:07:13,043: INFO/job_scheduler] Job 920: complete_job: Updating cache
[2017-06-01 19:07:13,044: INFO/job_scheduler] Job 920 completing (errored=False, cancelled=False)
[2017-06-01 19:07:13,116: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:07:13,171: INFO/job_scheduler] Job 922 finished 1 steps successfully
[2017-06-01 19:07:13,194: INFO/job_scheduler] Job 922: complete_job: Updating cache
[2017-06-01 19:07:13,194: INFO/job_scheduler] Job 922 completing (errored=False, cancelled=False)
[2017-06-01 19:07:13,239: INFO/job_scheduler] Job 921 finished 1 steps successfully
[2017-06-01 19:07:13,286: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:07:13,311: INFO/job_scheduler] Job 921: complete_job: Updating cache
[2017-06-01 19:07:13,311: INFO/job_scheduler] Job 921 completing (errored=False, cancelled=False)
[2017-06-01 19:07:13,392: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:07:15,292: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:15,292: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:15,310: INFO/command_plan] add_jobs: created Job 923 (Test for host connectivity)
[2017-06-01 19:07:15,364: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:07:15,366: INFO/job_scheduler] Job 923: RunJobThread.run
[2017-06-01 19:07:15,426: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:15,434: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:15,454: INFO/command_plan] add_jobs: created Job 924 (Test for host connectivity)
[2017-06-01 19:07:15,519: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:07:15,535: INFO/job_scheduler] Job 924: RunJobThread.run
[2017-06-01 19:07:15,579: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:15,580: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:15,602: INFO/command_plan] add_jobs: created Job 925 (Test for host connectivity)
[2017-06-01 19:07:15,664: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:07:15,669: INFO/job_scheduler] Job 925: RunJobThread.run
[2017-06-01 19:07:15,735: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:15,736: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:15,751: INFO/command_plan] add_jobs: created Job 926 (Test for host connectivity)
[2017-06-01 19:07:15,784: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:07:15,823: INFO/job_scheduler] Job 926: RunJobThread.run
[2017-06-01 19:07:17,863: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:07:17,879: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:07:18,086: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:07:18,119: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:07:19,188: INFO/job_scheduler] Job 924 finished 1 steps successfully
[2017-06-01 19:07:19,244: INFO/job_scheduler] Job 923 finished 1 steps successfully
[2017-06-01 19:07:19,258: INFO/job_scheduler] Job 924: complete_job: Updating cache
[2017-06-01 19:07:19,258: INFO/job_scheduler] Job 924 completing (errored=False, cancelled=False)
[2017-06-01 19:07:19,319: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:07:19,358: INFO/job_scheduler] Job 923: complete_job: Updating cache
[2017-06-01 19:07:19,358: INFO/job_scheduler] Job 923 completing (errored=False, cancelled=False)
[2017-06-01 19:07:19,466: INFO/job_scheduler] Job 926 finished 1 steps successfully
[2017-06-01 19:07:19,475: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:07:19,500: INFO/job_scheduler] Job 926: complete_job: Updating cache
[2017-06-01 19:07:19,500: INFO/job_scheduler] Job 926 completing (errored=False, cancelled=False)
[2017-06-01 19:07:19,592: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:07:19,641: INFO/job_scheduler] Job 925 finished 1 steps successfully
[2017-06-01 19:07:19,674: INFO/job_scheduler] Job 925: complete_job: Updating cache
[2017-06-01 19:07:19,675: INFO/job_scheduler] Job 925 completing (errored=False, cancelled=False)
[2017-06-01 19:07:19,725: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:07:21,442: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:21,444: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:21,459: INFO/command_plan] add_jobs: created Job 927 (Test for host connectivity)
[2017-06-01 19:07:21,500: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:07:21,512: INFO/job_scheduler] Job 927: RunJobThread.run
[2017-06-01 19:07:21,574: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:21,584: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:21,599: INFO/command_plan] add_jobs: created Job 928 (Test for host connectivity)
[2017-06-01 19:07:21,608: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:07:21,612: INFO/job_scheduler] Job 928: RunJobThread.run
[2017-06-01 19:07:21,737: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:21,737: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:21,783: INFO/command_plan] add_jobs: created Job 929 (Test for host connectivity)
[2017-06-01 19:07:21,934: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:21,934: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:21,957: INFO/command_plan] add_jobs: created Job 930 (Test for host connectivity)
[2017-06-01 19:07:21,974: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:07:21,988: INFO/job_scheduler] Job 929: RunJobThread.run
[2017-06-01 19:07:22,017: INFO/job_scheduler] Job 930: RunJobThread.run
[2017-06-01 19:07:22,033: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:07:23,915: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:07:23,998: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:07:24,295: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:07:24,384: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:07:25,228: INFO/job_scheduler] Job 927 finished 1 steps successfully
[2017-06-01 19:07:25,248: INFO/job_scheduler] Job 927: complete_job: Updating cache
[2017-06-01 19:07:25,249: INFO/job_scheduler] Job 927 completing (errored=False, cancelled=False)
[2017-06-01 19:07:25,318: INFO/job_scheduler] Job 928 finished 1 steps successfully
[2017-06-01 19:07:25,357: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:07:25,383: INFO/job_scheduler] Job 928: complete_job: Updating cache
[2017-06-01 19:07:25,384: INFO/job_scheduler] Job 928 completing (errored=False, cancelled=False)
[2017-06-01 19:07:25,524: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:07:25,761: INFO/job_scheduler] Job 930 finished 1 steps successfully
[2017-06-01 19:07:25,815: INFO/job_scheduler] Job 930: complete_job: Updating cache
[2017-06-01 19:07:25,815: INFO/job_scheduler] Job 930 completing (errored=False, cancelled=False)
[2017-06-01 19:07:25,849: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:07:25,897: INFO/job_scheduler] Job 929 finished 1 steps successfully
[2017-06-01 19:07:25,955: INFO/job_scheduler] Job 929: complete_job: Updating cache
[2017-06-01 19:07:25,956: INFO/job_scheduler] Job 929 completing (errored=False, cancelled=False)
[2017-06-01 19:07:26,019: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:07:27,648: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:27,649: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:27,659: INFO/command_plan] add_jobs: created Job 931 (Test for host connectivity)
[2017-06-01 19:07:27,788: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:27,788: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:27,799: INFO/command_plan] add_jobs: created Job 932 (Test for host connectivity)
[2017-06-01 19:07:27,831: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:07:27,882: INFO/job_scheduler] Job 931: RunJobThread.run
[2017-06-01 19:07:27,884: INFO/job_scheduler] Job 932: RunJobThread.run
[2017-06-01 19:07:27,892: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:07:27,940: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:27,940: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:27,976: INFO/command_plan] add_jobs: created Job 933 (Test for host connectivity)
[2017-06-01 19:07:28,039: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:07:28,075: INFO/job_scheduler] Job 933: RunJobThread.run
[2017-06-01 19:07:28,157: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:28,162: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:28,187: INFO/command_plan] add_jobs: created Job 934 (Test for host connectivity)
[2017-06-01 19:07:28,255: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:07:28,260: INFO/job_scheduler] Job 934: RunJobThread.run
[2017-06-01 19:07:30,119: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:07:30,211: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:07:30,367: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:07:30,431: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:07:31,461: INFO/job_scheduler] Job 931 finished 1 steps successfully
[2017-06-01 19:07:31,482: INFO/job_scheduler] Job 931: complete_job: Updating cache
[2017-06-01 19:07:31,482: INFO/job_scheduler] Job 931 completing (errored=False, cancelled=False)
[2017-06-01 19:07:31,555: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:07:31,718: INFO/job_scheduler] Job 932 finished 1 steps successfully
[2017-06-01 19:07:31,781: INFO/job_scheduler] Job 932: complete_job: Updating cache
[2017-06-01 19:07:31,781: INFO/job_scheduler] Job 932 completing (errored=False, cancelled=False)
[2017-06-01 19:07:31,784: INFO/job_scheduler] Job 933 finished 1 steps successfully
[2017-06-01 19:07:31,824: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:07:31,858: INFO/job_scheduler] Job 933: complete_job: Updating cache
[2017-06-01 19:07:31,858: INFO/job_scheduler] Job 933 completing (errored=False, cancelled=False)
[2017-06-01 19:07:31,881: INFO/job_scheduler] Job 934 finished 1 steps successfully
[2017-06-01 19:07:31,941: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:07:32,001: INFO/job_scheduler] Job 934: complete_job: Updating cache
[2017-06-01 19:07:32,001: INFO/job_scheduler] Job 934 completing (errored=False, cancelled=False)
[2017-06-01 19:07:32,072: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:07:33,942: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:33,943: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:33,957: INFO/command_plan] add_jobs: created Job 935 (Test for host connectivity)
[2017-06-01 19:07:34,037: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:34,037: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:34,064: INFO/command_plan] add_jobs: created Job 936 (Test for host connectivity)
[2017-06-01 19:07:34,083: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:07:34,108: INFO/job_scheduler] Job 936: RunJobThread.run
[2017-06-01 19:07:34,113: INFO/job_scheduler] Job 935: RunJobThread.run
[2017-06-01 19:07:34,127: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:07:34,157: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:34,158: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:34,210: INFO/command_plan] add_jobs: created Job 937 (Test for host connectivity)
[2017-06-01 19:07:34,267: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:07:34,335: INFO/job_scheduler] Job 937: RunJobThread.run
[2017-06-01 19:07:34,363: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:34,364: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:34,382: INFO/command_plan] add_jobs: created Job 938 (Test for host connectivity)
[2017-06-01 19:07:34,404: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:07:34,487: INFO/job_scheduler] Job 938: RunJobThread.run
[2017-06-01 19:07:36,507: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:07:36,553: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:07:36,605: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:07:36,791: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:07:37,939: INFO/job_scheduler] Job 935 finished 1 steps successfully
[2017-06-01 19:07:37,943: INFO/job_scheduler] Job 936 finished 1 steps successfully
[2017-06-01 19:07:37,980: INFO/job_scheduler] Job 935: complete_job: Updating cache
[2017-06-01 19:07:37,981: INFO/job_scheduler] Job 935 completing (errored=False, cancelled=False)
[2017-06-01 19:07:38,050: INFO/job_scheduler] Job 937 finished 1 steps successfully
[2017-06-01 19:07:38,064: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:07:38,114: INFO/job_scheduler] Job 936: complete_job: Updating cache
[2017-06-01 19:07:38,115: INFO/job_scheduler] Job 936 completing (errored=False, cancelled=False)
[2017-06-01 19:07:38,150: INFO/job_scheduler] Job 938 finished 1 steps successfully
[2017-06-01 19:07:38,214: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:07:38,252: INFO/job_scheduler] Job 937: complete_job: Updating cache
[2017-06-01 19:07:38,252: INFO/job_scheduler] Job 937 completing (errored=False, cancelled=False)
[2017-06-01 19:07:38,354: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:07:38,386: INFO/job_scheduler] Job 938: complete_job: Updating cache
[2017-06-01 19:07:38,386: INFO/job_scheduler] Job 938 completing (errored=False, cancelled=False)
[2017-06-01 19:07:38,506: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:07:40,123: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:40,123: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:40,155: INFO/command_plan] add_jobs: created Job 939 (Test for host connectivity)
[2017-06-01 19:07:40,173: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:07:40,208: INFO/job_scheduler] Job 939: RunJobThread.run
[2017-06-01 19:07:40,231: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:40,231: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:40,297: INFO/command_plan] add_jobs: created Job 940 (Test for host connectivity)
[2017-06-01 19:07:40,335: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:07:40,390: INFO/job_scheduler] Job 940: RunJobThread.run
[2017-06-01 19:07:40,428: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:40,431: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:40,463: INFO/command_plan] add_jobs: created Job 941 (Test for host connectivity)
[2017-06-01 19:07:40,549: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:07:40,554: INFO/job_scheduler] Job 941: RunJobThread.run
[2017-06-01 19:07:40,607: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:40,610: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:40,630: INFO/command_plan] add_jobs: created Job 942 (Test for host connectivity)
[2017-06-01 19:07:40,725: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:07:40,747: INFO/job_scheduler] Job 942: RunJobThread.run
[2017-06-01 19:07:42,528: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:07:42,658: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:07:42,802: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:07:43,057: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:07:43,905: INFO/job_scheduler] Job 939 finished 1 steps successfully
[2017-06-01 19:07:43,938: INFO/job_scheduler] Job 939: complete_job: Updating cache
[2017-06-01 19:07:43,938: INFO/job_scheduler] Job 939 completing (errored=False, cancelled=False)
[2017-06-01 19:07:44,066: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:07:44,120: INFO/job_scheduler] Job 941 finished 1 steps successfully
[2017-06-01 19:07:44,199: INFO/job_scheduler] Job 941: complete_job: Updating cache
[2017-06-01 19:07:44,199: INFO/job_scheduler] Job 941 completing (errored=False, cancelled=False)
[2017-06-01 19:07:44,230: INFO/job_scheduler] Job 940 finished 1 steps successfully
[2017-06-01 19:07:44,296: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:07:44,315: INFO/job_scheduler] Job 940: complete_job: Updating cache
[2017-06-01 19:07:44,315: INFO/job_scheduler] Job 940 completing (errored=False, cancelled=False)
[2017-06-01 19:07:44,362: INFO/job_scheduler] Job 942 finished 1 steps successfully
[2017-06-01 19:07:44,396: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:07:44,431: INFO/job_scheduler] Job 942: complete_job: Updating cache
[2017-06-01 19:07:44,431: INFO/job_scheduler] Job 942 completing (errored=False, cancelled=False)
[2017-06-01 19:07:44,493: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:07:46,396: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:46,396: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:46,404: INFO/command_plan] add_jobs: created Job 943 (Test for host connectivity)
[2017-06-01 19:07:46,492: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:46,492: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:46,596: INFO/command_plan] add_jobs: created Job 944 (Test for host connectivity)
[2017-06-01 19:07:46,604: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:07:46,608: INFO/job_scheduler] Job 944: RunJobThread.run
[2017-06-01 19:07:46,609: INFO/job_scheduler] Job 943: RunJobThread.run
[2017-06-01 19:07:46,614: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:07:46,692: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:46,692: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:46,740: INFO/command_plan] add_jobs: created Job 945 (Test for host connectivity)
[2017-06-01 19:07:46,769: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:07:46,773: INFO/job_scheduler] Job 945: RunJobThread.run
[2017-06-01 19:07:46,870: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:46,871: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:46,889: INFO/command_plan] add_jobs: created Job 946 (Test for host connectivity)
[2017-06-01 19:07:46,971: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:07:46,985: INFO/job_scheduler] Job 946: RunJobThread.run
[2017-06-01 19:07:48,940: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:07:48,993: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:07:49,094: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:07:49,348: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:07:50,305: INFO/job_scheduler] Job 944 finished 1 steps successfully
[2017-06-01 19:07:50,327: INFO/job_scheduler] Job 943 finished 1 steps successfully
[2017-06-01 19:07:50,363: INFO/job_scheduler] Job 944: complete_job: Updating cache
[2017-06-01 19:07:50,363: INFO/job_scheduler] Job 944 completing (errored=False, cancelled=False)
[2017-06-01 19:07:50,478: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:07:50,500: INFO/job_scheduler] Job 943: complete_job: Updating cache
[2017-06-01 19:07:50,500: INFO/job_scheduler] Job 943 completing (errored=False, cancelled=False)
[2017-06-01 19:07:50,567: INFO/job_scheduler] Job 945 finished 1 steps successfully
[2017-06-01 19:07:50,597: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:07:50,616: INFO/job_scheduler] Job 945: complete_job: Updating cache
[2017-06-01 19:07:50,616: INFO/job_scheduler] Job 945 completing (errored=False, cancelled=False)
[2017-06-01 19:07:50,681: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:07:50,715: INFO/job_scheduler] Job 946 finished 1 steps successfully
[2017-06-01 19:07:50,770: INFO/job_scheduler] Job 946: complete_job: Updating cache
[2017-06-01 19:07:50,771: INFO/job_scheduler] Job 946 completing (errored=False, cancelled=False)
[2017-06-01 19:07:50,855: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:07:52,624: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:52,625: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:52,635: INFO/command_plan] add_jobs: created Job 947 (Test for host connectivity)
[2017-06-01 19:07:52,685: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:07:52,695: INFO/job_scheduler] Job 947: RunJobThread.run
[2017-06-01 19:07:52,741: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:52,742: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:52,773: INFO/command_plan] add_jobs: created Job 948 (Test for host connectivity)
[2017-06-01 19:07:52,808: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:07:52,852: INFO/job_scheduler] Job 948: RunJobThread.run
[2017-06-01 19:07:52,895: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:52,896: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:52,919: INFO/command_plan] add_jobs: created Job 949 (Test for host connectivity)
[2017-06-01 19:07:53,009: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:07:53,036: INFO/job_scheduler] Job 949: RunJobThread.run
[2017-06-01 19:07:53,089: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:53,089: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:53,141: INFO/command_plan] add_jobs: created Job 950 (Test for host connectivity)
[2017-06-01 19:07:53,198: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:07:53,204: INFO/job_scheduler] Job 950: RunJobThread.run
[2017-06-01 19:07:55,033: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:07:55,094: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:07:55,203: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:07:55,432: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:07:56,388: INFO/job_scheduler] Job 947 finished 1 steps successfully
[2017-06-01 19:07:56,426: INFO/job_scheduler] Job 947: complete_job: Updating cache
[2017-06-01 19:07:56,426: INFO/job_scheduler] Job 947 completing (errored=False, cancelled=False)
[2017-06-01 19:07:56,436: INFO/job_scheduler] Job 948 finished 1 steps successfully
[2017-06-01 19:07:56,503: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:07:56,539: INFO/job_scheduler] Job 948: complete_job: Updating cache
[2017-06-01 19:07:56,540: INFO/job_scheduler] Job 948 completing (errored=False, cancelled=False)
[2017-06-01 19:07:56,551: INFO/job_scheduler] Job 949 finished 1 steps successfully
[2017-06-01 19:07:56,619: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:07:56,651: INFO/job_scheduler] Job 949: complete_job: Updating cache
[2017-06-01 19:07:56,651: INFO/job_scheduler] Job 949 completing (errored=False, cancelled=False)
[2017-06-01 19:07:56,692: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:07:56,765: INFO/job_scheduler] Job 950 finished 1 steps successfully
[2017-06-01 19:07:56,792: INFO/job_scheduler] Job 950: complete_job: Updating cache
[2017-06-01 19:07:56,792: INFO/job_scheduler] Job 950 completing (errored=False, cancelled=False)
[2017-06-01 19:07:56,831: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:07:59,563: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:59,565: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:59,578: INFO/command_plan] add_jobs: created Job 951 (Test for host connectivity)
[2017-06-01 19:07:59,606: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:07:59,630: INFO/job_scheduler] Job 951: RunJobThread.run
[2017-06-01 19:07:59,665: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:59,665: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:59,677: INFO/command_plan] add_jobs: created Job 952 (Test for host connectivity)
[2017-06-01 19:07:59,714: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:07:59,743: INFO/job_scheduler] Job 952: RunJobThread.run
[2017-06-01 19:07:59,837: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:07:59,838: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:07:59,984: INFO/command_plan] add_jobs: created Job 953 (Test for host connectivity)
[2017-06-01 19:08:00,060: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:00,060: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:00,083: INFO/command_plan] add_jobs: created Job 954 (Test for host connectivity)
[2017-06-01 19:08:00,102: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:08:00,106: INFO/job_scheduler] Job 953: RunJobThread.run
[2017-06-01 19:08:00,106: INFO/job_scheduler] Job 954: RunJobThread.run
[2017-06-01 19:08:00,118: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:08:01,959: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:08:02,040: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:08:02,448: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:08:02,457: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:08:03,285: INFO/job_scheduler] Job 951 finished 1 steps successfully
[2017-06-01 19:08:03,309: INFO/job_scheduler] Job 951: complete_job: Updating cache
[2017-06-01 19:08:03,309: INFO/job_scheduler] Job 951 completing (errored=False, cancelled=False)
[2017-06-01 19:08:03,353: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:08:03,401: INFO/job_scheduler] Job 952 finished 1 steps successfully
[2017-06-01 19:08:03,508: INFO/job_scheduler] Job 952: complete_job: Updating cache
[2017-06-01 19:08:03,508: INFO/job_scheduler] Job 952 completing (errored=False, cancelled=False)
[2017-06-01 19:08:03,577: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:08:03,878: INFO/job_scheduler] Job 953 finished 1 steps successfully
[2017-06-01 19:08:03,884: INFO/job_scheduler] Job 954 finished 1 steps successfully
[2017-06-01 19:08:03,933: INFO/job_scheduler] Job 953: complete_job: Updating cache
[2017-06-01 19:08:03,933: INFO/job_scheduler] Job 953 completing (errored=False, cancelled=False)
[2017-06-01 19:08:03,967: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:08:03,989: INFO/job_scheduler] Job 954: complete_job: Updating cache
[2017-06-01 19:08:03,989: INFO/job_scheduler] Job 954 completing (errored=False, cancelled=False)
[2017-06-01 19:08:04,077: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:08:05,844: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:05,844: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:05,866: INFO/command_plan] add_jobs: created Job 955 (Test for host connectivity)
[2017-06-01 19:08:05,896: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:08:05,917: INFO/job_scheduler] Job 955: RunJobThread.run
[2017-06-01 19:08:05,971: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:05,972: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:06,008: INFO/command_plan] add_jobs: created Job 956 (Test for host connectivity)
[2017-06-01 19:08:06,071: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:08:06,105: INFO/job_scheduler] Job 956: RunJobThread.run
[2017-06-01 19:08:06,148: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:06,151: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:06,183: INFO/command_plan] add_jobs: created Job 957 (Test for host connectivity)
[2017-06-01 19:08:06,312: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:08:06,319: INFO/job_scheduler] Job 957: RunJobThread.run
[2017-06-01 19:08:06,367: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:06,377: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:06,391: INFO/command_plan] add_jobs: created Job 958 (Test for host connectivity)
[2017-06-01 19:08:06,420: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:08:06,487: INFO/job_scheduler] Job 958: RunJobThread.run
[2017-06-01 19:08:08,285: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:08:08,383: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:08:08,530: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:08:08,754: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:08:09,673: INFO/job_scheduler] Job 955 finished 1 steps successfully
[2017-06-01 19:08:09,690: INFO/job_scheduler] Job 955: complete_job: Updating cache
[2017-06-01 19:08:09,690: INFO/job_scheduler] Job 955 completing (errored=False, cancelled=False)
[2017-06-01 19:08:09,724: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:08:09,772: INFO/job_scheduler] Job 956 finished 1 steps successfully
[2017-06-01 19:08:09,804: INFO/job_scheduler] Job 956: complete_job: Updating cache
[2017-06-01 19:08:09,804: INFO/job_scheduler] Job 956 completing (errored=False, cancelled=False)
[2017-06-01 19:08:09,866: INFO/job_scheduler] Job 957 finished 1 steps successfully
[2017-06-01 19:08:09,890: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:08:09,950: INFO/job_scheduler] Job 957: complete_job: Updating cache
[2017-06-01 19:08:09,951: INFO/job_scheduler] Job 957 completing (errored=False, cancelled=False)
[2017-06-01 19:08:10,048: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:08:10,275: INFO/job_scheduler] Job 958 finished 1 steps successfully
[2017-06-01 19:08:10,298: INFO/job_scheduler] Job 958: complete_job: Updating cache
[2017-06-01 19:08:10,298: INFO/job_scheduler] Job 958 completing (errored=False, cancelled=False)
[2017-06-01 19:08:10,354: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:08:12,133: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:12,133: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:12,160: INFO/command_plan] add_jobs: created Job 959 (Test for host connectivity)
[2017-06-01 19:08:12,192: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:08:12,204: INFO/job_scheduler] Job 959: RunJobThread.run
[2017-06-01 19:08:12,245: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:12,246: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:12,271: INFO/command_plan] add_jobs: created Job 960 (Test for host connectivity)
[2017-06-01 19:08:12,313: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:08:12,339: INFO/job_scheduler] Job 960: RunJobThread.run
[2017-06-01 19:08:12,463: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:12,485: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:12,499: INFO/command_plan] add_jobs: created Job 961 (Test for host connectivity)
[2017-06-01 19:08:12,586: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:08:12,619: INFO/job_scheduler] Job 961: RunJobThread.run
[2017-06-01 19:08:12,672: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:12,673: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:12,691: INFO/command_plan] add_jobs: created Job 962 (Test for host connectivity)
[2017-06-01 19:08:12,731: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:08:12,794: INFO/job_scheduler] Job 962: RunJobThread.run
[2017-06-01 19:08:14,628: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:08:14,656: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:08:14,902: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:08:14,985: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:08:15,957: INFO/job_scheduler] Job 959 finished 1 steps successfully
[2017-06-01 19:08:16,007: INFO/job_scheduler] Job 959: complete_job: Updating cache
[2017-06-01 19:08:16,007: INFO/job_scheduler] Job 959 completing (errored=False, cancelled=False)
[2017-06-01 19:08:16,012: INFO/job_scheduler] Job 960 finished 1 steps successfully
[2017-06-01 19:08:16,078: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:08:16,098: INFO/job_scheduler] Job 960: complete_job: Updating cache
[2017-06-01 19:08:16,102: INFO/job_scheduler] Job 960 completing (errored=False, cancelled=False)
[2017-06-01 19:08:16,215: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:08:16,311: INFO/job_scheduler] Job 962 finished 1 steps successfully
[2017-06-01 19:08:16,355: INFO/job_scheduler] Job 962: complete_job: Updating cache
[2017-06-01 19:08:16,356: INFO/job_scheduler] Job 962 completing (errored=False, cancelled=False)
[2017-06-01 19:08:16,407: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:08:16,444: INFO/job_scheduler] Job 961 finished 1 steps successfully
[2017-06-01 19:08:16,474: INFO/job_scheduler] Job 961: complete_job: Updating cache
[2017-06-01 19:08:16,474: INFO/job_scheduler] Job 961 completing (errored=False, cancelled=False)
[2017-06-01 19:08:16,543: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:08:18,478: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:18,479: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:18,497: INFO/command_plan] add_jobs: created Job 963 (Test for host connectivity)
[2017-06-01 19:08:18,515: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:08:18,523: INFO/job_scheduler] Job 963: RunJobThread.run
[2017-06-01 19:08:18,622: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:18,622: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:18,673: INFO/command_plan] add_jobs: created Job 964 (Test for host connectivity)
[2017-06-01 19:08:18,715: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:08:18,756: INFO/job_scheduler] Job 964: RunJobThread.run
[2017-06-01 19:08:18,802: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:18,803: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:18,819: INFO/command_plan] add_jobs: created Job 965 (Test for host connectivity)
[2017-06-01 19:08:18,849: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:08:18,928: INFO/job_scheduler] Job 965: RunJobThread.run
[2017-06-01 19:08:18,984: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:18,984: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:19,014: INFO/command_plan] add_jobs: created Job 966 (Test for host connectivity)
[2017-06-01 19:08:19,110: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:08:19,143: INFO/job_scheduler] Job 966: RunJobThread.run
[2017-06-01 19:08:20,839: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:08:21,136: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:08:21,301: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:08:21,329: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:08:22,156: INFO/job_scheduler] Job 963 finished 1 steps successfully
[2017-06-01 19:08:22,207: INFO/job_scheduler] Job 963: complete_job: Updating cache
[2017-06-01 19:08:22,207: INFO/job_scheduler] Job 963 completing (errored=False, cancelled=False)
[2017-06-01 19:08:22,255: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:08:22,529: INFO/job_scheduler] Job 964 finished 1 steps successfully
[2017-06-01 19:08:22,579: INFO/job_scheduler] Job 964: complete_job: Updating cache
[2017-06-01 19:08:22,579: INFO/job_scheduler] Job 964 completing (errored=False, cancelled=False)
[2017-06-01 19:08:22,646: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:08:22,719: INFO/job_scheduler] Job 966 finished 1 steps successfully
[2017-06-01 19:08:22,730: INFO/job_scheduler] Job 966: complete_job: Updating cache
[2017-06-01 19:08:22,730: INFO/job_scheduler] Job 966 completing (errored=False, cancelled=False)
[2017-06-01 19:08:22,798: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:08:22,819: INFO/job_scheduler] Job 965 finished 1 steps successfully
[2017-06-01 19:08:22,863: INFO/job_scheduler] Job 965: complete_job: Updating cache
[2017-06-01 19:08:22,864: INFO/job_scheduler] Job 965 completing (errored=False, cancelled=False)
[2017-06-01 19:08:22,998: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:08:24,783: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:24,784: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:24,804: INFO/command_plan] add_jobs: created Job 967 (Test for host connectivity)
[2017-06-01 19:08:24,837: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:08:24,849: INFO/job_scheduler] Job 967: RunJobThread.run
[2017-06-01 19:08:24,924: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:24,927: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:24,945: INFO/command_plan] add_jobs: created Job 968 (Test for host connectivity)
[2017-06-01 19:08:25,006: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:08:25,041: INFO/job_scheduler] Job 968: RunJobThread.run
[2017-06-01 19:08:25,080: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:25,089: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:25,111: INFO/command_plan] add_jobs: created Job 969 (Test for host connectivity)
[2017-06-01 19:08:25,179: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:08:25,188: INFO/job_scheduler] Job 969: RunJobThread.run
[2017-06-01 19:08:25,246: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:25,248: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:25,261: INFO/command_plan] add_jobs: created Job 970 (Test for host connectivity)
[2017-06-01 19:08:25,294: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:08:25,340: INFO/job_scheduler] Job 970: RunJobThread.run
[2017-06-01 19:08:27,167: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:08:27,304: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:08:27,524: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:08:27,607: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:08:28,468: INFO/job_scheduler] Job 967 finished 1 steps successfully
[2017-06-01 19:08:28,519: INFO/job_scheduler] Job 967: complete_job: Updating cache
[2017-06-01 19:08:28,519: INFO/job_scheduler] Job 967 completing (errored=False, cancelled=False)
[2017-06-01 19:08:28,626: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:08:28,651: INFO/job_scheduler] Job 968 finished 1 steps successfully
[2017-06-01 19:08:28,671: INFO/job_scheduler] Job 968: complete_job: Updating cache
[2017-06-01 19:08:28,678: INFO/job_scheduler] Job 968 completing (errored=False, cancelled=False)
[2017-06-01 19:08:28,760: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:08:28,874: INFO/job_scheduler] Job 969 finished 1 steps successfully
[2017-06-01 19:08:28,927: INFO/job_scheduler] Job 969: complete_job: Updating cache
[2017-06-01 19:08:28,928: INFO/job_scheduler] Job 969 completing (errored=False, cancelled=False)
[2017-06-01 19:08:28,978: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:08:29,163: INFO/job_scheduler] Job 970 finished 1 steps successfully
[2017-06-01 19:08:29,210: INFO/job_scheduler] Job 970: complete_job: Updating cache
[2017-06-01 19:08:29,211: INFO/job_scheduler] Job 970 completing (errored=False, cancelled=False)
[2017-06-01 19:08:29,247: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:08:31,128: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:31,128: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:31,161: INFO/command_plan] add_jobs: created Job 971 (Test for host connectivity)
[2017-06-01 19:08:31,217: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:08:31,221: INFO/job_scheduler] Job 971: RunJobThread.run
[2017-06-01 19:08:31,251: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:31,252: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:31,277: INFO/command_plan] add_jobs: created Job 972 (Test for host connectivity)
[2017-06-01 19:08:31,359: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:08:31,362: INFO/job_scheduler] Job 972: RunJobThread.run
[2017-06-01 19:08:31,404: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:31,404: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:31,441: INFO/command_plan] add_jobs: created Job 973 (Test for host connectivity)
[2017-06-01 19:08:31,462: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:08:31,483: INFO/job_scheduler] Job 973: RunJobThread.run
[2017-06-01 19:08:31,561: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:31,567: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:31,585: INFO/command_plan] add_jobs: created Job 974 (Test for host connectivity)
[2017-06-01 19:08:31,609: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:08:31,634: INFO/job_scheduler] Job 974: RunJobThread.run
[2017-06-01 19:08:33,539: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:08:33,679: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:08:33,732: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:08:33,916: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:08:34,875: INFO/job_scheduler] Job 971 finished 1 steps successfully
[2017-06-01 19:08:34,927: INFO/job_scheduler] Job 971: complete_job: Updating cache
[2017-06-01 19:08:34,927: INFO/job_scheduler] Job 971 completing (errored=False, cancelled=False)
[2017-06-01 19:08:35,018: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:08:35,079: INFO/job_scheduler] Job 972 finished 1 steps successfully
[2017-06-01 19:08:35,115: INFO/job_scheduler] Job 972: complete_job: Updating cache
[2017-06-01 19:08:35,115: INFO/job_scheduler] Job 972 completing (errored=False, cancelled=False)
[2017-06-01 19:08:35,159: INFO/job_scheduler] Job 973 finished 1 steps successfully
[2017-06-01 19:08:35,176: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:08:35,217: INFO/job_scheduler] Job 973: complete_job: Updating cache
[2017-06-01 19:08:35,217: INFO/job_scheduler] Job 973 completing (errored=False, cancelled=False)
[2017-06-01 19:08:35,249: INFO/job_scheduler] Job 974 finished 1 steps successfully
[2017-06-01 19:08:35,292: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:08:35,388: INFO/job_scheduler] Job 974: complete_job: Updating cache
[2017-06-01 19:08:35,390: INFO/job_scheduler] Job 974 completing (errored=False, cancelled=False)
[2017-06-01 19:08:35,443: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:08:37,310: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:37,312: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:37,334: INFO/command_plan] add_jobs: created Job 975 (Test for host connectivity)
[2017-06-01 19:08:37,385: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:37,385: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:37,409: INFO/command_plan] add_jobs: created Job 976 (Test for host connectivity)
[2017-06-01 19:08:37,451: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:08:37,481: INFO/job_scheduler] Job 976: RunJobThread.run
[2017-06-01 19:08:37,490: INFO/job_scheduler] Job 975: RunJobThread.run
[2017-06-01 19:08:37,503: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:08:37,564: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:37,565: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:37,586: INFO/command_plan] add_jobs: created Job 977 (Test for host connectivity)
[2017-06-01 19:08:37,669: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:08:37,671: INFO/job_scheduler] Job 977: RunJobThread.run
[2017-06-01 19:08:37,809: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:37,810: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:37,834: INFO/command_plan] add_jobs: created Job 978 (Test for host connectivity)
[2017-06-01 19:08:37,922: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:08:37,951: INFO/job_scheduler] Job 978: RunJobThread.run
[2017-06-01 19:08:39,659: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:08:39,819: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:08:39,915: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:08:40,087: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:08:40,944: INFO/job_scheduler] Job 975 finished 1 steps successfully
[2017-06-01 19:08:41,008: INFO/job_scheduler] Job 975: complete_job: Updating cache
[2017-06-01 19:08:41,008: INFO/job_scheduler] Job 975 completing (errored=False, cancelled=False)
[2017-06-01 19:08:41,118: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:08:41,189: INFO/job_scheduler] Job 976 finished 1 steps successfully
[2017-06-01 19:08:41,250: INFO/job_scheduler] Job 976: complete_job: Updating cache
[2017-06-01 19:08:41,250: INFO/job_scheduler] Job 976 completing (errored=False, cancelled=False)
[2017-06-01 19:08:41,316: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:08:41,352: INFO/job_scheduler] Job 977 finished 1 steps successfully
[2017-06-01 19:08:41,404: INFO/job_scheduler] Job 977: complete_job: Updating cache
[2017-06-01 19:08:41,404: INFO/job_scheduler] Job 977 completing (errored=False, cancelled=False)
[2017-06-01 19:08:41,443: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:08:41,521: INFO/job_scheduler] Job 978 finished 1 steps successfully
[2017-06-01 19:08:41,561: INFO/job_scheduler] Job 978: complete_job: Updating cache
[2017-06-01 19:08:41,562: INFO/job_scheduler] Job 978 completing (errored=False, cancelled=False)
[2017-06-01 19:08:41,617: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:08:43,558: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:43,558: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:43,576: INFO/command_plan] add_jobs: created Job 979 (Test for host connectivity)
[2017-06-01 19:08:43,621: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:08:43,640: INFO/job_scheduler] Job 979: RunJobThread.run
[2017-06-01 19:08:43,676: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:43,676: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:43,824: INFO/command_plan] add_jobs: created Job 980 (Test for host connectivity)
[2017-06-01 19:08:43,844: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:08:43,849: INFO/job_scheduler] Job 980: RunJobThread.run
[2017-06-01 19:08:43,881: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:43,881: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:43,958: INFO/command_plan] add_jobs: created Job 981 (Test for host connectivity)
[2017-06-01 19:08:44,041: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:08:44,047: INFO/job_scheduler] Job 981: RunJobThread.run
[2017-06-01 19:08:44,103: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:44,103: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:44,124: INFO/command_plan] add_jobs: created Job 982 (Test for host connectivity)
[2017-06-01 19:08:44,220: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:08:44,255: INFO/job_scheduler] Job 982: RunJobThread.run
[2017-06-01 19:08:45,931: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:08:46,242: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:08:46,302: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:08:46,566: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:08:47,261: INFO/job_scheduler] Job 979 finished 1 steps successfully
[2017-06-01 19:08:47,273: INFO/job_scheduler] Job 979: complete_job: Updating cache
[2017-06-01 19:08:47,274: INFO/job_scheduler] Job 979 completing (errored=False, cancelled=False)
[2017-06-01 19:08:47,330: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:08:47,725: INFO/job_scheduler] Job 981 finished 1 steps successfully
[2017-06-01 19:08:47,758: INFO/job_scheduler] Job 980 finished 1 steps successfully
[2017-06-01 19:08:47,773: INFO/job_scheduler] Job 981: complete_job: Updating cache
[2017-06-01 19:08:47,773: INFO/job_scheduler] Job 981 completing (errored=False, cancelled=False)
[2017-06-01 19:08:47,839: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:08:47,857: INFO/job_scheduler] Job 980: complete_job: Updating cache
[2017-06-01 19:08:47,857: INFO/job_scheduler] Job 980 completing (errored=False, cancelled=False)
[2017-06-01 19:08:47,904: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:08:47,936: INFO/job_scheduler] Job 982 finished 1 steps successfully
[2017-06-01 19:08:47,959: INFO/job_scheduler] Job 982: complete_job: Updating cache
[2017-06-01 19:08:47,959: INFO/job_scheduler] Job 982 completing (errored=False, cancelled=False)
[2017-06-01 19:08:48,007: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:08:49,861: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:49,863: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:49,881: INFO/command_plan] add_jobs: created Job 983 (Test for host connectivity)
[2017-06-01 19:08:49,926: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:08:49,941: INFO/job_scheduler] Job 983: RunJobThread.run
[2017-06-01 19:08:49,984: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:49,985: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:49,998: INFO/command_plan] add_jobs: created Job 984 (Test for host connectivity)
[2017-06-01 19:08:50,068: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:08:50,099: INFO/job_scheduler] Job 984: RunJobThread.run
[2017-06-01 19:08:50,167: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:50,167: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:50,184: INFO/command_plan] add_jobs: created Job 985 (Test for host connectivity)
[2017-06-01 19:08:50,274: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:08:50,287: INFO/job_scheduler] Job 985: RunJobThread.run
[2017-06-01 19:08:50,452: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:50,454: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:50,483: INFO/command_plan] add_jobs: created Job 986 (Test for host connectivity)
[2017-06-01 19:08:50,530: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:08:50,601: INFO/job_scheduler] Job 986: RunJobThread.run
[2017-06-01 19:08:52,216: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:08:52,443: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:08:52,600: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:08:52,766: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:08:53,545: INFO/job_scheduler] Job 983 finished 1 steps successfully
[2017-06-01 19:08:53,639: INFO/job_scheduler] Job 983: complete_job: Updating cache
[2017-06-01 19:08:53,639: INFO/job_scheduler] Job 983 completing (errored=False, cancelled=False)
[2017-06-01 19:08:53,707: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:08:53,759: INFO/job_scheduler] Job 984 finished 1 steps successfully
[2017-06-01 19:08:53,851: INFO/job_scheduler] Job 984: complete_job: Updating cache
[2017-06-01 19:08:53,851: INFO/job_scheduler] Job 984 completing (errored=False, cancelled=False)
[2017-06-01 19:08:53,989: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:08:54,028: INFO/job_scheduler] Job 985 finished 1 steps successfully
[2017-06-01 19:08:54,059: INFO/job_scheduler] Job 985: complete_job: Updating cache
[2017-06-01 19:08:54,059: INFO/job_scheduler] Job 985 completing (errored=False, cancelled=False)
[2017-06-01 19:08:54,108: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:08:54,180: INFO/job_scheduler] Job 986 finished 1 steps successfully
[2017-06-01 19:08:54,222: INFO/job_scheduler] Job 986: complete_job: Updating cache
[2017-06-01 19:08:54,222: INFO/job_scheduler] Job 986 completing (errored=False, cancelled=False)
[2017-06-01 19:08:54,297: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:08:56,218: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:56,218: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:56,239: INFO/command_plan] add_jobs: created Job 987 (Test for host connectivity)
[2017-06-01 19:08:56,268: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:08:56,277: INFO/job_scheduler] Job 987: RunJobThread.run
[2017-06-01 19:08:56,307: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:56,307: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:56,425: INFO/command_plan] add_jobs: created Job 988 (Test for host connectivity)
[2017-06-01 19:08:56,513: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:56,513: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:56,524: INFO/command_plan] add_jobs: created Job 989 (Test for host connectivity)
[2017-06-01 19:08:56,552: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:08:56,565: INFO/job_scheduler] Job 988: RunJobThread.run
[2017-06-01 19:08:56,596: INFO/job_scheduler] Job 989: RunJobThread.run
[2017-06-01 19:08:56,629: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:08:56,629: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:08:56,641: INFO/command_plan] add_jobs: created Job 990 (Test for host connectivity)
[2017-06-01 19:08:56,676: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:08:56,696: INFO/job_scheduler] Job 990: RunJobThread.run
[2017-06-01 19:08:56,778: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:08:58,694: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:08:58,905: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:08:58,987: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:08:59,014: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:09:00,087: INFO/job_scheduler] Job 987 finished 1 steps successfully
[2017-06-01 19:09:00,104: INFO/job_scheduler] Job 987: complete_job: Updating cache
[2017-06-01 19:09:00,104: INFO/job_scheduler] Job 987 completing (errored=False, cancelled=False)
[2017-06-01 19:09:00,168: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:09:00,334: INFO/job_scheduler] Job 989 finished 1 steps successfully
[2017-06-01 19:09:00,387: INFO/job_scheduler] Job 989: complete_job: Updating cache
[2017-06-01 19:09:00,388: INFO/job_scheduler] Job 989 completing (errored=False, cancelled=False)
[2017-06-01 19:09:00,436: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:09:00,464: INFO/job_scheduler] Job 988 finished 1 steps successfully
[2017-06-01 19:09:00,500: INFO/job_scheduler] Job 988: complete_job: Updating cache
[2017-06-01 19:09:00,501: INFO/job_scheduler] Job 988 completing (errored=False, cancelled=False)
[2017-06-01 19:09:00,587: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:09:00,592: INFO/job_scheduler] Job 990 finished 1 steps successfully
[2017-06-01 19:09:00,629: INFO/job_scheduler] Job 990: complete_job: Updating cache
[2017-06-01 19:09:00,629: INFO/job_scheduler] Job 990 completing (errored=False, cancelled=False)
[2017-06-01 19:09:00,722: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:09:02,420: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:02,420: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:02,479: INFO/command_plan] add_jobs: created Job 991 (Test for host connectivity)
[2017-06-01 19:09:02,502: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:09:02,511: INFO/job_scheduler] Job 991: RunJobThread.run
[2017-06-01 19:09:02,566: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:02,567: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:02,596: INFO/command_plan] add_jobs: created Job 992 (Test for host connectivity)
[2017-06-01 19:09:02,717: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:02,720: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:02,742: INFO/command_plan] add_jobs: created Job 993 (Test for host connectivity)
[2017-06-01 19:09:02,755: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:09:02,772: INFO/job_scheduler] Job 992: RunJobThread.run
[2017-06-01 19:09:02,802: INFO/job_scheduler] Job 993: RunJobThread.run
[2017-06-01 19:09:02,844: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:02,844: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:02,943: INFO/command_plan] add_jobs: created Job 994 (Test for host connectivity)
[2017-06-01 19:09:03,008: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:09:03,021: INFO/job_scheduler] Job 994: RunJobThread.run
[2017-06-01 19:09:03,073: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:09:04,926: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:09:05,265: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:09:05,281: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:09:05,384: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:09:06,271: INFO/job_scheduler] Job 991 finished 1 steps successfully
[2017-06-01 19:09:06,345: INFO/job_scheduler] Job 991: complete_job: Updating cache
[2017-06-01 19:09:06,345: INFO/job_scheduler] Job 991 completing (errored=False, cancelled=False)
[2017-06-01 19:09:06,420: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:09:06,651: INFO/job_scheduler] Job 992 finished 1 steps successfully
[2017-06-01 19:09:06,678: INFO/job_scheduler] Job 992: complete_job: Updating cache
[2017-06-01 19:09:06,678: INFO/job_scheduler] Job 992 completing (errored=False, cancelled=False)
[2017-06-01 19:09:06,719: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:09:06,742: INFO/job_scheduler] Job 993 finished 1 steps successfully
[2017-06-01 19:09:06,754: INFO/job_scheduler] Job 994 finished 1 steps successfully
[2017-06-01 19:09:06,763: INFO/job_scheduler] Job 993: complete_job: Updating cache
[2017-06-01 19:09:06,763: INFO/job_scheduler] Job 993 completing (errored=False, cancelled=False)
[2017-06-01 19:09:06,837: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:09:06,859: INFO/job_scheduler] Job 994: complete_job: Updating cache
[2017-06-01 19:09:06,859: INFO/job_scheduler] Job 994 completing (errored=False, cancelled=False)
[2017-06-01 19:09:06,937: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:09:08,836: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:08,836: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:08,862: INFO/command_plan] add_jobs: created Job 995 (Test for host connectivity)
[2017-06-01 19:09:08,912: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:09:08,916: INFO/job_scheduler] Job 995: RunJobThread.run
[2017-06-01 19:09:08,971: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:08,972: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:09,004: INFO/command_plan] add_jobs: created Job 996 (Test for host connectivity)
[2017-06-01 19:09:09,070: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:09:09,102: INFO/job_scheduler] Job 996: RunJobThread.run
[2017-06-01 19:09:09,129: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:09,138: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:09,153: INFO/command_plan] add_jobs: created Job 997 (Test for host connectivity)
[2017-06-01 19:09:09,304: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:09,306: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:09,319: INFO/command_plan] add_jobs: created Job 998 (Test for host connectivity)
[2017-06-01 19:09:09,345: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:09:09,382: INFO/job_scheduler] Job 997: RunJobThread.run
[2017-06-01 19:09:09,422: INFO/job_scheduler] Job 998: RunJobThread.run
[2017-06-01 19:09:09,429: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:09:11,312: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:09:11,328: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:09:11,561: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:09:11,707: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:09:12,663: INFO/job_scheduler] Job 995 finished 1 steps successfully
[2017-06-01 19:09:12,694: INFO/job_scheduler] Job 996 finished 1 steps successfully
[2017-06-01 19:09:12,710: INFO/job_scheduler] Job 995: complete_job: Updating cache
[2017-06-01 19:09:12,710: INFO/job_scheduler] Job 995 completing (errored=False, cancelled=False)
[2017-06-01 19:09:12,781: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:09:12,795: INFO/job_scheduler] Job 996: complete_job: Updating cache
[2017-06-01 19:09:12,796: INFO/job_scheduler] Job 996 completing (errored=False, cancelled=False)
[2017-06-01 19:09:12,836: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:09:12,924: INFO/job_scheduler] Job 998 finished 1 steps successfully
[2017-06-01 19:09:12,971: INFO/job_scheduler] Job 998: complete_job: Updating cache
[2017-06-01 19:09:12,992: INFO/job_scheduler] Job 998 completing (errored=False, cancelled=False)
[2017-06-01 19:09:13,058: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:09:13,189: INFO/job_scheduler] Job 997 finished 1 steps successfully
[2017-06-01 19:09:13,243: INFO/job_scheduler] Job 997: complete_job: Updating cache
[2017-06-01 19:09:13,243: INFO/job_scheduler] Job 997 completing (errored=False, cancelled=False)
[2017-06-01 19:09:13,285: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:09:15,035: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:15,036: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:15,068: INFO/command_plan] add_jobs: created Job 999 (Test for host connectivity)
[2017-06-01 19:09:15,110: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:09:15,121: INFO/job_scheduler] Job 999: RunJobThread.run
[2017-06-01 19:09:15,170: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:15,170: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:15,199: INFO/command_plan] add_jobs: created Job 1000 (Test for host connectivity)
[2017-06-01 19:09:15,220: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:09:15,253: INFO/job_scheduler] Job 1000: RunJobThread.run
[2017-06-01 19:09:15,339: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:15,339: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:15,369: INFO/command_plan] add_jobs: created Job 1001 (Test for host connectivity)
[2017-06-01 19:09:15,422: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:09:15,442: INFO/job_scheduler] Job 1001: RunJobThread.run
[2017-06-01 19:09:15,496: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:15,496: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:15,521: INFO/command_plan] add_jobs: created Job 1002 (Test for host connectivity)
[2017-06-01 19:09:15,570: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:09:15,606: INFO/job_scheduler] Job 1002: RunJobThread.run
[2017-06-01 19:09:17,415: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:09:17,648: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:09:17,805: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:09:17,885: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:09:18,744: INFO/job_scheduler] Job 999 finished 1 steps successfully
[2017-06-01 19:09:18,801: INFO/job_scheduler] Job 999: complete_job: Updating cache
[2017-06-01 19:09:18,801: INFO/job_scheduler] Job 999 completing (errored=False, cancelled=False)
[2017-06-01 19:09:18,847: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:09:19,018: INFO/job_scheduler] Job 1000 finished 1 steps successfully
[2017-06-01 19:09:19,076: INFO/job_scheduler] Job 1000: complete_job: Updating cache
[2017-06-01 19:09:19,076: INFO/job_scheduler] Job 1000 completing (errored=False, cancelled=False)
[2017-06-01 19:09:19,136: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:09:19,267: INFO/job_scheduler] Job 1002 finished 1 steps successfully
[2017-06-01 19:09:19,276: INFO/job_scheduler] Job 1001 finished 1 steps successfully
[2017-06-01 19:09:19,334: INFO/job_scheduler] Job 1002: complete_job: Updating cache
[2017-06-01 19:09:19,334: INFO/job_scheduler] Job 1002 completing (errored=False, cancelled=False)
[2017-06-01 19:09:19,377: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:09:19,392: INFO/job_scheduler] Job 1001: complete_job: Updating cache
[2017-06-01 19:09:19,394: INFO/job_scheduler] Job 1001 completing (errored=False, cancelled=False)
[2017-06-01 19:09:19,462: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:09:21,335: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:21,335: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:21,383: INFO/command_plan] add_jobs: created Job 1003 (Test for host connectivity)
[2017-06-01 19:09:21,437: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:09:21,446: INFO/job_scheduler] Job 1003: RunJobThread.run
[2017-06-01 19:09:21,587: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:21,587: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:21,621: INFO/command_plan] add_jobs: created Job 1004 (Test for host connectivity)
[2017-06-01 19:09:21,656: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:09:21,671: INFO/job_scheduler] Job 1004: RunJobThread.run
[2017-06-01 19:09:21,744: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:21,747: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:21,763: INFO/command_plan] add_jobs: created Job 1005 (Test for host connectivity)
[2017-06-01 19:09:21,862: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:09:21,867: INFO/job_scheduler] Job 1005: RunJobThread.run
[2017-06-01 19:09:21,923: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:21,923: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:21,947: INFO/command_plan] add_jobs: created Job 1006 (Test for host connectivity)
[2017-06-01 19:09:22,014: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:09:22,059: INFO/job_scheduler] Job 1006: RunJobThread.run
[2017-06-01 19:09:23,743: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:09:23,923: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:09:24,183: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:09:24,226: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:09:25,141: INFO/job_scheduler] Job 1003 finished 1 steps successfully
[2017-06-01 19:09:25,160: INFO/job_scheduler] Job 1003: complete_job: Updating cache
[2017-06-01 19:09:25,160: INFO/job_scheduler] Job 1003 completing (errored=False, cancelled=False)
[2017-06-01 19:09:25,243: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:09:25,253: INFO/job_scheduler] Job 1004 finished 1 steps successfully
[2017-06-01 19:09:25,284: INFO/job_scheduler] Job 1004: complete_job: Updating cache
[2017-06-01 19:09:25,284: INFO/job_scheduler] Job 1004 completing (errored=False, cancelled=False)
[2017-06-01 19:09:25,414: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:09:25,641: INFO/job_scheduler] Job 1006 finished 1 steps successfully
[2017-06-01 19:09:25,677: INFO/job_scheduler] Job 1006: complete_job: Updating cache
[2017-06-01 19:09:25,677: INFO/job_scheduler] Job 1006 completing (errored=False, cancelled=False)
[2017-06-01 19:09:25,711: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:09:25,830: INFO/job_scheduler] Job 1005 finished 1 steps successfully
[2017-06-01 19:09:25,868: INFO/job_scheduler] Job 1005: complete_job: Updating cache
[2017-06-01 19:09:25,868: INFO/job_scheduler] Job 1005 completing (errored=False, cancelled=False)
[2017-06-01 19:09:25,903: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:09:27,681: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:27,682: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:27,693: INFO/command_plan] add_jobs: created Job 1007 (Test for host connectivity)
[2017-06-01 19:09:27,743: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:09:27,781: INFO/job_scheduler] Job 1007: RunJobThread.run
[2017-06-01 19:09:27,817: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:27,818: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:27,859: INFO/command_plan] add_jobs: created Job 1008 (Test for host connectivity)
[2017-06-01 19:09:27,896: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:09:27,932: INFO/job_scheduler] Job 1008: RunJobThread.run
[2017-06-01 19:09:28,093: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:28,093: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:28,119: INFO/command_plan] add_jobs: created Job 1009 (Test for host connectivity)
[2017-06-01 19:09:28,205: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:09:28,207: INFO/job_scheduler] Job 1009: RunJobThread.run
[2017-06-01 19:09:28,252: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:28,252: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:28,269: INFO/command_plan] add_jobs: created Job 1010 (Test for host connectivity)
[2017-06-01 19:09:28,365: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:09:28,407: INFO/job_scheduler] Job 1010: RunJobThread.run
[2017-06-01 19:09:30,010: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:09:30,207: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:09:30,523: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:09:30,583: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:09:31,349: INFO/job_scheduler] Job 1007 finished 1 steps successfully
[2017-06-01 19:09:31,401: INFO/job_scheduler] Job 1007: complete_job: Updating cache
[2017-06-01 19:09:31,401: INFO/job_scheduler] Job 1007 completing (errored=False, cancelled=False)
[2017-06-01 19:09:31,462: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:09:31,540: INFO/job_scheduler] Job 1008 finished 1 steps successfully
[2017-06-01 19:09:31,609: INFO/job_scheduler] Job 1008: complete_job: Updating cache
[2017-06-01 19:09:31,609: INFO/job_scheduler] Job 1008 completing (errored=False, cancelled=False)
[2017-06-01 19:09:31,661: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:09:31,948: INFO/job_scheduler] Job 1009 finished 1 steps successfully
[2017-06-01 19:09:32,009: INFO/job_scheduler] Job 1009: complete_job: Updating cache
[2017-06-01 19:09:32,009: INFO/job_scheduler] Job 1009 completing (errored=False, cancelled=False)
[2017-06-01 19:09:32,132: INFO/job_scheduler] Job 1010 finished 1 steps successfully
[2017-06-01 19:09:32,136: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:09:32,151: INFO/job_scheduler] Job 1010: complete_job: Updating cache
[2017-06-01 19:09:32,152: INFO/job_scheduler] Job 1010 completing (errored=False, cancelled=False)
[2017-06-01 19:09:32,236: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:09:33,614: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:33,614: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:33,625: INFO/command_plan] add_jobs: created Job 1011 (Test for host connectivity)
[2017-06-01 19:09:33,667: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:09:33,679: INFO/job_scheduler] Job 1011: RunJobThread.run
[2017-06-01 19:09:33,731: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:33,738: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:33,858: INFO/command_plan] add_jobs: created Job 1012 (Test for host connectivity)
[2017-06-01 19:09:33,871: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:09:33,875: INFO/job_scheduler] Job 1012: RunJobThread.run
[2017-06-01 19:09:33,922: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:33,922: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:33,967: INFO/command_plan] add_jobs: created Job 1013 (Test for host connectivity)
[2017-06-01 19:09:34,022: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:09:34,065: INFO/job_scheduler] Job 1013: RunJobThread.run
[2017-06-01 19:09:34,128: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:34,131: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:34,142: INFO/command_plan] add_jobs: created Job 1014 (Test for host connectivity)
[2017-06-01 19:09:34,206: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:09:34,212: INFO/job_scheduler] Job 1014: RunJobThread.run
[2017-06-01 19:09:36,098: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:09:36,221: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:09:36,343: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:09:36,439: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:09:37,417: INFO/job_scheduler] Job 1011 finished 1 steps successfully
[2017-06-01 19:09:37,483: INFO/job_scheduler] Job 1011: complete_job: Updating cache
[2017-06-01 19:09:37,483: INFO/job_scheduler] Job 1011 completing (errored=False, cancelled=False)
[2017-06-01 19:09:37,576: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:09:37,634: INFO/job_scheduler] Job 1012 finished 1 steps successfully
[2017-06-01 19:09:37,698: INFO/job_scheduler] Job 1012: complete_job: Updating cache
[2017-06-01 19:09:37,698: INFO/job_scheduler] Job 1012 completing (errored=False, cancelled=False)
[2017-06-01 19:09:37,750: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:09:37,849: INFO/job_scheduler] Job 1013 finished 1 steps successfully
[2017-06-01 19:09:37,866: INFO/job_scheduler] Job 1014 finished 1 steps successfully
[2017-06-01 19:09:37,908: INFO/job_scheduler] Job 1013: complete_job: Updating cache
[2017-06-01 19:09:37,908: INFO/job_scheduler] Job 1013 completing (errored=False, cancelled=False)
[2017-06-01 19:09:37,952: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:09:37,966: INFO/job_scheduler] Job 1014: complete_job: Updating cache
[2017-06-01 19:09:37,966: INFO/job_scheduler] Job 1014 completing (errored=False, cancelled=False)
[2017-06-01 19:09:38,030: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:09:39,983: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:39,984: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:40,032: INFO/command_plan] add_jobs: created Job 1015 (Test for host connectivity)
[2017-06-01 19:09:40,043: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:09:40,064: INFO/job_scheduler] Job 1015: RunJobThread.run
[2017-06-01 19:09:40,098: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:40,099: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:40,123: INFO/command_plan] add_jobs: created Job 1016 (Test for host connectivity)
[2017-06-01 19:09:40,158: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:09:40,187: INFO/job_scheduler] Job 1016: RunJobThread.run
[2017-06-01 19:09:40,256: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:40,257: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:40,282: INFO/command_plan] add_jobs: created Job 1017 (Test for host connectivity)
[2017-06-01 19:09:40,367: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:09:40,385: INFO/job_scheduler] Job 1017: RunJobThread.run
[2017-06-01 19:09:40,451: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:40,451: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:40,560: INFO/command_plan] add_jobs: created Job 1018 (Test for host connectivity)
[2017-06-01 19:09:40,611: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:09:40,616: INFO/job_scheduler] Job 1018: RunJobThread.run
[2017-06-01 19:09:42,435: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:09:42,543: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:09:42,776: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:09:42,923: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:09:43,769: INFO/job_scheduler] Job 1016 finished 1 steps successfully
[2017-06-01 19:09:43,831: INFO/job_scheduler] Job 1016: complete_job: Updating cache
[2017-06-01 19:09:43,831: INFO/job_scheduler] Job 1016 completing (errored=False, cancelled=False)
[2017-06-01 19:09:43,902: INFO/job_scheduler] Job 1015 finished 1 steps successfully
[2017-06-01 19:09:43,915: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:09:43,938: INFO/job_scheduler] Job 1015: complete_job: Updating cache
[2017-06-01 19:09:43,942: INFO/job_scheduler] Job 1015 completing (errored=False, cancelled=False)
[2017-06-01 19:09:44,006: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:09:44,160: INFO/job_scheduler] Job 1017 finished 1 steps successfully
[2017-06-01 19:09:44,231: INFO/job_scheduler] Job 1017: complete_job: Updating cache
[2017-06-01 19:09:44,231: INFO/job_scheduler] Job 1017 completing (errored=False, cancelled=False)
[2017-06-01 19:09:44,274: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:09:44,342: INFO/job_scheduler] Job 1018 finished 1 steps successfully
[2017-06-01 19:09:44,366: INFO/job_scheduler] Job 1018: complete_job: Updating cache
[2017-06-01 19:09:44,366: INFO/job_scheduler] Job 1018 completing (errored=False, cancelled=False)
[2017-06-01 19:09:44,424: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:09:46,252: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:46,252: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:46,269: INFO/command_plan] add_jobs: created Job 1019 (Test for host connectivity)
[2017-06-01 19:09:46,315: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:09:46,333: INFO/job_scheduler] Job 1019: RunJobThread.run
[2017-06-01 19:09:46,379: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:46,379: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:46,401: INFO/command_plan] add_jobs: created Job 1020 (Test for host connectivity)
[2017-06-01 19:09:46,449: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:09:46,473: INFO/job_scheduler] Job 1020: RunJobThread.run
[2017-06-01 19:09:46,572: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:46,573: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:46,584: INFO/command_plan] add_jobs: created Job 1021 (Test for host connectivity)
[2017-06-01 19:09:46,632: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:09:46,684: INFO/job_scheduler] Job 1021: RunJobThread.run
[2017-06-01 19:09:46,754: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:46,756: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:46,780: INFO/command_plan] add_jobs: created Job 1022 (Test for host connectivity)
[2017-06-01 19:09:46,859: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:09:46,907: INFO/job_scheduler] Job 1022: RunJobThread.run
[2017-06-01 19:09:48,645: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:09:48,794: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:09:48,942: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:09:49,135: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:09:49,998: INFO/job_scheduler] Job 1019 finished 1 steps successfully
[2017-06-01 19:09:50,022: INFO/job_scheduler] Job 1019: complete_job: Updating cache
[2017-06-01 19:09:50,022: INFO/job_scheduler] Job 1019 completing (errored=False, cancelled=False)
[2017-06-01 19:09:50,063: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:09:50,184: INFO/job_scheduler] Job 1020 finished 1 steps successfully
[2017-06-01 19:09:50,230: INFO/job_scheduler] Job 1020: complete_job: Updating cache
[2017-06-01 19:09:50,230: INFO/job_scheduler] Job 1020 completing (errored=False, cancelled=False)
[2017-06-01 19:09:50,291: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:09:50,320: INFO/job_scheduler] Job 1021 finished 1 steps successfully
[2017-06-01 19:09:50,362: INFO/job_scheduler] Job 1021: complete_job: Updating cache
[2017-06-01 19:09:50,362: INFO/job_scheduler] Job 1021 completing (errored=False, cancelled=False)
[2017-06-01 19:09:50,424: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:09:50,640: INFO/job_scheduler] Job 1022 finished 1 steps successfully
[2017-06-01 19:09:50,680: INFO/job_scheduler] Job 1022: complete_job: Updating cache
[2017-06-01 19:09:50,680: INFO/job_scheduler] Job 1022 completing (errored=False, cancelled=False)
[2017-06-01 19:09:50,722: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:09:52,542: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:52,542: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:52,556: INFO/command_plan] add_jobs: created Job 1023 (Test for host connectivity)
[2017-06-01 19:09:52,617: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:52,617: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:52,633: INFO/command_plan] add_jobs: created Job 1024 (Test for host connectivity)
[2017-06-01 19:09:52,651: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:09:52,667: INFO/job_scheduler] Job 1024: RunJobThread.run
[2017-06-01 19:09:52,672: INFO/job_scheduler] Job 1023: RunJobThread.run
[2017-06-01 19:09:52,691: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:09:52,815: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:52,815: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:52,841: INFO/command_plan] add_jobs: created Job 1025 (Test for host connectivity)
[2017-06-01 19:09:52,871: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:09:52,923: INFO/job_scheduler] Job 1025: RunJobThread.run
[2017-06-01 19:09:53,012: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:53,013: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:53,056: INFO/command_plan] add_jobs: created Job 1026 (Test for host connectivity)
[2017-06-01 19:09:53,079: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:09:53,136: INFO/job_scheduler] Job 1026: RunJobThread.run
[2017-06-01 19:09:55,022: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:09:55,023: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:09:55,221: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:09:55,271: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:09:56,372: INFO/job_scheduler] Job 1023 finished 1 steps successfully
[2017-06-01 19:09:56,388: INFO/job_scheduler] Job 1023: complete_job: Updating cache
[2017-06-01 19:09:56,388: INFO/job_scheduler] Job 1023 completing (errored=False, cancelled=False)
[2017-06-01 19:09:56,425: INFO/job_scheduler] Job 1024 finished 1 steps successfully
[2017-06-01 19:09:56,486: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:09:56,515: INFO/job_scheduler] Job 1024: complete_job: Updating cache
[2017-06-01 19:09:56,515: INFO/job_scheduler] Job 1024 completing (errored=False, cancelled=False)
[2017-06-01 19:09:56,590: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:09:56,630: INFO/job_scheduler] Job 1025 finished 1 steps successfully
[2017-06-01 19:09:56,650: INFO/job_scheduler] Job 1025: complete_job: Updating cache
[2017-06-01 19:09:56,650: INFO/job_scheduler] Job 1025 completing (errored=False, cancelled=False)
[2017-06-01 19:09:56,778: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:09:56,789: INFO/job_scheduler] Job 1026 finished 1 steps successfully
[2017-06-01 19:09:56,873: INFO/job_scheduler] Job 1026: complete_job: Updating cache
[2017-06-01 19:09:56,873: INFO/job_scheduler] Job 1026 completing (errored=False, cancelled=False)
[2017-06-01 19:09:56,921: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:09:58,800: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:58,802: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:58,903: INFO/command_plan] add_jobs: created Job 1027 (Test for host connectivity)
[2017-06-01 19:09:58,939: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:09:58,942: INFO/job_scheduler] Job 1027: RunJobThread.run
[2017-06-01 19:09:58,990: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:58,990: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:59,014: INFO/command_plan] add_jobs: created Job 1028 (Test for host connectivity)
[2017-06-01 19:09:59,077: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:09:59,095: INFO/job_scheduler] Job 1028: RunJobThread.run
[2017-06-01 19:09:59,152: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:59,162: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:59,288: INFO/command_plan] add_jobs: created Job 1029 (Test for host connectivity)
[2017-06-01 19:09:59,359: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:09:59,361: INFO/job_scheduler] Job 1029: RunJobThread.run
[2017-06-01 19:09:59,438: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:09:59,440: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:09:59,461: INFO/command_plan] add_jobs: created Job 1030 (Test for host connectivity)
[2017-06-01 19:09:59,513: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:09:59,537: INFO/job_scheduler] Job 1030: RunJobThread.run
[2017-06-01 19:10:01,335: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:10:01,430: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:10:01,752: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:10:01,852: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:10:02,699: INFO/job_scheduler] Job 1027 finished 1 steps successfully
[2017-06-01 19:10:02,712: INFO/job_scheduler] Job 1027: complete_job: Updating cache
[2017-06-01 19:10:02,713: INFO/job_scheduler] Job 1027 completing (errored=False, cancelled=False)
[2017-06-01 19:10:02,764: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:10:02,765: INFO/job_scheduler] Job 1028 finished 1 steps successfully
[2017-06-01 19:10:02,800: INFO/job_scheduler] Job 1028: complete_job: Updating cache
[2017-06-01 19:10:02,800: INFO/job_scheduler] Job 1028 completing (errored=False, cancelled=False)
[2017-06-01 19:10:02,917: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:10:03,197: INFO/job_scheduler] Job 1029 finished 1 steps successfully
[2017-06-01 19:10:03,220: INFO/job_scheduler] Job 1029: complete_job: Updating cache
[2017-06-01 19:10:03,220: INFO/job_scheduler] Job 1029 completing (errored=False, cancelled=False)
[2017-06-01 19:10:03,253: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:10:03,343: INFO/job_scheduler] Job 1030 finished 1 steps successfully
[2017-06-01 19:10:03,403: INFO/job_scheduler] Job 1030: complete_job: Updating cache
[2017-06-01 19:10:03,403: INFO/job_scheduler] Job 1030 completing (errored=False, cancelled=False)
[2017-06-01 19:10:03,453: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:10:05,170: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:05,171: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:05,193: INFO/command_plan] add_jobs: created Job 1031 (Test for host connectivity)
[2017-06-01 19:10:05,222: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:10:05,241: INFO/job_scheduler] Job 1031: RunJobThread.run
[2017-06-01 19:10:05,289: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:05,289: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:05,303: INFO/command_plan] add_jobs: created Job 1032 (Test for host connectivity)
[2017-06-01 19:10:05,377: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:10:05,384: INFO/job_scheduler] Job 1032: RunJobThread.run
[2017-06-01 19:10:05,430: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:05,430: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:05,449: INFO/command_plan] add_jobs: created Job 1033 (Test for host connectivity)
[2017-06-01 19:10:05,506: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:10:05,521: INFO/job_scheduler] Job 1033: RunJobThread.run
[2017-06-01 19:10:05,607: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:05,608: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:05,629: INFO/command_plan] add_jobs: created Job 1034 (Test for host connectivity)
[2017-06-01 19:10:05,710: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:10:05,736: INFO/job_scheduler] Job 1034: RunJobThread.run
[2017-06-01 19:10:07,533: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:10:07,693: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:10:07,778: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:10:07,847: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:10:08,842: INFO/job_scheduler] Job 1031 finished 1 steps successfully
[2017-06-01 19:10:08,869: INFO/job_scheduler] Job 1031: complete_job: Updating cache
[2017-06-01 19:10:08,869: INFO/job_scheduler] Job 1031 completing (errored=False, cancelled=False)
[2017-06-01 19:10:08,946: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:10:09,013: INFO/job_scheduler] Job 1032 finished 1 steps successfully
[2017-06-01 19:10:09,069: INFO/job_scheduler] Job 1032: complete_job: Updating cache
[2017-06-01 19:10:09,069: INFO/job_scheduler] Job 1032 completing (errored=False, cancelled=False)
[2017-06-01 19:10:09,095: INFO/job_scheduler] Job 1034 finished 1 steps successfully
[2017-06-01 19:10:09,144: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:10:09,162: INFO/job_scheduler] Job 1034: complete_job: Updating cache
[2017-06-01 19:10:09,162: INFO/job_scheduler] Job 1034 completing (errored=False, cancelled=False)
[2017-06-01 19:10:09,199: INFO/job_scheduler] Job 1033 finished 1 steps successfully
[2017-06-01 19:10:09,240: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:10:09,262: INFO/job_scheduler] Job 1033: complete_job: Updating cache
[2017-06-01 19:10:09,262: INFO/job_scheduler] Job 1033 completing (errored=False, cancelled=False)
[2017-06-01 19:10:09,336: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:10:11,483: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:11,484: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:11,498: INFO/command_plan] add_jobs: created Job 1035 (Test for host connectivity)
[2017-06-01 19:10:11,556: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:11,556: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:11,587: INFO/command_plan] add_jobs: created Job 1036 (Test for host connectivity)
[2017-06-01 19:10:11,638: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:10:11,653: INFO/job_scheduler] Job 1035: RunJobThread.run
[2017-06-01 19:10:11,657: INFO/job_scheduler] Job 1036: RunJobThread.run
[2017-06-01 19:10:11,686: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:11,686: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:11,769: INFO/command_plan] add_jobs: created Job 1037 (Test for host connectivity)
[2017-06-01 19:10:11,797: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:10:11,837: INFO/job_scheduler] Job 1037: RunJobThread.run
[2017-06-01 19:10:11,940: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:11,948: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:12,049: INFO/command_plan] add_jobs: created Job 1038 (Test for host connectivity)
[2017-06-01 19:10:12,062: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:10:12,106: INFO/job_scheduler] Job 1038: RunJobThread.run
[2017-06-01 19:10:12,144: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:10:14,000: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:10:14,081: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:10:14,164: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:10:14,313: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:10:15,429: INFO/job_scheduler] Job 1035 finished 1 steps successfully
[2017-06-01 19:10:15,459: INFO/job_scheduler] Job 1036 finished 1 steps successfully
[2017-06-01 19:10:15,468: INFO/job_scheduler] Job 1035: complete_job: Updating cache
[2017-06-01 19:10:15,468: INFO/job_scheduler] Job 1035 completing (errored=False, cancelled=False)
[2017-06-01 19:10:15,502: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:10:15,528: INFO/job_scheduler] Job 1036: complete_job: Updating cache
[2017-06-01 19:10:15,528: INFO/job_scheduler] Job 1036 completing (errored=False, cancelled=False)
[2017-06-01 19:10:15,596: INFO/job_scheduler] Job 1037 finished 1 steps successfully
[2017-06-01 19:10:15,606: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:10:15,651: INFO/job_scheduler] Job 1037: complete_job: Updating cache
[2017-06-01 19:10:15,652: INFO/job_scheduler] Job 1037 completing (errored=False, cancelled=False)
[2017-06-01 19:10:15,678: INFO/job_scheduler] Job 1038 finished 1 steps successfully
[2017-06-01 19:10:15,753: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:10:15,782: INFO/job_scheduler] Job 1038: complete_job: Updating cache
[2017-06-01 19:10:15,782: INFO/job_scheduler] Job 1038 completing (errored=False, cancelled=False)
[2017-06-01 19:10:15,894: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:10:17,725: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:17,725: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:17,765: INFO/command_plan] add_jobs: created Job 1039 (Test for host connectivity)
[2017-06-01 19:10:17,815: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:10:17,826: INFO/job_scheduler] Job 1039: RunJobThread.run
[2017-06-01 19:10:17,867: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:17,867: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:17,876: INFO/command_plan] add_jobs: created Job 1040 (Test for host connectivity)
[2017-06-01 19:10:17,926: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:10:17,957: INFO/job_scheduler] Job 1040: RunJobThread.run
[2017-06-01 19:10:18,026: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:18,028: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:18,052: INFO/command_plan] add_jobs: created Job 1041 (Test for host connectivity)
[2017-06-01 19:10:18,131: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:10:18,146: INFO/job_scheduler] Job 1041: RunJobThread.run
[2017-06-01 19:10:18,178: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:18,183: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:18,293: INFO/command_plan] add_jobs: created Job 1042 (Test for host connectivity)
[2017-06-01 19:10:18,317: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:10:18,321: INFO/job_scheduler] Job 1042: RunJobThread.run
[2017-06-01 19:10:20,071: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:10:20,234: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:10:20,430: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:10:20,601: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:10:21,422: INFO/job_scheduler] Job 1039 finished 1 steps successfully
[2017-06-01 19:10:21,492: INFO/job_scheduler] Job 1039: complete_job: Updating cache
[2017-06-01 19:10:21,492: INFO/job_scheduler] Job 1039 completing (errored=False, cancelled=False)
[2017-06-01 19:10:21,583: INFO/job_scheduler] Job 1040 finished 1 steps successfully
[2017-06-01 19:10:21,603: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:10:21,663: INFO/job_scheduler] Job 1040: complete_job: Updating cache
[2017-06-01 19:10:21,663: INFO/job_scheduler] Job 1040 completing (errored=False, cancelled=False)
[2017-06-01 19:10:21,751: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:10:21,839: INFO/job_scheduler] Job 1041 finished 1 steps successfully
[2017-06-01 19:10:21,901: INFO/job_scheduler] Job 1041: complete_job: Updating cache
[2017-06-01 19:10:21,901: INFO/job_scheduler] Job 1041 completing (errored=False, cancelled=False)
[2017-06-01 19:10:21,936: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:10:21,960: INFO/job_scheduler] Job 1042 finished 1 steps successfully
[2017-06-01 19:10:22,069: INFO/job_scheduler] Job 1042: complete_job: Updating cache
[2017-06-01 19:10:22,069: INFO/job_scheduler] Job 1042 completing (errored=False, cancelled=False)
[2017-06-01 19:10:22,127: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:10:23,996: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:23,996: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:24,018: INFO/command_plan] add_jobs: created Job 1043 (Test for host connectivity)
[2017-06-01 19:10:24,054: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:10:24,072: INFO/job_scheduler] Job 1043: RunJobThread.run
[2017-06-01 19:10:24,114: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:24,125: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:24,138: INFO/command_plan] add_jobs: created Job 1044 (Test for host connectivity)
[2017-06-01 19:10:24,226: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:10:24,250: INFO/job_scheduler] Job 1044: RunJobThread.run
[2017-06-01 19:10:24,321: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:24,321: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:24,340: INFO/command_plan] add_jobs: created Job 1045 (Test for host connectivity)
[2017-06-01 19:10:24,444: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:10:24,446: INFO/job_scheduler] Job 1045: RunJobThread.run
[2017-06-01 19:10:24,497: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:24,499: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:24,511: INFO/command_plan] add_jobs: created Job 1046 (Test for host connectivity)
[2017-06-01 19:10:24,556: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:10:24,565: INFO/job_scheduler] Job 1046: RunJobThread.run
[2017-06-01 19:10:26,435: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:10:26,594: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:10:26,787: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:10:26,855: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:10:27,747: INFO/job_scheduler] Job 1043 finished 1 steps successfully
[2017-06-01 19:10:27,808: INFO/job_scheduler] Job 1043: complete_job: Updating cache
[2017-06-01 19:10:27,809: INFO/job_scheduler] Job 1043 completing (errored=False, cancelled=False)
[2017-06-01 19:10:27,896: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:10:27,982: INFO/job_scheduler] Job 1044 finished 1 steps successfully
[2017-06-01 19:10:28,028: INFO/job_scheduler] Job 1044: complete_job: Updating cache
[2017-06-01 19:10:28,028: INFO/job_scheduler] Job 1044 completing (errored=False, cancelled=False)
[2017-06-01 19:10:28,084: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:10:28,139: INFO/job_scheduler] Job 1045 finished 1 steps successfully
[2017-06-01 19:10:28,158: INFO/job_scheduler] Job 1045: complete_job: Updating cache
[2017-06-01 19:10:28,158: INFO/job_scheduler] Job 1045 completing (errored=False, cancelled=False)
[2017-06-01 19:10:28,226: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:10:28,421: INFO/job_scheduler] Job 1046 finished 1 steps successfully
[2017-06-01 19:10:28,467: INFO/job_scheduler] Job 1046: complete_job: Updating cache
[2017-06-01 19:10:28,467: INFO/job_scheduler] Job 1046 completing (errored=False, cancelled=False)
[2017-06-01 19:10:28,525: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:10:30,236: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:30,237: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:30,261: INFO/command_plan] add_jobs: created Job 1047 (Test for host connectivity)
[2017-06-01 19:10:30,304: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:10:30,312: INFO/job_scheduler] Job 1047: RunJobThread.run
[2017-06-01 19:10:30,342: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:30,343: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:30,387: INFO/command_plan] add_jobs: created Job 1048 (Test for host connectivity)
[2017-06-01 19:10:30,452: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:10:30,457: INFO/job_scheduler] Job 1048: RunJobThread.run
[2017-06-01 19:10:30,513: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:30,513: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:30,542: INFO/command_plan] add_jobs: created Job 1049 (Test for host connectivity)
[2017-06-01 19:10:30,669: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:30,669: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:30,711: INFO/command_plan] add_jobs: created Job 1050 (Test for host connectivity)
[2017-06-01 19:10:30,750: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:10:30,774: INFO/job_scheduler] Job 1049: RunJobThread.run
[2017-06-01 19:10:30,775: INFO/job_scheduler] Job 1050: RunJobThread.run
[2017-06-01 19:10:30,785: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:10:32,645: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:10:32,779: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:10:33,107: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:10:33,165: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:10:33,940: INFO/job_scheduler] Job 1047 finished 1 steps successfully
[2017-06-01 19:10:33,975: INFO/job_scheduler] Job 1047: complete_job: Updating cache
[2017-06-01 19:10:33,975: INFO/job_scheduler] Job 1047 completing (errored=False, cancelled=False)
[2017-06-01 19:10:34,011: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:10:34,114: INFO/job_scheduler] Job 1048 finished 1 steps successfully
[2017-06-01 19:10:34,191: INFO/job_scheduler] Job 1048: complete_job: Updating cache
[2017-06-01 19:10:34,192: INFO/job_scheduler] Job 1048 completing (errored=False, cancelled=False)
[2017-06-01 19:10:34,292: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:10:34,540: INFO/job_scheduler] Job 1049 finished 1 steps successfully
[2017-06-01 19:10:34,566: INFO/job_scheduler] Job 1049: complete_job: Updating cache
[2017-06-01 19:10:34,566: INFO/job_scheduler] Job 1049 completing (errored=False, cancelled=False)
[2017-06-01 19:10:34,600: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:10:34,643: INFO/job_scheduler] Job 1050 finished 1 steps successfully
[2017-06-01 19:10:34,683: INFO/job_scheduler] Job 1050: complete_job: Updating cache
[2017-06-01 19:10:34,683: INFO/job_scheduler] Job 1050 completing (errored=False, cancelled=False)
[2017-06-01 19:10:34,743: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:10:36,470: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:36,471: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:36,509: INFO/command_plan] add_jobs: created Job 1051 (Test for host connectivity)
[2017-06-01 19:10:36,563: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:36,564: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:36,599: INFO/command_plan] add_jobs: created Job 1052 (Test for host connectivity)
[2017-06-01 19:10:36,613: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:10:36,640: INFO/job_scheduler] Job 1051: RunJobThread.run
[2017-06-01 19:10:36,649: INFO/job_scheduler] Job 1052: RunJobThread.run
[2017-06-01 19:10:36,685: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:36,685: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:36,724: INFO/command_plan] add_jobs: created Job 1053 (Test for host connectivity)
[2017-06-01 19:10:36,761: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:10:36,791: INFO/job_scheduler] Job 1053: RunJobThread.run
[2017-06-01 19:10:36,915: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:36,917: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:36,944: INFO/command_plan] add_jobs: created Job 1054 (Test for host connectivity)
[2017-06-01 19:10:36,997: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:10:36,999: INFO/job_scheduler] Job 1054: RunJobThread.run
[2017-06-01 19:10:37,067: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:10:38,911: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:10:38,960: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:10:39,161: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:10:39,161: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:10:40,266: INFO/job_scheduler] Job 1051 finished 1 steps successfully
[2017-06-01 19:10:40,299: INFO/job_scheduler] Job 1051: complete_job: Updating cache
[2017-06-01 19:10:40,299: INFO/job_scheduler] Job 1051 completing (errored=False, cancelled=False)
[2017-06-01 19:10:40,323: INFO/job_scheduler] Job 1052 finished 1 steps successfully
[2017-06-01 19:10:40,350: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:10:40,376: INFO/job_scheduler] Job 1052: complete_job: Updating cache
[2017-06-01 19:10:40,376: INFO/job_scheduler] Job 1052 completing (errored=False, cancelled=False)
[2017-06-01 19:10:40,460: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:10:40,568: INFO/job_scheduler] Job 1054 finished 1 steps successfully
[2017-06-01 19:10:40,575: INFO/job_scheduler] Job 1053 finished 1 steps successfully
[2017-06-01 19:10:40,624: INFO/job_scheduler] Job 1054: complete_job: Updating cache
[2017-06-01 19:10:40,624: INFO/job_scheduler] Job 1054 completing (errored=False, cancelled=False)
[2017-06-01 19:10:40,658: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:10:40,782: INFO/job_scheduler] Job 1053: complete_job: Updating cache
[2017-06-01 19:10:40,782: INFO/job_scheduler] Job 1053 completing (errored=False, cancelled=False)
[2017-06-01 19:10:40,827: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:10:42,727: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:42,727: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:42,770: INFO/command_plan] add_jobs: created Job 1055 (Test for host connectivity)
[2017-06-01 19:10:42,800: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:10:42,803: INFO/job_scheduler] Job 1055: RunJobThread.run
[2017-06-01 19:10:42,836: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:42,837: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:42,850: INFO/command_plan] add_jobs: created Job 1056 (Test for host connectivity)
[2017-06-01 19:10:42,933: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:10:42,974: INFO/job_scheduler] Job 1056: RunJobThread.run
[2017-06-01 19:10:43,014: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:43,021: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:43,042: INFO/command_plan] add_jobs: created Job 1057 (Test for host connectivity)
[2017-06-01 19:10:43,125: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:10:43,163: INFO/job_scheduler] Job 1057: RunJobThread.run
[2017-06-01 19:10:43,224: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:43,224: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:43,240: INFO/command_plan] add_jobs: created Job 1058 (Test for host connectivity)
[2017-06-01 19:10:43,355: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:10:43,370: INFO/job_scheduler] Job 1058: RunJobThread.run
[2017-06-01 19:10:45,193: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:10:45,344: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:10:45,490: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:10:45,576: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:10:46,523: INFO/job_scheduler] Job 1055 finished 1 steps successfully
[2017-06-01 19:10:46,540: INFO/job_scheduler] Job 1055: complete_job: Updating cache
[2017-06-01 19:10:46,540: INFO/job_scheduler] Job 1055 completing (errored=False, cancelled=False)
[2017-06-01 19:10:46,591: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:10:46,752: INFO/job_scheduler] Job 1056 finished 1 steps successfully
[2017-06-01 19:10:46,807: INFO/job_scheduler] Job 1056: complete_job: Updating cache
[2017-06-01 19:10:46,807: INFO/job_scheduler] Job 1056 completing (errored=False, cancelled=False)
[2017-06-01 19:10:46,846: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:10:47,080: INFO/job_scheduler] Job 1057 finished 1 steps successfully
[2017-06-01 19:10:47,082: INFO/job_scheduler] Job 1058 finished 1 steps successfully
[2017-06-01 19:10:47,138: INFO/job_scheduler] Job 1057: complete_job: Updating cache
[2017-06-01 19:10:47,138: INFO/job_scheduler] Job 1057 completing (errored=False, cancelled=False)
[2017-06-01 19:10:47,174: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:10:47,191: INFO/job_scheduler] Job 1058: complete_job: Updating cache
[2017-06-01 19:10:47,191: INFO/job_scheduler] Job 1058 completing (errored=False, cancelled=False)
[2017-06-01 19:10:47,265: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:10:49,009: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:49,011: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:49,023: INFO/command_plan] add_jobs: created Job 1059 (Test for host connectivity)
[2017-06-01 19:10:49,038: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:10:49,054: INFO/job_scheduler] Job 1059: RunJobThread.run
[2017-06-01 19:10:49,122: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:49,123: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:49,145: INFO/command_plan] add_jobs: created Job 1060 (Test for host connectivity)
[2017-06-01 19:10:49,184: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:10:49,219: INFO/job_scheduler] Job 1060: RunJobThread.run
[2017-06-01 19:10:49,290: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:49,299: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:49,340: INFO/command_plan] add_jobs: created Job 1061 (Test for host connectivity)
[2017-06-01 19:10:49,496: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:49,496: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:49,543: INFO/command_plan] add_jobs: created Job 1062 (Test for host connectivity)
[2017-06-01 19:10:49,586: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:10:49,598: INFO/job_scheduler] Job 1061: RunJobThread.run
[2017-06-01 19:10:49,599: INFO/job_scheduler] Job 1062: RunJobThread.run
[2017-06-01 19:10:49,621: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:10:51,477: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:10:51,557: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:10:51,866: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:10:51,873: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:10:52,817: INFO/job_scheduler] Job 1060 finished 1 steps successfully
[2017-06-01 19:10:52,878: INFO/job_scheduler] Job 1060: complete_job: Updating cache
[2017-06-01 19:10:52,878: INFO/job_scheduler] Job 1060 completing (errored=False, cancelled=False)
[2017-06-01 19:10:52,947: INFO/job_scheduler] Job 1059 finished 1 steps successfully
[2017-06-01 19:10:52,987: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:10:53,028: INFO/job_scheduler] Job 1059: complete_job: Updating cache
[2017-06-01 19:10:53,040: INFO/job_scheduler] Job 1059 completing (errored=False, cancelled=False)
[2017-06-01 19:10:53,180: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:10:53,229: INFO/job_scheduler] Job 1062 finished 1 steps successfully
[2017-06-01 19:10:53,256: INFO/job_scheduler] Job 1062: complete_job: Updating cache
[2017-06-01 19:10:53,268: INFO/job_scheduler] Job 1062 completing (errored=False, cancelled=False)
[2017-06-01 19:10:53,322: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:10:53,434: INFO/job_scheduler] Job 1061 finished 1 steps successfully
[2017-06-01 19:10:53,478: INFO/job_scheduler] Job 1061: complete_job: Updating cache
[2017-06-01 19:10:53,478: INFO/job_scheduler] Job 1061 completing (errored=False, cancelled=False)
[2017-06-01 19:10:53,512: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:10:55,324: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:55,325: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:55,345: INFO/command_plan] add_jobs: created Job 1063 (Test for host connectivity)
[2017-06-01 19:10:55,389: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:10:55,400: INFO/job_scheduler] Job 1063: RunJobThread.run
[2017-06-01 19:10:55,458: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:55,460: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:55,470: INFO/command_plan] add_jobs: created Job 1064 (Test for host connectivity)
[2017-06-01 19:10:55,609: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:55,610: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:55,736: INFO/command_plan] add_jobs: created Job 1065 (Test for host connectivity)
[2017-06-01 19:10:55,745: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:10:55,747: INFO/job_scheduler] Job 1064: RunJobThread.run
[2017-06-01 19:10:55,748: INFO/job_scheduler] Job 1065: RunJobThread.run
[2017-06-01 19:10:55,754: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:10:55,812: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:10:55,812: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:10:55,834: INFO/command_plan] add_jobs: created Job 1066 (Test for host connectivity)
[2017-06-01 19:10:55,857: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:10:55,893: INFO/job_scheduler] Job 1066: RunJobThread.run
[2017-06-01 19:10:57,852: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:10:58,260: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:10:58,279: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:10:58,295: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:10:59,157: INFO/job_scheduler] Job 1063 finished 1 steps successfully
[2017-06-01 19:10:59,216: INFO/job_scheduler] Job 1063: complete_job: Updating cache
[2017-06-01 19:10:59,216: INFO/job_scheduler] Job 1063 completing (errored=False, cancelled=False)
[2017-06-01 19:10:59,291: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:10:59,730: INFO/job_scheduler] Job 1064 finished 1 steps successfully
[2017-06-01 19:10:59,756: INFO/job_scheduler] Job 1064: complete_job: Updating cache
[2017-06-01 19:10:59,756: INFO/job_scheduler] Job 1064 completing (errored=False, cancelled=False)
[2017-06-01 19:10:59,790: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:10:59,811: INFO/job_scheduler] Job 1066 finished 1 steps successfully
[2017-06-01 19:10:59,834: INFO/job_scheduler] Job 1065 finished 1 steps successfully
[2017-06-01 19:10:59,853: INFO/job_scheduler] Job 1066: complete_job: Updating cache
[2017-06-01 19:10:59,853: INFO/job_scheduler] Job 1066 completing (errored=False, cancelled=False)
[2017-06-01 19:10:59,939: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:10:59,957: INFO/job_scheduler] Job 1065: complete_job: Updating cache
[2017-06-01 19:10:59,958: INFO/job_scheduler] Job 1065 completing (errored=False, cancelled=False)
[2017-06-01 19:11:00,024: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:11:02,230: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:02,230: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:02,245: INFO/command_plan] add_jobs: created Job 1067 (Test for host connectivity)
[2017-06-01 19:11:02,344: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:02,344: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:02,353: INFO/command_plan] add_jobs: created Job 1068 (Test for host connectivity)
[2017-06-01 19:11:02,478: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:11:02,484: INFO/job_scheduler] Job 1067: RunJobThread.run
[2017-06-01 19:11:02,484: INFO/job_scheduler] Job 1068: RunJobThread.run
[2017-06-01 19:11:02,539: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:02,541: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:02,564: INFO/command_plan] add_jobs: created Job 1069 (Test for host connectivity)
[2017-06-01 19:11:02,580: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:11:02,583: INFO/job_scheduler] Job 1069: RunJobThread.run
[2017-06-01 19:11:02,739: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:02,742: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:02,755: INFO/command_plan] add_jobs: created Job 1070 (Test for host connectivity)
[2017-06-01 19:11:02,770: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:11:02,799: INFO/job_scheduler] Job 1070: RunJobThread.run
[2017-06-01 19:11:02,850: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:11:04,732: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:11:04,748: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:11:04,929: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:11:05,028: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:11:06,063: INFO/job_scheduler] Job 1067 finished 1 steps successfully
[2017-06-01 19:11:06,110: INFO/job_scheduler] Job 1067: complete_job: Updating cache
[2017-06-01 19:11:06,110: INFO/job_scheduler] Job 1067 completing (errored=False, cancelled=False)
[2017-06-01 19:11:06,143: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:11:06,193: INFO/job_scheduler] Job 1068 finished 1 steps successfully
[2017-06-01 19:11:06,259: INFO/job_scheduler] Job 1070 finished 1 steps successfully
[2017-06-01 19:11:06,277: INFO/job_scheduler] Job 1068: complete_job: Updating cache
[2017-06-01 19:11:06,278: INFO/job_scheduler] Job 1068 completing (errored=False, cancelled=False)
[2017-06-01 19:11:06,338: INFO/job_scheduler] Job 1069 finished 1 steps successfully
[2017-06-01 19:11:06,339: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:11:06,352: INFO/job_scheduler] Job 1070: complete_job: Updating cache
[2017-06-01 19:11:06,353: INFO/job_scheduler] Job 1070 completing (errored=False, cancelled=False)
[2017-06-01 19:11:06,438: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:11:06,452: INFO/job_scheduler] Job 1069: complete_job: Updating cache
[2017-06-01 19:11:06,452: INFO/job_scheduler] Job 1069 completing (errored=False, cancelled=False)
[2017-06-01 19:11:06,524: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:11:08,481: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:08,481: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:08,505: INFO/command_plan] add_jobs: created Job 1071 (Test for host connectivity)
[2017-06-01 19:11:08,537: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:11:08,551: INFO/job_scheduler] Job 1071: RunJobThread.run
[2017-06-01 19:11:08,633: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:08,634: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:08,726: INFO/command_plan] add_jobs: created Job 1072 (Test for host connectivity)
[2017-06-01 19:11:08,772: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:08,772: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:08,800: INFO/command_plan] add_jobs: created Job 1073 (Test for host connectivity)
[2017-06-01 19:11:08,811: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:11:08,863: INFO/job_scheduler] Job 1072: RunJobThread.run
[2017-06-01 19:11:08,873: INFO/job_scheduler] Job 1073: RunJobThread.run
[2017-06-01 19:11:08,886: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:11:08,962: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:08,962: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:09,006: INFO/command_plan] add_jobs: created Job 1074 (Test for host connectivity)
[2017-06-01 19:11:09,131: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:11:09,145: INFO/job_scheduler] Job 1074: RunJobThread.run
[2017-06-01 19:11:10,893: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:11:11,249: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:11:11,329: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:11:11,361: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:11:12,241: INFO/job_scheduler] Job 1071 finished 1 steps successfully
[2017-06-01 19:11:12,292: INFO/job_scheduler] Job 1071: complete_job: Updating cache
[2017-06-01 19:11:12,292: INFO/job_scheduler] Job 1071 completing (errored=False, cancelled=False)
[2017-06-01 19:11:12,367: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:11:12,604: INFO/job_scheduler] Job 1072 finished 1 steps successfully
[2017-06-01 19:11:12,659: INFO/job_scheduler] Job 1072: complete_job: Updating cache
[2017-06-01 19:11:12,659: INFO/job_scheduler] Job 1072 completing (errored=False, cancelled=False)
[2017-06-01 19:11:12,709: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:11:12,808: INFO/job_scheduler] Job 1074 finished 1 steps successfully
[2017-06-01 19:11:12,812: INFO/job_scheduler] Job 1073 finished 1 steps successfully
[2017-06-01 19:11:12,817: INFO/job_scheduler] Job 1074: complete_job: Updating cache
[2017-06-01 19:11:12,817: INFO/job_scheduler] Job 1074 completing (errored=False, cancelled=False)
[2017-06-01 19:11:12,920: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:11:12,963: INFO/job_scheduler] Job 1073: complete_job: Updating cache
[2017-06-01 19:11:12,963: INFO/job_scheduler] Job 1073 completing (errored=False, cancelled=False)
[2017-06-01 19:11:13,035: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:11:14,736: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:14,736: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:14,750: INFO/command_plan] add_jobs: created Job 1075 (Test for host connectivity)
[2017-06-01 19:11:14,817: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:11:14,820: INFO/job_scheduler] Job 1075: RunJobThread.run
[2017-06-01 19:11:14,860: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:14,863: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:14,889: INFO/command_plan] add_jobs: created Job 1076 (Test for host connectivity)
[2017-06-01 19:11:14,931: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:11:14,944: INFO/job_scheduler] Job 1076: RunJobThread.run
[2017-06-01 19:11:15,018: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:15,019: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:15,050: INFO/command_plan] add_jobs: created Job 1077 (Test for host connectivity)
[2017-06-01 19:11:15,076: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:11:15,101: INFO/job_scheduler] Job 1077: RunJobThread.run
[2017-06-01 19:11:15,208: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:15,325: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:15,387: INFO/command_plan] add_jobs: created Job 1078 (Test for host connectivity)
[2017-06-01 19:11:15,446: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:11:15,457: INFO/job_scheduler] Job 1078: RunJobThread.run
[2017-06-01 19:11:17,159: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:11:17,353: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:11:17,490: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:11:17,683: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:11:18,442: INFO/job_scheduler] Job 1075 finished 1 steps successfully
[2017-06-01 19:11:18,532: INFO/job_scheduler] Job 1075: complete_job: Updating cache
[2017-06-01 19:11:18,533: INFO/job_scheduler] Job 1075 completing (errored=False, cancelled=False)
[2017-06-01 19:11:18,602: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:11:18,688: INFO/job_scheduler] Job 1076 finished 1 steps successfully
[2017-06-01 19:11:18,712: INFO/job_scheduler] Job 1076: complete_job: Updating cache
[2017-06-01 19:11:18,714: INFO/job_scheduler] Job 1076 completing (errored=False, cancelled=False)
[2017-06-01 19:11:18,792: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:11:18,891: INFO/job_scheduler] Job 1077 finished 1 steps successfully
[2017-06-01 19:11:18,942: INFO/job_scheduler] Job 1077: complete_job: Updating cache
[2017-06-01 19:11:18,942: INFO/job_scheduler] Job 1077 completing (errored=False, cancelled=False)
[2017-06-01 19:11:18,983: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:11:19,168: INFO/job_scheduler] Job 1078 finished 1 steps successfully
[2017-06-01 19:11:19,191: INFO/job_scheduler] Job 1078: complete_job: Updating cache
[2017-06-01 19:11:19,191: INFO/job_scheduler] Job 1078 completing (errored=False, cancelled=False)
[2017-06-01 19:11:19,225: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:11:21,054: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:21,054: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:21,091: INFO/command_plan] add_jobs: created Job 1079 (Test for host connectivity)
[2017-06-01 19:11:21,170: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:21,170: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:21,196: INFO/command_plan] add_jobs: created Job 1080 (Test for host connectivity)
[2017-06-01 19:11:21,209: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:11:21,218: INFO/job_scheduler] Job 1080: RunJobThread.run
[2017-06-01 19:11:21,236: INFO/job_scheduler] Job 1079: RunJobThread.run
[2017-06-01 19:11:21,343: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:21,343: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:21,369: INFO/command_plan] add_jobs: created Job 1081 (Test for host connectivity)
[2017-06-01 19:11:21,406: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:11:21,445: INFO/job_scheduler] Job 1081: RunJobThread.run
[2017-06-01 19:11:21,546: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:21,547: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:21,581: INFO/command_plan] add_jobs: created Job 1082 (Test for host connectivity)
[2017-06-01 19:11:21,611: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:11:21,629: INFO/job_scheduler] Job 1082: RunJobThread.run
[2017-06-01 19:11:21,665: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:11:23,593: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:11:23,708: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:11:23,802: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:11:24,001: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:11:24,904: INFO/job_scheduler] Job 1079 finished 1 steps successfully
[2017-06-01 19:11:24,956: INFO/job_scheduler] Job 1079: complete_job: Updating cache
[2017-06-01 19:11:24,956: INFO/job_scheduler] Job 1079 completing (errored=False, cancelled=False)
[2017-06-01 19:11:24,991: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:11:25,105: INFO/job_scheduler] Job 1080 finished 1 steps successfully
[2017-06-01 19:11:25,149: INFO/job_scheduler] Job 1080: complete_job: Updating cache
[2017-06-01 19:11:25,150: INFO/job_scheduler] Job 1080 completing (errored=False, cancelled=False)
[2017-06-01 19:11:25,157: INFO/job_scheduler] Job 1081 finished 1 steps successfully
[2017-06-01 19:11:25,214: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:11:25,223: INFO/job_scheduler] Job 1081: complete_job: Updating cache
[2017-06-01 19:11:25,238: INFO/job_scheduler] Job 1081 completing (errored=False, cancelled=False)
[2017-06-01 19:11:25,225: INFO/job_scheduler] Job 1082 finished 1 steps successfully
[2017-06-01 19:11:25,299: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:11:25,331: INFO/job_scheduler] Job 1082: complete_job: Updating cache
[2017-06-01 19:11:25,331: INFO/job_scheduler] Job 1082 completing (errored=False, cancelled=False)
[2017-06-01 19:11:25,388: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:11:27,289: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:27,289: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:27,299: INFO/command_plan] add_jobs: created Job 1083 (Test for host connectivity)
[2017-06-01 19:11:27,392: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:27,392: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:27,431: INFO/command_plan] add_jobs: created Job 1084 (Test for host connectivity)
[2017-06-01 19:11:27,441: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:11:27,518: INFO/job_scheduler] Job 1083: RunJobThread.run
[2017-06-01 19:11:27,530: INFO/job_scheduler] Job 1084: RunJobThread.run
[2017-06-01 19:11:27,540: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:11:27,580: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:27,581: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:27,614: INFO/command_plan] add_jobs: created Job 1085 (Test for host connectivity)
[2017-06-01 19:11:27,661: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:11:27,730: INFO/job_scheduler] Job 1085: RunJobThread.run
[2017-06-01 19:11:27,778: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:27,783: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:27,798: INFO/command_plan] add_jobs: created Job 1086 (Test for host connectivity)
[2017-06-01 19:11:27,885: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:11:27,905: INFO/job_scheduler] Job 1086: RunJobThread.run
[2017-06-01 19:11:29,731: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:11:29,881: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:11:30,024: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:11:30,079: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:11:31,095: INFO/job_scheduler] Job 1083 finished 1 steps successfully
[2017-06-01 19:11:31,138: INFO/job_scheduler] Job 1083: complete_job: Updating cache
[2017-06-01 19:11:31,138: INFO/job_scheduler] Job 1083 completing (errored=False, cancelled=False)
[2017-06-01 19:11:31,322: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:11:31,328: INFO/job_scheduler] Job 1084 finished 1 steps successfully
[2017-06-01 19:11:31,381: INFO/job_scheduler] Job 1084: complete_job: Updating cache
[2017-06-01 19:11:31,382: INFO/job_scheduler] Job 1084 completing (errored=False, cancelled=False)
[2017-06-01 19:11:31,445: INFO/job_scheduler] Job 1085 finished 1 steps successfully
[2017-06-01 19:11:31,481: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:11:31,500: INFO/job_scheduler] Job 1085: complete_job: Updating cache
[2017-06-01 19:11:31,500: INFO/job_scheduler] Job 1085 completing (errored=False, cancelled=False)
[2017-06-01 19:11:31,574: INFO/job_scheduler] Job 1086 finished 1 steps successfully
[2017-06-01 19:11:31,606: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:11:31,644: INFO/job_scheduler] Job 1086: complete_job: Updating cache
[2017-06-01 19:11:31,644: INFO/job_scheduler] Job 1086 completing (errored=False, cancelled=False)
[2017-06-01 19:11:31,731: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:11:33,532: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:33,534: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:33,549: INFO/command_plan] add_jobs: created Job 1087 (Test for host connectivity)
[2017-06-01 19:11:33,696: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:33,696: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:33,742: INFO/command_plan] add_jobs: created Job 1088 (Test for host connectivity)
[2017-06-01 19:11:33,765: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:11:33,767: INFO/job_scheduler] Job 1088: RunJobThread.run
[2017-06-01 19:11:33,767: INFO/job_scheduler] Job 1087: RunJobThread.run
[2017-06-01 19:11:33,773: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:11:33,857: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:33,857: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:33,872: INFO/command_plan] add_jobs: created Job 1089 (Test for host connectivity)
[2017-06-01 19:11:33,948: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:11:33,992: INFO/job_scheduler] Job 1089: RunJobThread.run
[2017-06-01 19:11:34,071: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:34,071: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:34,089: INFO/command_plan] add_jobs: created Job 1090 (Test for host connectivity)
[2017-06-01 19:11:34,149: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:11:34,178: INFO/job_scheduler] Job 1090: RunJobThread.run
[2017-06-01 19:11:36,037: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:11:36,149: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:11:36,306: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:11:36,374: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:11:37,404: INFO/job_scheduler] Job 1087 finished 1 steps successfully
[2017-06-01 19:11:37,462: INFO/job_scheduler] Job 1087: complete_job: Updating cache
[2017-06-01 19:11:37,462: INFO/job_scheduler] Job 1087 completing (errored=False, cancelled=False)
[2017-06-01 19:11:37,466: INFO/job_scheduler] Job 1088 finished 1 steps successfully
[2017-06-01 19:11:37,549: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:11:37,615: INFO/job_scheduler] Job 1088: complete_job: Updating cache
[2017-06-01 19:11:37,615: INFO/job_scheduler] Job 1088 completing (errored=False, cancelled=False)
[2017-06-01 19:11:37,657: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:11:37,740: INFO/job_scheduler] Job 1090 finished 1 steps successfully
[2017-06-01 19:11:37,760: INFO/job_scheduler] Job 1089 finished 1 steps successfully
[2017-06-01 19:11:37,795: INFO/job_scheduler] Job 1090: complete_job: Updating cache
[2017-06-01 19:11:37,795: INFO/job_scheduler] Job 1090 completing (errored=False, cancelled=False)
[2017-06-01 19:11:37,829: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:11:37,848: INFO/job_scheduler] Job 1089: complete_job: Updating cache
[2017-06-01 19:11:37,849: INFO/job_scheduler] Job 1089 completing (errored=False, cancelled=False)
[2017-06-01 19:11:37,938: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:11:39,833: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:39,841: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:39,853: INFO/command_plan] add_jobs: created Job 1091 (Test for host connectivity)
[2017-06-01 19:11:39,911: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:39,911: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:39,939: INFO/command_plan] add_jobs: created Job 1092 (Test for host connectivity)
[2017-06-01 19:11:39,969: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:11:39,991: INFO/job_scheduler] Job 1091: RunJobThread.run
[2017-06-01 19:11:40,027: INFO/job_scheduler] Job 1092: RunJobThread.run
[2017-06-01 19:11:40,085: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:40,085: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:40,095: INFO/command_plan] add_jobs: created Job 1093 (Test for host connectivity)
[2017-06-01 19:11:40,126: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:11:40,211: INFO/job_scheduler] Job 1093: RunJobThread.run
[2017-06-01 19:11:40,315: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:40,317: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:40,328: INFO/command_plan] add_jobs: created Job 1094 (Test for host connectivity)
[2017-06-01 19:11:40,339: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:11:40,411: INFO/job_scheduler] Job 1094: RunJobThread.run
[2017-06-01 19:11:40,436: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:11:42,311: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:11:42,411: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:11:42,506: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:11:42,758: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:11:43,669: INFO/job_scheduler] Job 1091 finished 1 steps successfully
[2017-06-01 19:11:43,711: INFO/job_scheduler] Job 1091: complete_job: Updating cache
[2017-06-01 19:11:43,711: INFO/job_scheduler] Job 1091 completing (errored=False, cancelled=False)
[2017-06-01 19:11:43,775: INFO/job_scheduler] Job 1092 finished 1 steps successfully
[2017-06-01 19:11:43,812: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:11:43,846: INFO/job_scheduler] Job 1092: complete_job: Updating cache
[2017-06-01 19:11:43,853: INFO/job_scheduler] Job 1092 completing (errored=False, cancelled=False)
[2017-06-01 19:11:43,968: INFO/job_scheduler] Job 1093 finished 1 steps successfully
[2017-06-01 19:11:43,986: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:11:44,008: INFO/job_scheduler] Job 1093: complete_job: Updating cache
[2017-06-01 19:11:44,008: INFO/job_scheduler] Job 1093 completing (errored=False, cancelled=False)
[2017-06-01 19:11:44,079: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:11:44,250: INFO/job_scheduler] Job 1094 finished 1 steps successfully
[2017-06-01 19:11:44,269: INFO/job_scheduler] Job 1094: complete_job: Updating cache
[2017-06-01 19:11:44,269: INFO/job_scheduler] Job 1094 completing (errored=False, cancelled=False)
[2017-06-01 19:11:44,303: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:11:45,997: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:45,998: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:46,011: INFO/command_plan] add_jobs: created Job 1095 (Test for host connectivity)
[2017-06-01 19:11:46,025: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:11:46,086: INFO/job_scheduler] Job 1095: RunJobThread.run
[2017-06-01 19:11:46,173: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:46,179: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:46,194: INFO/command_plan] add_jobs: created Job 1096 (Test for host connectivity)
[2017-06-01 19:11:46,224: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:11:46,298: INFO/job_scheduler] Job 1096: RunJobThread.run
[2017-06-01 19:11:46,326: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:46,330: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:46,357: INFO/command_plan] add_jobs: created Job 1097 (Test for host connectivity)
[2017-06-01 19:11:46,409: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:11:46,453: INFO/job_scheduler] Job 1097: RunJobThread.run
[2017-06-01 19:11:46,513: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:46,522: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:46,609: INFO/command_plan] add_jobs: created Job 1098 (Test for host connectivity)
[2017-06-01 19:11:46,627: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:11:46,817: INFO/job_scheduler] Job 1098: RunJobThread.run
[2017-06-01 19:11:48,578: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:11:48,745: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:11:48,814: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:11:49,130: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:11:49,961: INFO/job_scheduler] Job 1095 finished 1 steps successfully
[2017-06-01 19:11:50,005: INFO/job_scheduler] Job 1095: complete_job: Updating cache
[2017-06-01 19:11:50,006: INFO/job_scheduler] Job 1095 completing (errored=False, cancelled=False)
[2017-06-01 19:11:50,119: INFO/job_scheduler] Job 1096 finished 1 steps successfully
[2017-06-01 19:11:50,187: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:11:50,219: INFO/job_scheduler] Job 1096: complete_job: Updating cache
[2017-06-01 19:11:50,219: INFO/job_scheduler] Job 1096 completing (errored=False, cancelled=False)
[2017-06-01 19:11:50,348: INFO/job_scheduler] Job 1097 finished 1 steps successfully
[2017-06-01 19:11:50,351: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:11:50,369: INFO/job_scheduler] Job 1097: complete_job: Updating cache
[2017-06-01 19:11:50,369: INFO/job_scheduler] Job 1097 completing (errored=False, cancelled=False)
[2017-06-01 19:11:50,410: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:11:50,543: INFO/job_scheduler] Job 1098 finished 1 steps successfully
[2017-06-01 19:11:50,593: INFO/job_scheduler] Job 1098: complete_job: Updating cache
[2017-06-01 19:11:50,593: INFO/job_scheduler] Job 1098 completing (errored=False, cancelled=False)
[2017-06-01 19:11:50,636: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:11:52,301: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:52,303: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:52,309: INFO/command_plan] add_jobs: created Job 1099 (Test for host connectivity)
[2017-06-01 19:11:52,400: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:52,400: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:52,439: INFO/command_plan] add_jobs: created Job 1100 (Test for host connectivity)
[2017-06-01 19:11:52,470: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:11:52,487: INFO/job_scheduler] Job 1099: RunJobThread.run
[2017-06-01 19:11:52,488: INFO/job_scheduler] Job 1100: RunJobThread.run
[2017-06-01 19:11:52,499: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:11:52,610: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:52,610: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:52,703: INFO/command_plan] add_jobs: created Job 1101 (Test for host connectivity)
[2017-06-01 19:11:52,787: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:52,788: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:52,820: INFO/command_plan] add_jobs: created Job 1102 (Test for host connectivity)
[2017-06-01 19:11:52,867: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:11:52,871: INFO/job_scheduler] Job 1101: RunJobThread.run
[2017-06-01 19:11:52,871: INFO/job_scheduler] Job 1102: RunJobThread.run
[2017-06-01 19:11:52,881: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:11:54,759: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:11:54,924: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:11:55,180: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:11:55,216: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:11:56,113: INFO/job_scheduler] Job 1099 finished 1 steps successfully
[2017-06-01 19:11:56,141: INFO/job_scheduler] Job 1099: complete_job: Updating cache
[2017-06-01 19:11:56,142: INFO/job_scheduler] Job 1099 completing (errored=False, cancelled=False)
[2017-06-01 19:11:56,201: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:11:56,258: INFO/job_scheduler] Job 1100 finished 1 steps successfully
[2017-06-01 19:11:56,333: INFO/job_scheduler] Job 1100: complete_job: Updating cache
[2017-06-01 19:11:56,333: INFO/job_scheduler] Job 1100 completing (errored=False, cancelled=False)
[2017-06-01 19:11:56,445: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:11:56,629: INFO/job_scheduler] Job 1102 finished 1 steps successfully
[2017-06-01 19:11:56,647: INFO/job_scheduler] Job 1101 finished 1 steps successfully
[2017-06-01 19:11:56,717: INFO/job_scheduler] Job 1102: complete_job: Updating cache
[2017-06-01 19:11:56,717: INFO/job_scheduler] Job 1102 completing (errored=False, cancelled=False)
[2017-06-01 19:11:56,777: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:11:56,820: INFO/job_scheduler] Job 1101: complete_job: Updating cache
[2017-06-01 19:11:56,823: INFO/job_scheduler] Job 1101 completing (errored=False, cancelled=False)
[2017-06-01 19:11:56,885: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:11:58,583: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:58,584: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:58,595: INFO/command_plan] add_jobs: created Job 1103 (Test for host connectivity)
[2017-06-01 19:11:58,695: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:58,696: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:58,709: INFO/command_plan] add_jobs: created Job 1104 (Test for host connectivity)
[2017-06-01 19:11:58,736: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:11:58,741: INFO/job_scheduler] Job 1104: RunJobThread.run
[2017-06-01 19:11:58,806: INFO/job_scheduler] Job 1103: RunJobThread.run
[2017-06-01 19:11:58,853: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:58,853: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:58,875: INFO/command_plan] add_jobs: created Job 1105 (Test for host connectivity)
[2017-06-01 19:11:58,904: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:11:58,988: INFO/job_scheduler] Job 1105: RunJobThread.run
[2017-06-01 19:11:59,087: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:11:59,088: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:11:59,106: INFO/command_plan] add_jobs: created Job 1106 (Test for host connectivity)
[2017-06-01 19:11:59,144: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:11:59,145: INFO/job_scheduler] Job 1106: RunJobThread.run
[2017-06-01 19:11:59,187: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:12:01,060: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:12:01,135: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:12:01,292: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:12:01,393: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:12:02,438: INFO/job_scheduler] Job 1103 finished 1 steps successfully
[2017-06-01 19:12:02,482: INFO/job_scheduler] Job 1103: complete_job: Updating cache
[2017-06-01 19:12:02,482: INFO/job_scheduler] Job 1103 completing (errored=False, cancelled=False)
[2017-06-01 19:12:02,488: INFO/job_scheduler] Job 1104 finished 1 steps successfully
[2017-06-01 19:12:02,544: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:12:02,576: INFO/job_scheduler] Job 1104: complete_job: Updating cache
[2017-06-01 19:12:02,576: INFO/job_scheduler] Job 1104 completing (errored=False, cancelled=False)
[2017-06-01 19:12:02,668: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:12:02,682: INFO/job_scheduler] Job 1105 finished 1 steps successfully
[2017-06-01 19:12:02,720: INFO/job_scheduler] Job 1105: complete_job: Updating cache
[2017-06-01 19:12:02,720: INFO/job_scheduler] Job 1105 completing (errored=False, cancelled=False)
[2017-06-01 19:12:02,768: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:12:02,786: INFO/job_scheduler] Job 1106 finished 1 steps successfully
[2017-06-01 19:12:02,800: INFO/job_scheduler] Job 1106: complete_job: Updating cache
[2017-06-01 19:12:02,800: INFO/job_scheduler] Job 1106 completing (errored=False, cancelled=False)
[2017-06-01 19:12:02,888: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:12:04,802: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:04,802: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:04,825: INFO/command_plan] add_jobs: created Job 1107 (Test for host connectivity)
[2017-06-01 19:12:04,865: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:12:04,893: INFO/job_scheduler] Job 1107: RunJobThread.run
[2017-06-01 19:12:04,938: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:04,941: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:04,949: INFO/command_plan] add_jobs: created Job 1108 (Test for host connectivity)
[2017-06-01 19:12:05,033: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:12:05,046: INFO/job_scheduler] Job 1108: RunJobThread.run
[2017-06-01 19:12:05,111: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:05,112: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:05,124: INFO/command_plan] add_jobs: created Job 1109 (Test for host connectivity)
[2017-06-01 19:12:05,215: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:12:05,257: INFO/job_scheduler] Job 1109: RunJobThread.run
[2017-06-01 19:12:05,311: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:05,312: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:05,326: INFO/command_plan] add_jobs: created Job 1110 (Test for host connectivity)
[2017-06-01 19:12:05,400: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:12:05,487: INFO/job_scheduler] Job 1110: RunJobThread.run
[2017-06-01 19:12:07,135: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:12:07,396: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:12:07,637: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:12:07,642: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:12:08,516: INFO/job_scheduler] Job 1107 finished 1 steps successfully
[2017-06-01 19:12:08,532: INFO/job_scheduler] Job 1107: complete_job: Updating cache
[2017-06-01 19:12:08,532: INFO/job_scheduler] Job 1107 completing (errored=False, cancelled=False)
[2017-06-01 19:12:08,599: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:12:08,795: INFO/job_scheduler] Job 1108 finished 1 steps successfully
[2017-06-01 19:12:08,865: INFO/job_scheduler] Job 1108: complete_job: Updating cache
[2017-06-01 19:12:08,866: INFO/job_scheduler] Job 1108 completing (errored=False, cancelled=False)
[2017-06-01 19:12:08,998: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:12:09,153: INFO/job_scheduler] Job 1109 finished 1 steps successfully
[2017-06-01 19:12:09,172: INFO/job_scheduler] Job 1110 finished 1 steps successfully
[2017-06-01 19:12:09,214: INFO/job_scheduler] Job 1109: complete_job: Updating cache
[2017-06-01 19:12:09,214: INFO/job_scheduler] Job 1109 completing (errored=False, cancelled=False)
[2017-06-01 19:12:09,248: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:12:09,285: INFO/job_scheduler] Job 1110: complete_job: Updating cache
[2017-06-01 19:12:09,285: INFO/job_scheduler] Job 1110 completing (errored=False, cancelled=False)
[2017-06-01 19:12:09,349: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:12:11,109: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:11,109: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:11,122: INFO/command_plan] add_jobs: created Job 1111 (Test for host connectivity)
[2017-06-01 19:12:11,211: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:11,211: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:11,222: INFO/command_plan] add_jobs: created Job 1112 (Test for host connectivity)
[2017-06-01 19:12:11,256: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:12:11,305: INFO/job_scheduler] Job 1112: RunJobThread.run
[2017-06-01 19:12:11,331: INFO/job_scheduler] Job 1111: RunJobThread.run
[2017-06-01 19:12:11,381: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:11,381: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:11,392: INFO/command_plan] add_jobs: created Job 1113 (Test for host connectivity)
[2017-06-01 19:12:11,444: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:12:11,507: INFO/job_scheduler] Job 1113: RunJobThread.run
[2017-06-01 19:12:11,604: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:11,605: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:11,632: INFO/command_plan] add_jobs: created Job 1114 (Test for host connectivity)
[2017-06-01 19:12:11,650: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:12:11,660: INFO/job_scheduler] Job 1114: RunJobThread.run
[2017-06-01 19:12:11,768: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:12:13,611: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:12:13,643: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:12:13,761: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:12:14,037: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:12:14,981: INFO/job_scheduler] Job 1111 finished 1 steps successfully
[2017-06-01 19:12:15,043: INFO/job_scheduler] Job 1112 finished 1 steps successfully
[2017-06-01 19:12:15,055: INFO/job_scheduler] Job 1111: complete_job: Updating cache
[2017-06-01 19:12:15,055: INFO/job_scheduler] Job 1111 completing (errored=False, cancelled=False)
[2017-06-01 19:12:15,113: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:12:15,130: INFO/job_scheduler] Job 1112: complete_job: Updating cache
[2017-06-01 19:12:15,131: INFO/job_scheduler] Job 1112 completing (errored=False, cancelled=False)
[2017-06-01 19:12:15,273: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:12:15,366: INFO/job_scheduler] Job 1113 finished 1 steps successfully
[2017-06-01 19:12:15,405: INFO/job_scheduler] Job 1113: complete_job: Updating cache
[2017-06-01 19:12:15,405: INFO/job_scheduler] Job 1113 completing (errored=False, cancelled=False)
[2017-06-01 19:12:15,444: INFO/job_scheduler] Job 1114 finished 1 steps successfully
[2017-06-01 19:12:15,464: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:12:15,483: INFO/job_scheduler] Job 1114: complete_job: Updating cache
[2017-06-01 19:12:15,483: INFO/job_scheduler] Job 1114 completing (errored=False, cancelled=False)
[2017-06-01 19:12:15,576: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:12:17,800: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:17,803: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:17,826: INFO/command_plan] add_jobs: created Job 1115 (Test for host connectivity)
[2017-06-01 19:12:17,909: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:12:17,928: INFO/job_scheduler] Job 1115: RunJobThread.run
[2017-06-01 19:12:17,975: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:17,976: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:17,988: INFO/command_plan] add_jobs: created Job 1116 (Test for host connectivity)
[2017-06-01 19:12:18,089: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:12:18,101: INFO/job_scheduler] Job 1116: RunJobThread.run
[2017-06-01 19:12:18,163: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:18,163: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:18,192: INFO/command_plan] add_jobs: created Job 1117 (Test for host connectivity)
[2017-06-01 19:12:18,327: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:12:18,334: INFO/job_scheduler] Job 1117: RunJobThread.run
[2017-06-01 19:12:18,370: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:18,370: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:18,405: INFO/command_plan] add_jobs: created Job 1118 (Test for host connectivity)
[2017-06-01 19:12:18,470: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:12:18,526: INFO/job_scheduler] Job 1118: RunJobThread.run
[2017-06-01 19:12:20,237: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:12:20,423: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:12:20,639: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:12:20,779: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:12:21,526: INFO/job_scheduler] Job 1115 finished 1 steps successfully
[2017-06-01 19:12:21,570: INFO/job_scheduler] Job 1115: complete_job: Updating cache
[2017-06-01 19:12:21,570: INFO/job_scheduler] Job 1115 completing (errored=False, cancelled=False)
[2017-06-01 19:12:21,712: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:12:21,793: INFO/job_scheduler] Job 1116 finished 1 steps successfully
[2017-06-01 19:12:21,812: INFO/job_scheduler] Job 1116: complete_job: Updating cache
[2017-06-01 19:12:21,812: INFO/job_scheduler] Job 1116 completing (errored=False, cancelled=False)
[2017-06-01 19:12:21,888: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:12:22,128: INFO/job_scheduler] Job 1117 finished 1 steps successfully
[2017-06-01 19:12:22,170: INFO/job_scheduler] Job 1117: complete_job: Updating cache
[2017-06-01 19:12:22,170: INFO/job_scheduler] Job 1117 completing (errored=False, cancelled=False)
[2017-06-01 19:12:22,220: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:12:22,267: INFO/job_scheduler] Job 1118 finished 1 steps successfully
[2017-06-01 19:12:22,309: INFO/job_scheduler] Job 1118: complete_job: Updating cache
[2017-06-01 19:12:22,309: INFO/job_scheduler] Job 1118 completing (errored=False, cancelled=False)
[2017-06-01 19:12:22,373: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:12:24,069: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:24,070: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:24,112: INFO/command_plan] add_jobs: created Job 1119 (Test for host connectivity)
[2017-06-01 19:12:24,205: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:12:24,207: INFO/job_scheduler] Job 1119: RunJobThread.run
[2017-06-01 19:12:24,227: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:24,229: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:24,294: INFO/command_plan] add_jobs: created Job 1120 (Test for host connectivity)
[2017-06-01 19:12:24,318: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:12:24,324: INFO/job_scheduler] Job 1120: RunJobThread.run
[2017-06-01 19:12:24,410: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:24,411: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:24,425: INFO/command_plan] add_jobs: created Job 1121 (Test for host connectivity)
[2017-06-01 19:12:24,514: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:12:24,516: INFO/job_scheduler] Job 1121: RunJobThread.run
[2017-06-01 19:12:24,604: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:24,605: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:24,628: INFO/command_plan] add_jobs: created Job 1122 (Test for host connectivity)
[2017-06-01 19:12:24,645: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:12:24,697: INFO/job_scheduler] Job 1122: RunJobThread.run
[2017-06-01 19:12:26,587: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:12:26,672: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:12:26,779: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:12:26,894: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:12:27,915: INFO/job_scheduler] Job 1119 finished 1 steps successfully
[2017-06-01 19:12:27,952: INFO/job_scheduler] Job 1119: complete_job: Updating cache
[2017-06-01 19:12:27,952: INFO/job_scheduler] Job 1119 completing (errored=False, cancelled=False)
[2017-06-01 19:12:28,041: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:12:28,043: INFO/job_scheduler] Job 1120 finished 1 steps successfully
[2017-06-01 19:12:28,061: INFO/job_scheduler] Job 1120: complete_job: Updating cache
[2017-06-01 19:12:28,061: INFO/job_scheduler] Job 1120 completing (errored=False, cancelled=False)
[2017-06-01 19:12:28,149: INFO/job_scheduler] Job 1121 finished 1 steps successfully
[2017-06-01 19:12:28,152: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:12:28,204: INFO/job_scheduler] Job 1121: complete_job: Updating cache
[2017-06-01 19:12:28,204: INFO/job_scheduler] Job 1121 completing (errored=False, cancelled=False)
[2017-06-01 19:12:28,255: INFO/job_scheduler] Job 1122 finished 1 steps successfully
[2017-06-01 19:12:28,274: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:12:28,355: INFO/job_scheduler] Job 1122: complete_job: Updating cache
[2017-06-01 19:12:28,358: INFO/job_scheduler] Job 1122 completing (errored=False, cancelled=False)
[2017-06-01 19:12:28,394: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:12:30,980: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:30,980: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:31,027: INFO/command_plan] add_jobs: created Job 1123 (Test for host connectivity)
[2017-06-01 19:12:31,062: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:12:31,068: INFO/job_scheduler] Job 1123: RunJobThread.run
[2017-06-01 19:12:31,101: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:31,101: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:31,157: INFO/command_plan] add_jobs: created Job 1124 (Test for host connectivity)
[2017-06-01 19:12:31,220: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:12:31,252: INFO/job_scheduler] Job 1124: RunJobThread.run
[2017-06-01 19:12:31,301: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:31,303: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:31,311: INFO/command_plan] add_jobs: created Job 1125 (Test for host connectivity)
[2017-06-01 19:12:31,357: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:12:31,382: INFO/job_scheduler] Job 1125: RunJobThread.run
[2017-06-01 19:12:31,463: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:31,463: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:31,481: INFO/command_plan] add_jobs: created Job 1126 (Test for host connectivity)
[2017-06-01 19:12:31,527: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:12:31,582: INFO/job_scheduler] Job 1126: RunJobThread.run
[2017-06-01 19:12:33,460: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:12:33,608: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:12:33,755: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:12:33,869: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:12:34,893: INFO/job_scheduler] Job 1123 finished 1 steps successfully
[2017-06-01 19:12:34,978: INFO/job_scheduler] Job 1124 finished 1 steps successfully
[2017-06-01 19:12:34,984: INFO/job_scheduler] Job 1123: complete_job: Updating cache
[2017-06-01 19:12:34,984: INFO/job_scheduler] Job 1123 completing (errored=False, cancelled=False)
[2017-06-01 19:12:35,036: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:12:35,055: INFO/job_scheduler] Job 1124: complete_job: Updating cache
[2017-06-01 19:12:35,055: INFO/job_scheduler] Job 1124 completing (errored=False, cancelled=False)
[2017-06-01 19:12:35,120: INFO/job_scheduler] Job 1125 finished 1 steps successfully
[2017-06-01 19:12:35,169: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:12:35,206: INFO/job_scheduler] Job 1125: complete_job: Updating cache
[2017-06-01 19:12:35,206: INFO/job_scheduler] Job 1125 completing (errored=False, cancelled=False)
[2017-06-01 19:12:35,254: INFO/job_scheduler] Job 1126 finished 1 steps successfully
[2017-06-01 19:12:35,298: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:12:35,443: INFO/job_scheduler] Job 1126: complete_job: Updating cache
[2017-06-01 19:12:35,443: INFO/job_scheduler] Job 1126 completing (errored=False, cancelled=False)
[2017-06-01 19:12:35,487: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:12:37,288: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:37,290: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:37,310: INFO/command_plan] add_jobs: created Job 1127 (Test for host connectivity)
[2017-06-01 19:12:37,367: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:12:37,387: INFO/job_scheduler] Job 1127: RunJobThread.run
[2017-06-01 19:12:37,444: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:37,445: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:37,536: INFO/command_plan] add_jobs: created Job 1128 (Test for host connectivity)
[2017-06-01 19:12:37,575: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:12:37,593: INFO/job_scheduler] Job 1128: RunJobThread.run
[2017-06-01 19:12:37,644: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:37,645: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:37,660: INFO/command_plan] add_jobs: created Job 1129 (Test for host connectivity)
[2017-06-01 19:12:37,708: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:12:37,711: INFO/job_scheduler] Job 1129: RunJobThread.run
[2017-06-01 19:12:37,824: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:37,825: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:37,851: INFO/command_plan] add_jobs: created Job 1130 (Test for host connectivity)
[2017-06-01 19:12:37,969: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:12:37,996: INFO/job_scheduler] Job 1130: RunJobThread.run
[2017-06-01 19:12:39,601: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:12:39,959: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:12:40,074: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:12:40,190: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:12:40,947: INFO/job_scheduler] Job 1127 finished 1 steps successfully
[2017-06-01 19:12:41,001: INFO/job_scheduler] Job 1127: complete_job: Updating cache
[2017-06-01 19:12:41,001: INFO/job_scheduler] Job 1127 completing (errored=False, cancelled=False)
[2017-06-01 19:12:41,069: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:12:41,299: INFO/job_scheduler] Job 1128 finished 1 steps successfully
[2017-06-01 19:12:41,333: INFO/job_scheduler] Job 1128: complete_job: Updating cache
[2017-06-01 19:12:41,333: INFO/job_scheduler] Job 1128 completing (errored=False, cancelled=False)
[2017-06-01 19:12:41,375: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:12:41,485: INFO/job_scheduler] Job 1129 finished 1 steps successfully
[2017-06-01 19:12:41,541: INFO/job_scheduler] Job 1129: complete_job: Updating cache
[2017-06-01 19:12:41,541: INFO/job_scheduler] Job 1129 completing (errored=False, cancelled=False)
[2017-06-01 19:12:41,576: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:12:41,664: INFO/job_scheduler] Job 1130 finished 1 steps successfully
[2017-06-01 19:12:41,761: INFO/job_scheduler] Job 1130: complete_job: Updating cache
[2017-06-01 19:12:41,762: INFO/job_scheduler] Job 1130 completing (errored=False, cancelled=False)
[2017-06-01 19:12:41,801: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:12:43,727: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:43,730: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:43,751: INFO/command_plan] add_jobs: created Job 1131 (Test for host connectivity)
[2017-06-01 19:12:43,776: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:12:43,815: INFO/job_scheduler] Job 1131: RunJobThread.run
[2017-06-01 19:12:43,889: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:43,890: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:43,926: INFO/command_plan] add_jobs: created Job 1132 (Test for host connectivity)
[2017-06-01 19:12:44,023: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:12:44,046: INFO/job_scheduler] Job 1132: RunJobThread.run
[2017-06-01 19:12:44,066: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:44,066: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:44,100: INFO/command_plan] add_jobs: created Job 1133 (Test for host connectivity)
[2017-06-01 19:12:44,162: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:12:44,169: INFO/job_scheduler] Job 1133: RunJobThread.run
[2017-06-01 19:12:44,244: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:44,244: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:44,258: INFO/command_plan] add_jobs: created Job 1134 (Test for host connectivity)
[2017-06-01 19:12:44,363: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:12:44,392: INFO/job_scheduler] Job 1134: RunJobThread.run
[2017-06-01 19:12:46,165: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:12:46,444: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:12:46,552: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:12:46,701: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:12:47,511: INFO/job_scheduler] Job 1131 finished 1 steps successfully
[2017-06-01 19:12:47,548: INFO/job_scheduler] Job 1131: complete_job: Updating cache
[2017-06-01 19:12:47,548: INFO/job_scheduler] Job 1131 completing (errored=False, cancelled=False)
[2017-06-01 19:12:47,734: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:12:47,807: INFO/job_scheduler] Job 1132 finished 1 steps successfully
[2017-06-01 19:12:47,832: INFO/job_scheduler] Job 1132: complete_job: Updating cache
[2017-06-01 19:12:47,833: INFO/job_scheduler] Job 1132 completing (errored=False, cancelled=False)
[2017-06-01 19:12:47,883: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:12:48,006: INFO/job_scheduler] Job 1133 finished 1 steps successfully
[2017-06-01 19:12:48,048: INFO/job_scheduler] Job 1133: complete_job: Updating cache
[2017-06-01 19:12:48,048: INFO/job_scheduler] Job 1133 completing (errored=False, cancelled=False)
[2017-06-01 19:12:48,090: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:12:48,098: INFO/job_scheduler] Job 1134 finished 1 steps successfully
[2017-06-01 19:12:48,115: INFO/job_scheduler] Job 1134: complete_job: Updating cache
[2017-06-01 19:12:48,115: INFO/job_scheduler] Job 1134 completing (errored=False, cancelled=False)
[2017-06-01 19:12:48,201: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:12:50,068: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:50,069: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:50,089: INFO/command_plan] add_jobs: created Job 1135 (Test for host connectivity)
[2017-06-01 19:12:50,154: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:50,154: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:50,183: INFO/command_plan] add_jobs: created Job 1136 (Test for host connectivity)
[2017-06-01 19:12:50,197: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:12:50,242: INFO/job_scheduler] Job 1136: RunJobThread.run
[2017-06-01 19:12:50,252: INFO/job_scheduler] Job 1135: RunJobThread.run
[2017-06-01 19:12:50,273: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:12:50,352: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:50,353: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:50,389: INFO/command_plan] add_jobs: created Job 1137 (Test for host connectivity)
[2017-06-01 19:12:50,424: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:12:50,469: INFO/job_scheduler] Job 1137: RunJobThread.run
[2017-06-01 19:12:50,546: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:50,547: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:50,566: INFO/command_plan] add_jobs: created Job 1138 (Test for host connectivity)
[2017-06-01 19:12:50,621: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:12:50,638: INFO/job_scheduler] Job 1138: RunJobThread.run
[2017-06-01 19:12:52,555: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:12:52,749: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:12:52,781: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:12:53,001: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:12:53,950: INFO/job_scheduler] Job 1135 finished 1 steps successfully
[2017-06-01 19:12:53,980: INFO/job_scheduler] Job 1135: complete_job: Updating cache
[2017-06-01 19:12:53,981: INFO/job_scheduler] Job 1135 completing (errored=False, cancelled=False)
[2017-06-01 19:12:54,031: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:12:54,078: INFO/job_scheduler] Job 1136 finished 1 steps successfully
[2017-06-01 19:12:54,141: INFO/job_scheduler] Job 1136: complete_job: Updating cache
[2017-06-01 19:12:54,143: INFO/job_scheduler] Job 1136 completing (errored=False, cancelled=False)
[2017-06-01 19:12:54,216: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:12:54,301: INFO/job_scheduler] Job 1137 finished 1 steps successfully
[2017-06-01 19:12:54,323: INFO/job_scheduler] Job 1137: complete_job: Updating cache
[2017-06-01 19:12:54,324: INFO/job_scheduler] Job 1137 completing (errored=False, cancelled=False)
[2017-06-01 19:12:54,333: INFO/job_scheduler] Job 1138 finished 1 steps successfully
[2017-06-01 19:12:54,406: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:12:54,422: INFO/job_scheduler] Job 1138: complete_job: Updating cache
[2017-06-01 19:12:54,422: INFO/job_scheduler] Job 1138 completing (errored=False, cancelled=False)
[2017-06-01 19:12:54,600: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:12:56,431: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:56,438: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:56,465: INFO/command_plan] add_jobs: created Job 1139 (Test for host connectivity)
[2017-06-01 19:12:56,518: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:12:56,532: INFO/job_scheduler] Job 1139: RunJobThread.run
[2017-06-01 19:12:56,598: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:56,600: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:56,634: INFO/command_plan] add_jobs: created Job 1140 (Test for host connectivity)
[2017-06-01 19:12:56,684: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:12:56,707: INFO/job_scheduler] Job 1140: RunJobThread.run
[2017-06-01 19:12:56,799: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:56,800: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:56,864: INFO/command_plan] add_jobs: created Job 1141 (Test for host connectivity)
[2017-06-01 19:12:56,920: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:12:56,950: INFO/job_scheduler] Job 1141: RunJobThread.run
[2017-06-01 19:12:57,012: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:12:57,012: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:12:57,049: INFO/command_plan] add_jobs: created Job 1142 (Test for host connectivity)
[2017-06-01 19:12:57,143: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:12:57,178: INFO/job_scheduler] Job 1142: RunJobThread.run
[2017-06-01 19:12:58,837: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:12:59,041: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:12:59,250: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:12:59,533: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:13:00,136: INFO/job_scheduler] Job 1139 finished 1 steps successfully
[2017-06-01 19:13:00,174: INFO/job_scheduler] Job 1139: complete_job: Updating cache
[2017-06-01 19:13:00,174: INFO/job_scheduler] Job 1139 completing (errored=False, cancelled=False)
[2017-06-01 19:13:00,217: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:13:00,391: INFO/job_scheduler] Job 1140 finished 1 steps successfully
[2017-06-01 19:13:00,432: INFO/job_scheduler] Job 1140: complete_job: Updating cache
[2017-06-01 19:13:00,432: INFO/job_scheduler] Job 1140 completing (errored=False, cancelled=False)
[2017-06-01 19:13:00,483: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:13:00,659: INFO/job_scheduler] Job 1141 finished 1 steps successfully
[2017-06-01 19:13:00,741: INFO/job_scheduler] Job 1141: complete_job: Updating cache
[2017-06-01 19:13:00,741: INFO/job_scheduler] Job 1141 completing (errored=False, cancelled=False)
[2017-06-01 19:13:00,784: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:13:00,930: INFO/job_scheduler] Job 1142 finished 1 steps successfully
[2017-06-01 19:13:01,058: INFO/job_scheduler] Job 1142: complete_job: Updating cache
[2017-06-01 19:13:01,058: INFO/job_scheduler] Job 1142 completing (errored=False, cancelled=False)
[2017-06-01 19:13:01,100: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:13:02,919: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:02,919: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:02,969: INFO/command_plan] add_jobs: created Job 1143 (Test for host connectivity)
[2017-06-01 19:13:03,019: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:13:03,020: INFO/job_scheduler] Job 1143: RunJobThread.run
[2017-06-01 19:13:03,056: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:03,056: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:03,109: INFO/command_plan] add_jobs: created Job 1144 (Test for host connectivity)
[2017-06-01 19:13:03,160: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:13:03,181: INFO/job_scheduler] Job 1144: RunJobThread.run
[2017-06-01 19:13:03,256: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:03,257: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:03,291: INFO/command_plan] add_jobs: created Job 1145 (Test for host connectivity)
[2017-06-01 19:13:03,392: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:13:03,394: INFO/job_scheduler] Job 1145: RunJobThread.run
[2017-06-01 19:13:03,426: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:03,427: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:03,452: INFO/command_plan] add_jobs: created Job 1146 (Test for host connectivity)
[2017-06-01 19:13:03,498: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:13:03,522: INFO/job_scheduler] Job 1146: RunJobThread.run
[2017-06-01 19:13:05,412: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:13:05,571: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:13:05,628: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:13:05,714: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:13:06,672: INFO/job_scheduler] Job 1143 finished 1 steps successfully
[2017-06-01 19:13:06,815: INFO/job_scheduler] Job 1143: complete_job: Updating cache
[2017-06-01 19:13:06,817: INFO/job_scheduler] Job 1143 completing (errored=False, cancelled=False)
[2017-06-01 19:13:06,872: INFO/job_scheduler] Job 1144 finished 1 steps successfully
[2017-06-01 19:13:06,915: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:13:06,932: INFO/job_scheduler] Job 1144: complete_job: Updating cache
[2017-06-01 19:13:06,933: INFO/job_scheduler] Job 1144 completing (errored=False, cancelled=False)
[2017-06-01 19:13:07,035: INFO/job_scheduler] Job 1146 finished 1 steps successfully
[2017-06-01 19:13:07,051: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:13:07,078: INFO/job_scheduler] Job 1146: complete_job: Updating cache
[2017-06-01 19:13:07,078: INFO/job_scheduler] Job 1146 completing (errored=False, cancelled=False)
[2017-06-01 19:13:07,132: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:13:07,155: INFO/job_scheduler] Job 1145 finished 1 steps successfully
[2017-06-01 19:13:07,165: INFO/job_scheduler] Job 1145: complete_job: Updating cache
[2017-06-01 19:13:07,166: INFO/job_scheduler] Job 1145 completing (errored=False, cancelled=False)
[2017-06-01 19:13:07,215: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:13:09,244: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:09,244: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:09,274: INFO/command_plan] add_jobs: created Job 1147 (Test for host connectivity)
[2017-06-01 19:13:09,346: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:13:09,395: INFO/job_scheduler] Job 1147: RunJobThread.run
[2017-06-01 19:13:09,423: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:09,431: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:09,450: INFO/command_plan] add_jobs: created Job 1148 (Test for host connectivity)
[2017-06-01 19:13:09,470: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:13:09,484: INFO/job_scheduler] Job 1148: RunJobThread.run
[2017-06-01 19:13:09,585: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:09,585: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:09,631: INFO/command_plan] add_jobs: created Job 1149 (Test for host connectivity)
[2017-06-01 19:13:09,687: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:13:09,720: INFO/job_scheduler] Job 1149: RunJobThread.run
[2017-06-01 19:13:09,761: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:09,762: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:09,794: INFO/command_plan] add_jobs: created Job 1150 (Test for host connectivity)
[2017-06-01 19:13:09,917: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:13:09,928: INFO/job_scheduler] Job 1150: RunJobThread.run
[2017-06-01 19:13:11,783: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:13:11,831: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:13:12,107: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:13:12,187: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:13:13,125: INFO/job_scheduler] Job 1147 finished 1 steps successfully
[2017-06-01 19:13:13,155: INFO/job_scheduler] Job 1147: complete_job: Updating cache
[2017-06-01 19:13:13,155: INFO/job_scheduler] Job 1147 completing (errored=False, cancelled=False)
[2017-06-01 19:13:13,205: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:13:13,251: INFO/job_scheduler] Job 1148 finished 1 steps successfully
[2017-06-01 19:13:13,314: INFO/job_scheduler] Job 1148: complete_job: Updating cache
[2017-06-01 19:13:13,314: INFO/job_scheduler] Job 1148 completing (errored=False, cancelled=False)
[2017-06-01 19:13:13,388: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:13:13,619: INFO/job_scheduler] Job 1149 finished 1 steps successfully
[2017-06-01 19:13:13,672: INFO/job_scheduler] Job 1149: complete_job: Updating cache
[2017-06-01 19:13:13,672: INFO/job_scheduler] Job 1149 completing (errored=False, cancelled=False)
[2017-06-01 19:13:13,705: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:13:13,741: INFO/job_scheduler] Job 1150 finished 1 steps successfully
[2017-06-01 19:13:13,763: INFO/job_scheduler] Job 1150: complete_job: Updating cache
[2017-06-01 19:13:13,763: INFO/job_scheduler] Job 1150 completing (errored=False, cancelled=False)
[2017-06-01 19:13:13,814: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:13:15,623: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:15,623: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:15,657: INFO/command_plan] add_jobs: created Job 1151 (Test for host connectivity)
[2017-06-01 19:13:15,695: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:13:15,704: INFO/job_scheduler] Job 1151: RunJobThread.run
[2017-06-01 19:13:15,735: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:15,735: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:15,785: INFO/command_plan] add_jobs: created Job 1152 (Test for host connectivity)
[2017-06-01 19:13:15,815: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:13:15,818: INFO/job_scheduler] Job 1152: RunJobThread.run
[2017-06-01 19:13:15,916: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:15,916: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:15,947: INFO/command_plan] add_jobs: created Job 1153 (Test for host connectivity)
[2017-06-01 19:13:16,029: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:13:16,045: INFO/job_scheduler] Job 1153: RunJobThread.run
[2017-06-01 19:13:16,119: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:16,119: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:16,156: INFO/command_plan] add_jobs: created Job 1154 (Test for host connectivity)
[2017-06-01 19:13:16,264: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:13:16,305: INFO/job_scheduler] Job 1154: RunJobThread.run
[2017-06-01 19:13:17,976: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:13:18,077: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:13:18,369: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:13:18,515: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:13:19,278: INFO/job_scheduler] Job 1151 finished 1 steps successfully
[2017-06-01 19:13:19,321: INFO/job_scheduler] Job 1151: complete_job: Updating cache
[2017-06-01 19:13:19,321: INFO/job_scheduler] Job 1151 completing (errored=False, cancelled=False)
[2017-06-01 19:13:19,364: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:13:19,468: INFO/job_scheduler] Job 1152 finished 1 steps successfully
[2017-06-01 19:13:19,513: INFO/job_scheduler] Job 1152: complete_job: Updating cache
[2017-06-01 19:13:19,514: INFO/job_scheduler] Job 1152 completing (errored=False, cancelled=False)
[2017-06-01 19:13:19,572: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:13:19,768: INFO/job_scheduler] Job 1153 finished 1 steps successfully
[2017-06-01 19:13:19,795: INFO/job_scheduler] Job 1153: complete_job: Updating cache
[2017-06-01 19:13:19,795: INFO/job_scheduler] Job 1153 completing (errored=False, cancelled=False)
[2017-06-01 19:13:19,830: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:13:20,034: INFO/job_scheduler] Job 1154 finished 1 steps successfully
[2017-06-01 19:13:20,062: INFO/job_scheduler] Job 1154: complete_job: Updating cache
[2017-06-01 19:13:20,062: INFO/job_scheduler] Job 1154 completing (errored=False, cancelled=False)
[2017-06-01 19:13:20,112: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:13:21,972: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:21,978: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:22,014: INFO/command_plan] add_jobs: created Job 1155 (Test for host connectivity)
[2017-06-01 19:13:22,060: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:13:22,070: INFO/job_scheduler] Job 1155: RunJobThread.run
[2017-06-01 19:13:22,128: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:22,135: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:22,153: INFO/command_plan] add_jobs: created Job 1156 (Test for host connectivity)
[2017-06-01 19:13:22,255: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:13:22,288: INFO/job_scheduler] Job 1156: RunJobThread.run
[2017-06-01 19:13:22,329: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:22,331: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:22,367: INFO/command_plan] add_jobs: created Job 1157 (Test for host connectivity)
[2017-06-01 19:13:22,443: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:13:22,466: INFO/job_scheduler] Job 1157: RunJobThread.run
[2017-06-01 19:13:22,510: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:22,510: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:22,528: INFO/command_plan] add_jobs: created Job 1158 (Test for host connectivity)
[2017-06-01 19:13:22,569: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:13:22,597: INFO/job_scheduler] Job 1158: RunJobThread.run
[2017-06-01 19:13:24,493: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:13:24,520: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:13:24,584: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:13:24,755: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:13:25,812: INFO/job_scheduler] Job 1155 finished 1 steps successfully
[2017-06-01 19:13:25,831: INFO/job_scheduler] Job 1156 finished 1 steps successfully
[2017-06-01 19:13:25,869: INFO/job_scheduler] Job 1155: complete_job: Updating cache
[2017-06-01 19:13:25,869: INFO/job_scheduler] Job 1155 completing (errored=False, cancelled=False)
[2017-06-01 19:13:25,946: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:13:26,006: INFO/job_scheduler] Job 1156: complete_job: Updating cache
[2017-06-01 19:13:26,006: INFO/job_scheduler] Job 1156 completing (errored=False, cancelled=False)
[2017-06-01 19:13:26,103: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:13:26,128: INFO/job_scheduler] Job 1157 finished 1 steps successfully
[2017-06-01 19:13:26,176: INFO/job_scheduler] Job 1157: complete_job: Updating cache
[2017-06-01 19:13:26,178: INFO/job_scheduler] Job 1157 completing (errored=False, cancelled=False)
[2017-06-01 19:13:26,215: INFO/job_scheduler] Job 1158 finished 1 steps successfully
[2017-06-01 19:13:26,336: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:13:26,344: INFO/job_scheduler] Job 1158: complete_job: Updating cache
[2017-06-01 19:13:26,344: INFO/job_scheduler] Job 1158 completing (errored=False, cancelled=False)
[2017-06-01 19:13:26,413: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:13:28,371: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:28,371: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:28,386: INFO/command_plan] add_jobs: created Job 1159 (Test for host connectivity)
[2017-06-01 19:13:28,415: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:13:28,440: INFO/job_scheduler] Job 1159: RunJobThread.run
[2017-06-01 19:13:28,486: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:28,486: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:28,511: INFO/command_plan] add_jobs: created Job 1160 (Test for host connectivity)
[2017-06-01 19:13:28,560: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:13:28,598: INFO/job_scheduler] Job 1160: RunJobThread.run
[2017-06-01 19:13:28,649: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:28,650: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:28,663: INFO/command_plan] add_jobs: created Job 1161 (Test for host connectivity)
[2017-06-01 19:13:28,706: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:13:28,711: INFO/job_scheduler] Job 1161: RunJobThread.run
[2017-06-01 19:13:28,848: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:28,848: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:28,861: INFO/command_plan] add_jobs: created Job 1162 (Test for host connectivity)
[2017-06-01 19:13:28,996: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:13:29,025: INFO/job_scheduler] Job 1162: RunJobThread.run
[2017-06-01 19:13:30,755: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:13:30,913: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:13:31,010: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:13:31,308: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:13:32,085: INFO/job_scheduler] Job 1159 finished 1 steps successfully
[2017-06-01 19:13:32,103: INFO/job_scheduler] Job 1159: complete_job: Updating cache
[2017-06-01 19:13:32,103: INFO/job_scheduler] Job 1159 completing (errored=False, cancelled=False)
[2017-06-01 19:13:32,144: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:13:32,292: INFO/job_scheduler] Job 1160 finished 1 steps successfully
[2017-06-01 19:13:32,345: INFO/job_scheduler] Job 1160: complete_job: Updating cache
[2017-06-01 19:13:32,345: INFO/job_scheduler] Job 1160 completing (errored=False, cancelled=False)
[2017-06-01 19:13:32,385: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:13:32,409: INFO/job_scheduler] Job 1161 finished 1 steps successfully
[2017-06-01 19:13:32,434: INFO/job_scheduler] Job 1161: complete_job: Updating cache
[2017-06-01 19:13:32,434: INFO/job_scheduler] Job 1161 completing (errored=False, cancelled=False)
[2017-06-01 19:13:32,505: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:13:32,763: INFO/job_scheduler] Job 1162 finished 1 steps successfully
[2017-06-01 19:13:32,777: INFO/job_scheduler] Job 1162: complete_job: Updating cache
[2017-06-01 19:13:32,777: INFO/job_scheduler] Job 1162 completing (errored=False, cancelled=False)
[2017-06-01 19:13:32,819: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:13:34,687: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:34,687: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:34,710: INFO/command_plan] add_jobs: created Job 1163 (Test for host connectivity)
[2017-06-01 19:13:34,742: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:13:34,769: INFO/job_scheduler] Job 1163: RunJobThread.run
[2017-06-01 19:13:34,809: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:34,815: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:34,828: INFO/command_plan] add_jobs: created Job 1164 (Test for host connectivity)
[2017-06-01 19:13:34,874: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:13:34,920: INFO/job_scheduler] Job 1164: RunJobThread.run
[2017-06-01 19:13:34,984: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:34,984: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:35,036: INFO/command_plan] add_jobs: created Job 1165 (Test for host connectivity)
[2017-06-01 19:13:35,092: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:13:35,111: INFO/job_scheduler] Job 1165: RunJobThread.run
[2017-06-01 19:13:35,161: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:35,161: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:35,188: INFO/command_plan] add_jobs: created Job 1166 (Test for host connectivity)
[2017-06-01 19:13:35,266: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:13:35,270: INFO/job_scheduler] Job 1166: RunJobThread.run
[2017-06-01 19:13:37,311: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:13:37,345: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:13:37,445: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:13:37,596: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:13:38,615: INFO/job_scheduler] Job 1163 finished 1 steps successfully
[2017-06-01 19:13:38,676: INFO/job_scheduler] Job 1163: complete_job: Updating cache
[2017-06-01 19:13:38,677: INFO/job_scheduler] Job 1163 completing (errored=False, cancelled=False)
[2017-06-01 19:13:38,680: INFO/job_scheduler] Job 1164 finished 1 steps successfully
[2017-06-01 19:13:38,729: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:13:38,744: INFO/job_scheduler] Job 1164: complete_job: Updating cache
[2017-06-01 19:13:38,746: INFO/job_scheduler] Job 1164 completing (errored=False, cancelled=False)
[2017-06-01 19:13:38,776: INFO/job_scheduler] Job 1165 finished 1 steps successfully
[2017-06-01 19:13:38,870: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:13:38,948: INFO/job_scheduler] Job 1165: complete_job: Updating cache
[2017-06-01 19:13:38,948: INFO/job_scheduler] Job 1165 completing (errored=False, cancelled=False)
[2017-06-01 19:13:39,002: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:13:39,097: INFO/job_scheduler] Job 1166 finished 1 steps successfully
[2017-06-01 19:13:39,135: INFO/job_scheduler] Job 1166: complete_job: Updating cache
[2017-06-01 19:13:39,135: INFO/job_scheduler] Job 1166 completing (errored=False, cancelled=False)
[2017-06-01 19:13:39,178: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:13:40,961: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:40,962: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:40,976: INFO/command_plan] add_jobs: created Job 1167 (Test for host connectivity)
[2017-06-01 19:13:41,043: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:13:41,051: INFO/job_scheduler] Job 1167: RunJobThread.run
[2017-06-01 19:13:41,084: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:41,085: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:41,097: INFO/command_plan] add_jobs: created Job 1168 (Test for host connectivity)
[2017-06-01 19:13:41,168: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:13:41,184: INFO/job_scheduler] Job 1168: RunJobThread.run
[2017-06-01 19:13:41,260: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:41,262: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:41,381: INFO/command_plan] add_jobs: created Job 1169 (Test for host connectivity)
[2017-06-01 19:13:41,427: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:13:41,448: INFO/job_scheduler] Job 1169: RunJobThread.run
[2017-06-01 19:13:41,484: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:41,485: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:41,549: INFO/command_plan] add_jobs: created Job 1170 (Test for host connectivity)
[2017-06-01 19:13:41,618: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:13:41,624: INFO/job_scheduler] Job 1170: RunJobThread.run
[2017-06-01 19:13:43,423: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:13:43,563: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:13:43,708: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:13:43,830: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:13:44,761: INFO/job_scheduler] Job 1167 finished 1 steps successfully
[2017-06-01 19:13:44,803: INFO/job_scheduler] Job 1167: complete_job: Updating cache
[2017-06-01 19:13:44,804: INFO/job_scheduler] Job 1167 completing (errored=False, cancelled=False)
[2017-06-01 19:13:44,866: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:13:44,878: INFO/job_scheduler] Job 1168 finished 1 steps successfully
[2017-06-01 19:13:44,893: INFO/job_scheduler] Job 1168: complete_job: Updating cache
[2017-06-01 19:13:44,893: INFO/job_scheduler] Job 1168 completing (errored=False, cancelled=False)
[2017-06-01 19:13:44,996: INFO/job_scheduler] Job 1169 finished 1 steps successfully
[2017-06-01 19:13:44,997: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:13:45,060: INFO/job_scheduler] Job 1169: complete_job: Updating cache
[2017-06-01 19:13:45,060: INFO/job_scheduler] Job 1169 completing (errored=False, cancelled=False)
[2017-06-01 19:13:45,143: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:13:45,265: INFO/job_scheduler] Job 1170 finished 1 steps successfully
[2017-06-01 19:13:45,301: INFO/job_scheduler] Job 1170: complete_job: Updating cache
[2017-06-01 19:13:45,301: INFO/job_scheduler] Job 1170 completing (errored=False, cancelled=False)
[2017-06-01 19:13:45,359: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:13:47,434: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:47,436: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:47,459: INFO/command_plan] add_jobs: created Job 1171 (Test for host connectivity)
[2017-06-01 19:13:47,525: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:13:47,535: INFO/job_scheduler] Job 1171: RunJobThread.run
[2017-06-01 19:13:47,587: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:47,589: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:47,609: INFO/command_plan] add_jobs: created Job 1172 (Test for host connectivity)
[2017-06-01 19:13:47,720: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:47,720: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:47,751: INFO/command_plan] add_jobs: created Job 1173 (Test for host connectivity)
[2017-06-01 19:13:47,771: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:13:47,823: INFO/job_scheduler] Job 1172: RunJobThread.run
[2017-06-01 19:13:47,828: INFO/job_scheduler] Job 1173: RunJobThread.run
[2017-06-01 19:13:47,840: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:13:47,910: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:47,911: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:47,937: INFO/command_plan] add_jobs: created Job 1174 (Test for host connectivity)
[2017-06-01 19:13:48,037: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:13:48,075: INFO/job_scheduler] Job 1174: RunJobThread.run
[2017-06-01 19:13:49,796: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:13:50,091: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:13:50,221: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:13:50,335: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:13:51,127: INFO/job_scheduler] Job 1171 finished 1 steps successfully
[2017-06-01 19:13:51,159: INFO/job_scheduler] Job 1171: complete_job: Updating cache
[2017-06-01 19:13:51,159: INFO/job_scheduler] Job 1171 completing (errored=False, cancelled=False)
[2017-06-01 19:13:51,234: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:13:51,468: INFO/job_scheduler] Job 1172 finished 1 steps successfully
[2017-06-01 19:13:51,500: INFO/job_scheduler] Job 1172: complete_job: Updating cache
[2017-06-01 19:13:51,500: INFO/job_scheduler] Job 1172 completing (errored=False, cancelled=False)
[2017-06-01 19:13:51,544: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:13:51,653: INFO/job_scheduler] Job 1173 finished 1 steps successfully
[2017-06-01 19:13:51,685: INFO/job_scheduler] Job 1173: complete_job: Updating cache
[2017-06-01 19:13:51,685: INFO/job_scheduler] Job 1173 completing (errored=False, cancelled=False)
[2017-06-01 19:13:51,744: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:13:51,802: INFO/job_scheduler] Job 1174 finished 1 steps successfully
[2017-06-01 19:13:51,850: INFO/job_scheduler] Job 1174: complete_job: Updating cache
[2017-06-01 19:13:51,856: INFO/job_scheduler] Job 1174 completing (errored=False, cancelled=False)
[2017-06-01 19:13:51,919: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:13:53,752: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:53,753: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:53,767: INFO/command_plan] add_jobs: created Job 1175 (Test for host connectivity)
[2017-06-01 19:13:53,860: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:53,860: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:53,893: INFO/command_plan] add_jobs: created Job 1176 (Test for host connectivity)
[2017-06-01 19:13:53,933: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:13:53,954: INFO/job_scheduler] Job 1176: RunJobThread.run
[2017-06-01 19:13:53,986: INFO/job_scheduler] Job 1175: RunJobThread.run
[2017-06-01 19:13:54,019: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:54,026: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:54,043: INFO/command_plan] add_jobs: created Job 1177 (Test for host connectivity)
[2017-06-01 19:13:54,061: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:13:54,072: INFO/job_scheduler] Job 1177: RunJobThread.run
[2017-06-01 19:13:54,222: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:13:54,222: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:13:54,233: INFO/command_plan] add_jobs: created Job 1178 (Test for host connectivity)
[2017-06-01 19:13:54,253: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:13:54,273: INFO/job_scheduler] Job 1178: RunJobThread.run
[2017-06-01 19:13:54,356: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:13:56,266: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:13:56,358: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:13:56,396: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:13:56,455: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:13:57,621: INFO/job_scheduler] Job 1175 finished 1 steps successfully
[2017-06-01 19:13:57,649: INFO/job_scheduler] Job 1175: complete_job: Updating cache
[2017-06-01 19:13:57,649: INFO/job_scheduler] Job 1175 completing (errored=False, cancelled=False)
[2017-06-01 19:13:57,724: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:13:57,728: INFO/job_scheduler] Job 1176 finished 1 steps successfully
[2017-06-01 19:13:57,733: INFO/job_scheduler] Job 1178 finished 1 steps successfully
[2017-06-01 19:13:57,769: INFO/job_scheduler] Job 1176: complete_job: Updating cache
[2017-06-01 19:13:57,770: INFO/job_scheduler] Job 1176 completing (errored=False, cancelled=False)
[2017-06-01 19:13:57,855: INFO/job_scheduler] Job 1177 finished 1 steps successfully
[2017-06-01 19:13:57,984: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:13:58,036: INFO/job_scheduler] Job 1178: complete_job: Updating cache
[2017-06-01 19:13:58,036: INFO/job_scheduler] Job 1178 completing (errored=False, cancelled=False)
[2017-06-01 19:13:58,191: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:13:58,201: INFO/job_scheduler] Job 1177: complete_job: Updating cache
[2017-06-01 19:13:58,201: INFO/job_scheduler] Job 1177 completing (errored=False, cancelled=False)
[2017-06-01 19:13:58,268: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:14:00,072: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:00,072: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:00,157: INFO/command_plan] add_jobs: created Job 1179 (Test for host connectivity)
[2017-06-01 19:14:00,188: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:14:00,199: INFO/job_scheduler] Job 1179: RunJobThread.run
[2017-06-01 19:14:00,232: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:00,233: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:00,244: INFO/command_plan] add_jobs: created Job 1180 (Test for host connectivity)
[2017-06-01 19:14:00,306: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:14:00,331: INFO/job_scheduler] Job 1180: RunJobThread.run
[2017-06-01 19:14:00,422: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:00,424: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:00,451: INFO/command_plan] add_jobs: created Job 1181 (Test for host connectivity)
[2017-06-01 19:14:00,513: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:14:00,567: INFO/job_scheduler] Job 1181: RunJobThread.run
[2017-06-01 19:14:00,599: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:00,601: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:00,608: INFO/command_plan] add_jobs: created Job 1182 (Test for host connectivity)
[2017-06-01 19:14:00,661: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:14:00,709: INFO/job_scheduler] Job 1182: RunJobThread.run
[2017-06-01 19:14:02,568: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:14:02,576: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:14:02,868: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:14:03,056: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:14:03,921: INFO/job_scheduler] Job 1179 finished 1 steps successfully
[2017-06-01 19:14:03,959: INFO/job_scheduler] Job 1180 finished 1 steps successfully
[2017-06-01 19:14:03,965: INFO/job_scheduler] Job 1179: complete_job: Updating cache
[2017-06-01 19:14:03,965: INFO/job_scheduler] Job 1179 completing (errored=False, cancelled=False)
[2017-06-01 19:14:04,073: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:14:04,099: INFO/job_scheduler] Job 1180: complete_job: Updating cache
[2017-06-01 19:14:04,099: INFO/job_scheduler] Job 1180 completing (errored=False, cancelled=False)
[2017-06-01 19:14:04,196: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:14:04,369: INFO/job_scheduler] Job 1181 finished 1 steps successfully
[2017-06-01 19:14:04,398: INFO/job_scheduler] Job 1181: complete_job: Updating cache
[2017-06-01 19:14:04,399: INFO/job_scheduler] Job 1181 completing (errored=False, cancelled=False)
[2017-06-01 19:14:04,441: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:14:04,453: INFO/job_scheduler] Job 1182 finished 1 steps successfully
[2017-06-01 19:14:04,501: INFO/job_scheduler] Job 1182: complete_job: Updating cache
[2017-06-01 19:14:04,509: INFO/job_scheduler] Job 1182 completing (errored=False, cancelled=False)
[2017-06-01 19:14:04,549: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:14:05,904: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:05,904: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:05,916: INFO/command_plan] add_jobs: created Job 1183 (Test for host connectivity)
[2017-06-01 19:14:05,988: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:14:05,994: INFO/job_scheduler] Job 1183: RunJobThread.run
[2017-06-01 19:14:06,054: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:06,056: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:06,094: INFO/command_plan] add_jobs: created Job 1184 (Test for host connectivity)
[2017-06-01 19:14:06,179: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:14:06,186: INFO/job_scheduler] Job 1184: RunJobThread.run
[2017-06-01 19:14:06,226: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:06,227: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:06,265: INFO/command_plan] add_jobs: created Job 1185 (Test for host connectivity)
[2017-06-01 19:14:06,344: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:14:06,358: INFO/job_scheduler] Job 1185: RunJobThread.run
[2017-06-01 19:14:06,386: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:06,386: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:06,400: INFO/command_plan] add_jobs: created Job 1186 (Test for host connectivity)
[2017-06-01 19:14:06,445: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:14:06,498: INFO/job_scheduler] Job 1186: RunJobThread.run
[2017-06-01 19:14:08,365: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:14:08,388: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:14:08,612: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:14:08,658: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:14:09,688: INFO/job_scheduler] Job 1183 finished 1 steps successfully
[2017-06-01 19:14:09,756: INFO/job_scheduler] Job 1183: complete_job: Updating cache
[2017-06-01 19:14:09,756: INFO/job_scheduler] Job 1183 completing (errored=False, cancelled=False)
[2017-06-01 19:14:09,790: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:14:09,850: INFO/job_scheduler] Job 1184 finished 1 steps successfully
[2017-06-01 19:14:09,899: INFO/job_scheduler] Job 1184: complete_job: Updating cache
[2017-06-01 19:14:09,899: INFO/job_scheduler] Job 1184 completing (errored=False, cancelled=False)
[2017-06-01 19:14:09,940: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:14:09,997: INFO/job_scheduler] Job 1186 finished 1 steps successfully
[2017-06-01 19:14:10,079: INFO/job_scheduler] Job 1185 finished 1 steps successfully
[2017-06-01 19:14:10,115: INFO/job_scheduler] Job 1186: complete_job: Updating cache
[2017-06-01 19:14:10,115: INFO/job_scheduler] Job 1186 completing (errored=False, cancelled=False)
[2017-06-01 19:14:10,262: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:14:10,281: INFO/job_scheduler] Job 1185: complete_job: Updating cache
[2017-06-01 19:14:10,281: INFO/job_scheduler] Job 1185 completing (errored=False, cancelled=False)
[2017-06-01 19:14:10,317: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:14:12,239: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:12,240: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:12,258: INFO/command_plan] add_jobs: created Job 1187 (Test for host connectivity)
[2017-06-01 19:14:12,292: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:14:12,295: INFO/job_scheduler] Job 1187: RunJobThread.run
[2017-06-01 19:14:12,399: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:12,400: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:12,423: INFO/command_plan] add_jobs: created Job 1188 (Test for host connectivity)
[2017-06-01 19:14:12,440: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:14:12,454: INFO/job_scheduler] Job 1188: RunJobThread.run
[2017-06-01 19:14:12,552: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:12,553: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:12,580: INFO/command_plan] add_jobs: created Job 1189 (Test for host connectivity)
[2017-06-01 19:14:12,666: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:14:12,692: INFO/job_scheduler] Job 1189: RunJobThread.run
[2017-06-01 19:14:12,764: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:12,766: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:12,793: INFO/command_plan] add_jobs: created Job 1190 (Test for host connectivity)
[2017-06-01 19:14:12,868: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:14:12,873: INFO/job_scheduler] Job 1190: RunJobThread.run
[2017-06-01 19:14:14,642: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:14:14,838: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:14:15,010: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:14:15,010: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:14:15,988: INFO/job_scheduler] Job 1187 finished 1 steps successfully
[2017-06-01 19:14:16,023: INFO/job_scheduler] Job 1187: complete_job: Updating cache
[2017-06-01 19:14:16,023: INFO/job_scheduler] Job 1187 completing (errored=False, cancelled=False)
[2017-06-01 19:14:16,095: INFO/job_scheduler] Job 1188 finished 1 steps successfully
[2017-06-01 19:14:16,149: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:14:16,212: INFO/job_scheduler] Job 1188: complete_job: Updating cache
[2017-06-01 19:14:16,212: INFO/job_scheduler] Job 1188 completing (errored=False, cancelled=False)
[2017-06-01 19:14:16,273: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:14:16,376: INFO/job_scheduler] Job 1190 finished 1 steps successfully
[2017-06-01 19:14:16,383: INFO/job_scheduler] Job 1189 finished 1 steps successfully
[2017-06-01 19:14:16,439: INFO/job_scheduler] Job 1190: complete_job: Updating cache
[2017-06-01 19:14:16,439: INFO/job_scheduler] Job 1190 completing (errored=False, cancelled=False)
[2017-06-01 19:14:16,473: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:14:16,528: INFO/job_scheduler] Job 1189: complete_job: Updating cache
[2017-06-01 19:14:16,528: INFO/job_scheduler] Job 1189 completing (errored=False, cancelled=False)
[2017-06-01 19:14:16,647: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:14:18,661: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:18,661: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:18,676: INFO/command_plan] add_jobs: created Job 1191 (Test for host connectivity)
[2017-06-01 19:14:18,742: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:14:18,747: INFO/job_scheduler] Job 1191: RunJobThread.run
[2017-06-01 19:14:18,795: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:18,797: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:18,807: INFO/command_plan] add_jobs: created Job 1192 (Test for host connectivity)
[2017-06-01 19:14:18,846: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:14:18,882: INFO/job_scheduler] Job 1192: RunJobThread.run
[2017-06-01 19:14:18,973: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:18,973: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:18,988: INFO/command_plan] add_jobs: created Job 1193 (Test for host connectivity)
[2017-06-01 19:14:19,051: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:14:19,146: INFO/job_scheduler] Job 1193: RunJobThread.run
[2017-06-01 19:14:19,210: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:19,211: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:19,223: INFO/command_plan] add_jobs: created Job 1194 (Test for host connectivity)
[2017-06-01 19:14:19,256: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:14:19,272: INFO/job_scheduler] Job 1194: RunJobThread.run
[2017-06-01 19:14:21,137: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:14:21,274: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:14:21,395: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:14:21,580: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:14:22,434: INFO/job_scheduler] Job 1191 finished 1 steps successfully
[2017-06-01 19:14:22,463: INFO/job_scheduler] Job 1191: complete_job: Updating cache
[2017-06-01 19:14:22,463: INFO/job_scheduler] Job 1191 completing (errored=False, cancelled=False)
[2017-06-01 19:14:22,532: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:14:22,634: INFO/job_scheduler] Job 1192 finished 1 steps successfully
[2017-06-01 19:14:22,683: INFO/job_scheduler] Job 1192: complete_job: Updating cache
[2017-06-01 19:14:22,683: INFO/job_scheduler] Job 1192 completing (errored=False, cancelled=False)
[2017-06-01 19:14:22,739: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:14:22,816: INFO/job_scheduler] Job 1193 finished 1 steps successfully
[2017-06-01 19:14:22,855: INFO/job_scheduler] Job 1193: complete_job: Updating cache
[2017-06-01 19:14:22,856: INFO/job_scheduler] Job 1193 completing (errored=False, cancelled=False)
[2017-06-01 19:14:22,938: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:14:23,010: INFO/job_scheduler] Job 1194 finished 1 steps successfully
[2017-06-01 19:14:23,030: INFO/job_scheduler] Job 1194: complete_job: Updating cache
[2017-06-01 19:14:23,044: INFO/job_scheduler] Job 1194 completing (errored=False, cancelled=False)
[2017-06-01 19:14:23,130: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:14:25,132: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:25,132: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:25,147: INFO/command_plan] add_jobs: created Job 1195 (Test for host connectivity)
[2017-06-01 19:14:25,230: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:25,230: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:25,238: INFO/command_plan] add_jobs: created Job 1196 (Test for host connectivity)
[2017-06-01 19:14:25,259: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:14:25,318: INFO/job_scheduler] Job 1195: RunJobThread.run
[2017-06-01 19:14:25,324: INFO/job_scheduler] Job 1196: RunJobThread.run
[2017-06-01 19:14:25,340: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:14:25,422: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:25,423: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:25,475: INFO/command_plan] add_jobs: created Job 1197 (Test for host connectivity)
[2017-06-01 19:14:25,576: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:25,577: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:25,709: INFO/command_plan] add_jobs: created Job 1198 (Test for host connectivity)
[2017-06-01 19:14:25,720: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:14:25,733: INFO/job_scheduler] Job 1197: RunJobThread.run
[2017-06-01 19:14:25,734: INFO/job_scheduler] Job 1198: RunJobThread.run
[2017-06-01 19:14:25,742: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:14:27,566: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:14:27,782: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:14:28,046: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:14:28,079: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:14:28,863: INFO/job_scheduler] Job 1195 finished 1 steps successfully
[2017-06-01 19:14:28,882: INFO/job_scheduler] Job 1195: complete_job: Updating cache
[2017-06-01 19:14:28,882: INFO/job_scheduler] Job 1195 completing (errored=False, cancelled=False)
[2017-06-01 19:14:28,940: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:14:29,156: INFO/job_scheduler] Job 1196 finished 1 steps successfully
[2017-06-01 19:14:29,182: INFO/job_scheduler] Job 1196: complete_job: Updating cache
[2017-06-01 19:14:29,182: INFO/job_scheduler] Job 1196 completing (errored=False, cancelled=False)
[2017-06-01 19:14:29,232: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:14:29,448: INFO/job_scheduler] Job 1198 finished 1 steps successfully
[2017-06-01 19:14:29,482: INFO/job_scheduler] Job 1198: complete_job: Updating cache
[2017-06-01 19:14:29,482: INFO/job_scheduler] Job 1198 completing (errored=False, cancelled=False)
[2017-06-01 19:14:29,520: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:14:29,539: INFO/job_scheduler] Job 1197 finished 1 steps successfully
[2017-06-01 19:14:29,583: INFO/job_scheduler] Job 1197: complete_job: Updating cache
[2017-06-01 19:14:29,583: INFO/job_scheduler] Job 1197 completing (errored=False, cancelled=False)
[2017-06-01 19:14:29,668: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:14:31,526: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:31,527: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:31,541: INFO/command_plan] add_jobs: created Job 1199 (Test for host connectivity)
[2017-06-01 19:14:31,594: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:14:31,629: INFO/job_scheduler] Job 1199: RunJobThread.run
[2017-06-01 19:14:31,677: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:31,677: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:31,716: INFO/command_plan] add_jobs: created Job 1200 (Test for host connectivity)
[2017-06-01 19:14:31,826: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:31,828: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:31,840: INFO/command_plan] add_jobs: created Job 1201 (Test for host connectivity)
[2017-06-01 19:14:31,923: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:14:31,939: INFO/job_scheduler] Job 1200: RunJobThread.run
[2017-06-01 19:14:31,948: INFO/job_scheduler] Job 1201: RunJobThread.run
[2017-06-01 19:14:31,995: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:32,002: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:32,023: INFO/command_plan] add_jobs: created Job 1202 (Test for host connectivity)
[2017-06-01 19:14:32,033: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:14:32,041: INFO/job_scheduler] Job 1202: RunJobThread.run
[2017-06-01 19:14:32,150: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:14:33,999: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:14:34,251: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:14:34,298: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:14:34,457: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:14:35,301: INFO/job_scheduler] Job 1199 finished 1 steps successfully
[2017-06-01 19:14:35,355: INFO/job_scheduler] Job 1199: complete_job: Updating cache
[2017-06-01 19:14:35,356: INFO/job_scheduler] Job 1199 completing (errored=False, cancelled=False)
[2017-06-01 19:14:35,506: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:14:35,619: INFO/job_scheduler] Job 1200 finished 1 steps successfully
[2017-06-01 19:14:35,622: INFO/job_scheduler] Job 1202 finished 1 steps successfully
[2017-06-01 19:14:35,680: INFO/job_scheduler] Job 1200: complete_job: Updating cache
[2017-06-01 19:14:35,681: INFO/job_scheduler] Job 1200 completing (errored=False, cancelled=False)
[2017-06-01 19:14:35,715: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:14:35,734: INFO/job_scheduler] Job 1202: complete_job: Updating cache
[2017-06-01 19:14:35,734: INFO/job_scheduler] Job 1202 completing (errored=False, cancelled=False)
[2017-06-01 19:14:35,845: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:14:35,906: INFO/job_scheduler] Job 1201 finished 1 steps successfully
[2017-06-01 19:14:35,981: INFO/job_scheduler] Job 1201: complete_job: Updating cache
[2017-06-01 19:14:35,982: INFO/job_scheduler] Job 1201 completing (errored=False, cancelled=False)
[2017-06-01 19:14:36,040: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:14:37,834: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:37,835: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:37,847: INFO/command_plan] add_jobs: created Job 1203 (Test for host connectivity)
[2017-06-01 19:14:37,914: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:14:37,930: INFO/job_scheduler] Job 1203: RunJobThread.run
[2017-06-01 19:14:37,983: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:37,983: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:38,000: INFO/command_plan] add_jobs: created Job 1204 (Test for host connectivity)
[2017-06-01 19:14:38,102: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:14:38,121: INFO/job_scheduler] Job 1204: RunJobThread.run
[2017-06-01 19:14:38,162: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:38,164: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:38,181: INFO/command_plan] add_jobs: created Job 1205 (Test for host connectivity)
[2017-06-01 19:14:38,219: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:14:38,244: INFO/job_scheduler] Job 1205: RunJobThread.run
[2017-06-01 19:14:38,355: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:38,355: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:38,380: INFO/command_plan] add_jobs: created Job 1206 (Test for host connectivity)
[2017-06-01 19:14:38,414: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:14:38,469: INFO/job_scheduler] Job 1206: RunJobThread.run
[2017-06-01 19:14:40,188: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:14:40,400: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:14:40,571: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:14:40,745: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:14:41,514: INFO/job_scheduler] Job 1203 finished 1 steps successfully
[2017-06-01 19:14:41,546: INFO/job_scheduler] Job 1203: complete_job: Updating cache
[2017-06-01 19:14:41,546: INFO/job_scheduler] Job 1203 completing (errored=False, cancelled=False)
[2017-06-01 19:14:41,637: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:14:41,754: INFO/job_scheduler] Job 1204 finished 1 steps successfully
[2017-06-01 19:14:41,806: INFO/job_scheduler] Job 1204: complete_job: Updating cache
[2017-06-01 19:14:41,807: INFO/job_scheduler] Job 1204 completing (errored=False, cancelled=False)
[2017-06-01 19:14:41,872: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:14:42,020: INFO/job_scheduler] Job 1205 finished 1 steps successfully
[2017-06-01 19:14:42,071: INFO/job_scheduler] Job 1205: complete_job: Updating cache
[2017-06-01 19:14:42,071: INFO/job_scheduler] Job 1205 completing (errored=False, cancelled=False)
[2017-06-01 19:14:42,106: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:14:42,237: INFO/job_scheduler] Job 1206 finished 1 steps successfully
[2017-06-01 19:14:42,279: INFO/job_scheduler] Job 1206: complete_job: Updating cache
[2017-06-01 19:14:42,279: INFO/job_scheduler] Job 1206 completing (errored=False, cancelled=False)
[2017-06-01 19:14:42,322: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:14:44,190: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:44,191: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:44,213: INFO/command_plan] add_jobs: created Job 1207 (Test for host connectivity)
[2017-06-01 19:14:44,245: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:14:44,293: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:44,293: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:44,270: INFO/job_scheduler] Job 1207: RunJobThread.run
[2017-06-01 19:14:44,305: INFO/command_plan] add_jobs: created Job 1208 (Test for host connectivity)
[2017-06-01 19:14:44,350: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:14:44,368: INFO/job_scheduler] Job 1208: RunJobThread.run
[2017-06-01 19:14:44,475: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:44,475: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:44,496: INFO/command_plan] add_jobs: created Job 1209 (Test for host connectivity)
[2017-06-01 19:14:44,575: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:14:44,585: INFO/job_scheduler] Job 1209: RunJobThread.run
[2017-06-01 19:14:44,675: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:44,676: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:44,688: INFO/command_plan] add_jobs: created Job 1210 (Test for host connectivity)
[2017-06-01 19:14:44,782: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:14:44,824: INFO/job_scheduler] Job 1210: RunJobThread.run
[2017-06-01 19:14:46,567: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:14:46,691: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:14:46,923: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:14:46,983: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:14:47,888: INFO/job_scheduler] Job 1207 finished 1 steps successfully
[2017-06-01 19:14:47,928: INFO/job_scheduler] Job 1207: complete_job: Updating cache
[2017-06-01 19:14:47,929: INFO/job_scheduler] Job 1207 completing (errored=False, cancelled=False)
[2017-06-01 19:14:47,995: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:14:48,045: INFO/job_scheduler] Job 1208 finished 1 steps successfully
[2017-06-01 19:14:48,113: INFO/job_scheduler] Job 1208: complete_job: Updating cache
[2017-06-01 19:14:48,114: INFO/job_scheduler] Job 1208 completing (errored=False, cancelled=False)
[2017-06-01 19:14:48,161: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:14:48,359: INFO/job_scheduler] Job 1209 finished 1 steps successfully
[2017-06-01 19:14:48,394: INFO/job_scheduler] Job 1209: complete_job: Updating cache
[2017-06-01 19:14:48,395: INFO/job_scheduler] Job 1209 completing (errored=False, cancelled=False)
[2017-06-01 19:14:48,429: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:14:48,433: INFO/job_scheduler] Job 1210 finished 1 steps successfully
[2017-06-01 19:14:48,452: INFO/job_scheduler] Job 1210: complete_job: Updating cache
[2017-06-01 19:14:48,455: INFO/job_scheduler] Job 1210 completing (errored=False, cancelled=False)
[2017-06-01 19:14:48,507: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:14:50,508: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:50,508: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:50,520: INFO/command_plan] add_jobs: created Job 1211 (Test for host connectivity)
[2017-06-01 19:14:50,558: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:14:50,586: INFO/job_scheduler] Job 1211: RunJobThread.run
[2017-06-01 19:14:50,635: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:50,635: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:50,669: INFO/command_plan] add_jobs: created Job 1212 (Test for host connectivity)
[2017-06-01 19:14:50,771: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:50,771: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:50,794: INFO/command_plan] add_jobs: created Job 1213 (Test for host connectivity)
[2017-06-01 19:14:50,845: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:14:50,861: INFO/job_scheduler] Job 1212: RunJobThread.run
[2017-06-01 19:14:50,866: INFO/job_scheduler] Job 1213: RunJobThread.run
[2017-06-01 19:14:50,882: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:14:50,944: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:50,953: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:50,989: INFO/command_plan] add_jobs: created Job 1214 (Test for host connectivity)
[2017-06-01 19:14:51,081: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:14:51,095: INFO/job_scheduler] Job 1214: RunJobThread.run
[2017-06-01 19:14:52,864: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:14:53,135: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:14:53,194: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:14:53,339: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:14:54,200: INFO/job_scheduler] Job 1211 finished 1 steps successfully
[2017-06-01 19:14:54,234: INFO/job_scheduler] Job 1211: complete_job: Updating cache
[2017-06-01 19:14:54,235: INFO/job_scheduler] Job 1211 completing (errored=False, cancelled=False)
[2017-06-01 19:14:54,351: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:14:54,754: INFO/job_scheduler] Job 1213 finished 1 steps successfully
[2017-06-01 19:14:54,767: INFO/job_scheduler] Job 1214 finished 1 steps successfully
[2017-06-01 19:14:54,775: INFO/job_scheduler] Job 1212 finished 1 steps successfully
[2017-06-01 19:14:54,801: INFO/job_scheduler] Job 1213: complete_job: Updating cache
[2017-06-01 19:14:54,801: INFO/job_scheduler] Job 1213 completing (errored=False, cancelled=False)
[2017-06-01 19:14:54,836: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:14:54,852: INFO/job_scheduler] Job 1214: complete_job: Updating cache
[2017-06-01 19:14:54,859: INFO/job_scheduler] Job 1214 completing (errored=False, cancelled=False)
[2017-06-01 19:14:54,931: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:14:54,960: INFO/job_scheduler] Job 1212: complete_job: Updating cache
[2017-06-01 19:14:54,960: INFO/job_scheduler] Job 1212 completing (errored=False, cancelled=False)
[2017-06-01 19:14:55,002: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:14:56,813: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:56,814: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:56,828: INFO/command_plan] add_jobs: created Job 1215 (Test for host connectivity)
[2017-06-01 19:14:56,913: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:56,914: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:56,926: INFO/command_plan] add_jobs: created Job 1216 (Test for host connectivity)
[2017-06-01 19:14:56,936: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:14:56,963: INFO/job_scheduler] Job 1216: RunJobThread.run
[2017-06-01 19:14:56,969: INFO/job_scheduler] Job 1215: RunJobThread.run
[2017-06-01 19:14:56,997: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:14:57,082: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:57,082: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:57,102: INFO/command_plan] add_jobs: created Job 1217 (Test for host connectivity)
[2017-06-01 19:14:57,148: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:14:57,196: INFO/job_scheduler] Job 1217: RunJobThread.run
[2017-06-01 19:14:57,280: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:14:57,284: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:14:57,307: INFO/command_plan] add_jobs: created Job 1218 (Test for host connectivity)
[2017-06-01 19:14:57,371: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:14:57,414: INFO/job_scheduler] Job 1218: RunJobThread.run
[2017-06-01 19:14:59,226: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:14:59,425: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:14:59,511: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:14:59,687: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:15:00,610: INFO/job_scheduler] Job 1215 finished 1 steps successfully
[2017-06-01 19:15:00,684: INFO/job_scheduler] Job 1215: complete_job: Updating cache
[2017-06-01 19:15:00,684: INFO/job_scheduler] Job 1215 completing (errored=False, cancelled=False)
[2017-06-01 19:15:00,764: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:15:00,774: INFO/job_scheduler] Job 1216 finished 1 steps successfully
[2017-06-01 19:15:00,811: INFO/job_scheduler] Job 1216: complete_job: Updating cache
[2017-06-01 19:15:00,811: INFO/job_scheduler] Job 1216 completing (errored=False, cancelled=False)
[2017-06-01 19:15:00,903: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:15:00,923: INFO/job_scheduler] Job 1217 finished 1 steps successfully
[2017-06-01 19:15:00,998: INFO/job_scheduler] Job 1217: complete_job: Updating cache
[2017-06-01 19:15:00,998: INFO/job_scheduler] Job 1217 completing (errored=False, cancelled=False)
[2017-06-01 19:15:01,052: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:15:01,080: INFO/job_scheduler] Job 1218 finished 1 steps successfully
[2017-06-01 19:15:01,159: INFO/job_scheduler] Job 1218: complete_job: Updating cache
[2017-06-01 19:15:01,159: INFO/job_scheduler] Job 1218 completing (errored=False, cancelled=False)
[2017-06-01 19:15:01,193: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:15:03,132: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:03,134: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:03,142: INFO/command_plan] add_jobs: created Job 1219 (Test for host connectivity)
[2017-06-01 19:15:03,165: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:15:03,187: INFO/job_scheduler] Job 1219: RunJobThread.run
[2017-06-01 19:15:03,267: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:03,269: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:03,275: INFO/command_plan] add_jobs: created Job 1220 (Test for host connectivity)
[2017-06-01 19:15:03,335: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:15:03,389: INFO/job_scheduler] Job 1220: RunJobThread.run
[2017-06-01 19:15:03,432: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:03,433: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:03,458: INFO/command_plan] add_jobs: created Job 1221 (Test for host connectivity)
[2017-06-01 19:15:03,532: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:15:03,540: INFO/job_scheduler] Job 1221: RunJobThread.run
[2017-06-01 19:15:03,593: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:03,593: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:03,635: INFO/command_plan] add_jobs: created Job 1222 (Test for host connectivity)
[2017-06-01 19:15:03,737: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:15:03,751: INFO/job_scheduler] Job 1222: RunJobThread.run
[2017-06-01 19:15:05,606: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:15:05,637: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:15:05,856: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:15:05,989: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:15:06,935: INFO/job_scheduler] Job 1219 finished 1 steps successfully
[2017-06-01 19:15:06,948: INFO/job_scheduler] Job 1220 finished 1 steps successfully
[2017-06-01 19:15:06,983: INFO/job_scheduler] Job 1219: complete_job: Updating cache
[2017-06-01 19:15:06,983: INFO/job_scheduler] Job 1219 completing (errored=False, cancelled=False)
[2017-06-01 19:15:07,049: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:15:07,058: INFO/job_scheduler] Job 1220: complete_job: Updating cache
[2017-06-01 19:15:07,058: INFO/job_scheduler] Job 1220 completing (errored=False, cancelled=False)
[2017-06-01 19:15:07,145: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:15:07,212: INFO/job_scheduler] Job 1221 finished 1 steps successfully
[2017-06-01 19:15:07,253: INFO/job_scheduler] Job 1221: complete_job: Updating cache
[2017-06-01 19:15:07,253: INFO/job_scheduler] Job 1221 completing (errored=False, cancelled=False)
[2017-06-01 19:15:07,320: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:15:07,417: INFO/job_scheduler] Job 1222 finished 1 steps successfully
[2017-06-01 19:15:07,484: INFO/job_scheduler] Job 1222: complete_job: Updating cache
[2017-06-01 19:15:07,484: INFO/job_scheduler] Job 1222 completing (errored=False, cancelled=False)
[2017-06-01 19:15:07,526: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:15:09,525: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:09,527: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:09,552: INFO/command_plan] add_jobs: created Job 1223 (Test for host connectivity)
[2017-06-01 19:15:09,601: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:15:09,611: INFO/job_scheduler] Job 1223: RunJobThread.run
[2017-06-01 19:15:09,676: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:09,676: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:09,684: INFO/command_plan] add_jobs: created Job 1224 (Test for host connectivity)
[2017-06-01 19:15:09,708: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:15:09,712: INFO/job_scheduler] Job 1224: RunJobThread.run
[2017-06-01 19:15:09,830: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:09,831: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:09,843: INFO/command_plan] add_jobs: created Job 1225 (Test for host connectivity)
[2017-06-01 19:15:09,889: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:15:09,901: INFO/job_scheduler] Job 1225: RunJobThread.run
[2017-06-01 19:15:10,024: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:10,025: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:10,042: INFO/command_plan] add_jobs: created Job 1226 (Test for host connectivity)
[2017-06-01 19:15:10,073: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:15:10,104: INFO/job_scheduler] Job 1226: RunJobThread.run
[2017-06-01 19:15:11,930: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:15:12,021: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:15:12,305: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:15:12,438: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:15:13,323: INFO/job_scheduler] Job 1223 finished 1 steps successfully
[2017-06-01 19:15:13,393: INFO/job_scheduler] Job 1223: complete_job: Updating cache
[2017-06-01 19:15:13,393: INFO/job_scheduler] Job 1223 completing (errored=False, cancelled=False)
[2017-06-01 19:15:13,429: INFO/job_scheduler] Job 1224 finished 1 steps successfully
[2017-06-01 19:15:13,575: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:15:13,616: INFO/job_scheduler] Job 1224: complete_job: Updating cache
[2017-06-01 19:15:13,617: INFO/job_scheduler] Job 1224 completing (errored=False, cancelled=False)
[2017-06-01 19:15:13,670: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:15:13,682: INFO/job_scheduler] Job 1225 finished 1 steps successfully
[2017-06-01 19:15:13,703: INFO/job_scheduler] Job 1225: complete_job: Updating cache
[2017-06-01 19:15:13,703: INFO/job_scheduler] Job 1225 completing (errored=False, cancelled=False)
[2017-06-01 19:15:13,808: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:15:13,909: INFO/job_scheduler] Job 1226 finished 1 steps successfully
[2017-06-01 19:15:13,958: INFO/job_scheduler] Job 1226: complete_job: Updating cache
[2017-06-01 19:15:13,958: INFO/job_scheduler] Job 1226 completing (errored=False, cancelled=False)
[2017-06-01 19:15:13,987: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:15:15,854: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:15,854: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:15,866: INFO/command_plan] add_jobs: created Job 1227 (Test for host connectivity)
[2017-06-01 19:15:15,928: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:15:15,951: INFO/job_scheduler] Job 1227: RunJobThread.run
[2017-06-01 19:15:15,983: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:15,983: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:16,012: INFO/command_plan] add_jobs: created Job 1228 (Test for host connectivity)
[2017-06-01 19:15:16,120: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:15:16,139: INFO/job_scheduler] Job 1228: RunJobThread.run
[2017-06-01 19:15:16,191: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:16,191: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:16,218: INFO/command_plan] add_jobs: created Job 1229 (Test for host connectivity)
[2017-06-01 19:15:16,307: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:15:16,312: INFO/job_scheduler] Job 1229: RunJobThread.run
[2017-06-01 19:15:16,349: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:16,350: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:16,359: INFO/command_plan] add_jobs: created Job 1230 (Test for host connectivity)
[2017-06-01 19:15:16,499: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:15:16,501: INFO/job_scheduler] Job 1230: RunJobThread.run
[2017-06-01 19:15:18,244: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:15:18,333: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:15:18,644: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:15:18,697: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:15:19,643: INFO/job_scheduler] Job 1227 finished 1 steps successfully
[2017-06-01 19:15:19,699: INFO/job_scheduler] Job 1228 finished 1 steps successfully
[2017-06-01 19:15:19,723: INFO/job_scheduler] Job 1227: complete_job: Updating cache
[2017-06-01 19:15:19,723: INFO/job_scheduler] Job 1227 completing (errored=False, cancelled=False)
[2017-06-01 19:15:19,815: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:15:19,838: INFO/job_scheduler] Job 1228: complete_job: Updating cache
[2017-06-01 19:15:19,838: INFO/job_scheduler] Job 1228 completing (errored=False, cancelled=False)
[2017-06-01 19:15:19,932: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:15:20,044: INFO/job_scheduler] Job 1229 finished 1 steps successfully
[2017-06-01 19:15:20,056: INFO/job_scheduler] Job 1229: complete_job: Updating cache
[2017-06-01 19:15:20,057: INFO/job_scheduler] Job 1229 completing (errored=False, cancelled=False)
[2017-06-01 19:15:20,061: INFO/job_scheduler] Job 1230 finished 1 steps successfully
[2017-06-01 19:15:20,124: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:15:20,140: INFO/job_scheduler] Job 1230: complete_job: Updating cache
[2017-06-01 19:15:20,141: INFO/job_scheduler] Job 1230 completing (errored=False, cancelled=False)
[2017-06-01 19:15:20,199: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:15:22,344: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:22,344: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:22,365: INFO/command_plan] add_jobs: created Job 1231 (Test for host connectivity)
[2017-06-01 19:15:22,429: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:15:22,435: INFO/job_scheduler] Job 1231: RunJobThread.run
[2017-06-01 19:15:22,496: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:22,497: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:22,532: INFO/command_plan] add_jobs: created Job 1232 (Test for host connectivity)
[2017-06-01 19:15:22,552: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:15:22,572: INFO/job_scheduler] Job 1232: RunJobThread.run
[2017-06-01 19:15:22,642: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:22,642: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:22,673: INFO/command_plan] add_jobs: created Job 1233 (Test for host connectivity)
[2017-06-01 19:15:22,727: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:15:22,773: INFO/job_scheduler] Job 1233: RunJobThread.run
[2017-06-01 19:15:22,835: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:22,836: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:22,867: INFO/command_plan] add_jobs: created Job 1234 (Test for host connectivity)
[2017-06-01 19:15:22,979: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:15:22,984: INFO/job_scheduler] Job 1234: RunJobThread.run
[2017-06-01 19:15:24,770: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:15:25,002: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:15:25,150: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:15:25,255: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:15:26,171: INFO/job_scheduler] Job 1231 finished 1 steps successfully
[2017-06-01 19:15:26,205: INFO/job_scheduler] Job 1231: complete_job: Updating cache
[2017-06-01 19:15:26,206: INFO/job_scheduler] Job 1231 completing (errored=False, cancelled=False)
[2017-06-01 19:15:26,259: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:15:26,431: INFO/job_scheduler] Job 1232 finished 1 steps successfully
[2017-06-01 19:15:26,489: INFO/job_scheduler] Job 1232: complete_job: Updating cache
[2017-06-01 19:15:26,489: INFO/job_scheduler] Job 1232 completing (errored=False, cancelled=False)
[2017-06-01 19:15:26,551: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:15:26,555: INFO/job_scheduler] Job 1233 finished 1 steps successfully
[2017-06-01 19:15:26,599: INFO/job_scheduler] Job 1233: complete_job: Updating cache
[2017-06-01 19:15:26,599: INFO/job_scheduler] Job 1233 completing (errored=False, cancelled=False)
[2017-06-01 19:15:26,663: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:15:27,087: INFO/job_scheduler] Job 1234 finished 1 steps successfully
[2017-06-01 19:15:27,113: INFO/job_scheduler] Job 1234: complete_job: Updating cache
[2017-06-01 19:15:27,114: INFO/job_scheduler] Job 1234 completing (errored=False, cancelled=False)
[2017-06-01 19:15:27,198: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:15:28,746: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:28,746: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:28,780: INFO/command_plan] add_jobs: created Job 1235 (Test for host connectivity)
[2017-06-01 19:15:28,860: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:15:28,873: INFO/job_scheduler] Job 1235: RunJobThread.run
[2017-06-01 19:15:28,902: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:28,905: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:28,920: INFO/command_plan] add_jobs: created Job 1236 (Test for host connectivity)
[2017-06-01 19:15:28,996: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:15:28,999: INFO/job_scheduler] Job 1236: RunJobThread.run
[2017-06-01 19:15:29,147: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:29,148: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:29,180: INFO/command_plan] add_jobs: created Job 1237 (Test for host connectivity)
[2017-06-01 19:15:29,267: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:15:29,269: INFO/job_scheduler] Job 1237: RunJobThread.run
[2017-06-01 19:15:29,321: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:29,323: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:29,366: INFO/command_plan] add_jobs: created Job 1238 (Test for host connectivity)
[2017-06-01 19:15:29,427: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:15:29,438: INFO/job_scheduler] Job 1238: RunJobThread.run
[2017-06-01 19:15:31,171: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:15:31,298: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:15:31,464: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:15:31,723: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:15:32,483: INFO/job_scheduler] Job 1235 finished 1 steps successfully
[2017-06-01 19:15:32,537: INFO/job_scheduler] Job 1235: complete_job: Updating cache
[2017-06-01 19:15:32,537: INFO/job_scheduler] Job 1235 completing (errored=False, cancelled=False)
[2017-06-01 19:15:32,581: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:15:32,742: INFO/job_scheduler] Job 1236 finished 1 steps successfully
[2017-06-01 19:15:32,832: INFO/job_scheduler] Job 1237 finished 1 steps successfully
[2017-06-01 19:15:32,846: INFO/job_scheduler] Job 1236: complete_job: Updating cache
[2017-06-01 19:15:32,846: INFO/job_scheduler] Job 1236 completing (errored=False, cancelled=False)
[2017-06-01 19:15:32,905: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:15:32,943: INFO/job_scheduler] Job 1237: complete_job: Updating cache
[2017-06-01 19:15:32,944: INFO/job_scheduler] Job 1237 completing (errored=False, cancelled=False)
[2017-06-01 19:15:33,048: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:15:33,152: INFO/job_scheduler] Job 1238 finished 1 steps successfully
[2017-06-01 19:15:33,188: INFO/job_scheduler] Job 1238: complete_job: Updating cache
[2017-06-01 19:15:33,188: INFO/job_scheduler] Job 1238 completing (errored=False, cancelled=False)
[2017-06-01 19:15:33,222: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:15:35,820: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:35,822: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:35,845: INFO/command_plan] add_jobs: created Job 1239 (Test for host connectivity)
[2017-06-01 19:15:35,914: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:15:35,927: INFO/job_scheduler] Job 1239: RunJobThread.run
[2017-06-01 19:15:35,985: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:35,985: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:36,025: INFO/command_plan] add_jobs: created Job 1240 (Test for host connectivity)
[2017-06-01 19:15:36,074: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:15:36,081: INFO/job_scheduler] Job 1240: RunJobThread.run
[2017-06-01 19:15:36,190: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:36,192: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:36,213: INFO/command_plan] add_jobs: created Job 1241 (Test for host connectivity)
[2017-06-01 19:15:36,284: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:15:36,338: INFO/job_scheduler] Job 1241: RunJobThread.run
[2017-06-01 19:15:36,393: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:36,395: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:36,406: INFO/command_plan] add_jobs: created Job 1242 (Test for host connectivity)
[2017-06-01 19:15:36,466: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:15:36,503: INFO/job_scheduler] Job 1242: RunJobThread.run
[2017-06-01 19:15:38,427: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:15:38,433: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:15:38,533: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:15:38,665: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:15:39,797: INFO/job_scheduler] Job 1239 finished 1 steps successfully
[2017-06-01 19:15:39,830: INFO/job_scheduler] Job 1240 finished 1 steps successfully
[2017-06-01 19:15:39,836: INFO/job_scheduler] Job 1239: complete_job: Updating cache
[2017-06-01 19:15:39,836: INFO/job_scheduler] Job 1239 completing (errored=False, cancelled=False)
[2017-06-01 19:15:39,859: INFO/job_scheduler] Job 1241 finished 1 steps successfully
[2017-06-01 19:15:39,892: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:15:39,902: INFO/job_scheduler] Job 1240: complete_job: Updating cache
[2017-06-01 19:15:39,903: INFO/job_scheduler] Job 1240 completing (errored=False, cancelled=False)
[2017-06-01 19:15:39,979: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:15:40,003: INFO/job_scheduler] Job 1241: complete_job: Updating cache
[2017-06-01 19:15:40,003: INFO/job_scheduler] Job 1241 completing (errored=False, cancelled=False)
[2017-06-01 19:15:40,051: INFO/job_scheduler] Job 1242 finished 1 steps successfully
[2017-06-01 19:15:40,095: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:15:40,114: INFO/job_scheduler] Job 1242: complete_job: Updating cache
[2017-06-01 19:15:40,114: INFO/job_scheduler] Job 1242 completing (errored=False, cancelled=False)
[2017-06-01 19:15:40,245: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:15:42,110: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:42,110: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:42,145: INFO/command_plan] add_jobs: created Job 1243 (Test for host connectivity)
[2017-06-01 19:15:42,212: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:42,213: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:42,228: INFO/command_plan] add_jobs: created Job 1244 (Test for host connectivity)
[2017-06-01 19:15:42,238: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:15:42,302: INFO/job_scheduler] Job 1243: RunJobThread.run
[2017-06-01 19:15:42,346: INFO/job_scheduler] Job 1244: RunJobThread.run
[2017-06-01 19:15:42,397: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:42,399: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:42,420: INFO/command_plan] add_jobs: created Job 1245 (Test for host connectivity)
[2017-06-01 19:15:42,439: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:15:42,493: INFO/job_scheduler] Job 1245: RunJobThread.run
[2017-06-01 19:15:42,585: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:42,585: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:42,611: INFO/command_plan] add_jobs: created Job 1246 (Test for host connectivity)
[2017-06-01 19:15:42,640: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:15:42,647: INFO/job_scheduler] Job 1246: RunJobThread.run
[2017-06-01 19:15:42,701: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:15:44,553: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:15:44,616: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:15:44,875: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:15:44,899: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:15:45,896: INFO/job_scheduler] Job 1243 finished 1 steps successfully
[2017-06-01 19:15:45,976: INFO/job_scheduler] Job 1243: complete_job: Updating cache
[2017-06-01 19:15:45,976: INFO/job_scheduler] Job 1243 completing (errored=False, cancelled=False)
[2017-06-01 19:15:45,980: INFO/job_scheduler] Job 1244 finished 1 steps successfully
[2017-06-01 19:15:46,047: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:15:46,107: INFO/job_scheduler] Job 1244: complete_job: Updating cache
[2017-06-01 19:15:46,108: INFO/job_scheduler] Job 1244 completing (errored=False, cancelled=False)
[2017-06-01 19:15:46,153: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:15:46,251: INFO/job_scheduler] Job 1246 finished 1 steps successfully
[2017-06-01 19:15:46,276: INFO/job_scheduler] Job 1246: complete_job: Updating cache
[2017-06-01 19:15:46,276: INFO/job_scheduler] Job 1246 completing (errored=False, cancelled=False)
[2017-06-01 19:15:46,327: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:15:46,352: INFO/job_scheduler] Job 1245 finished 1 steps successfully
[2017-06-01 19:15:46,368: INFO/job_scheduler] Job 1245: complete_job: Updating cache
[2017-06-01 19:15:46,368: INFO/job_scheduler] Job 1245 completing (errored=False, cancelled=False)
[2017-06-01 19:15:46,416: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:15:48,346: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:48,346: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:48,409: INFO/command_plan] add_jobs: created Job 1247 (Test for host connectivity)
[2017-06-01 19:15:48,454: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:15:48,475: INFO/job_scheduler] Job 1247: RunJobThread.run
[2017-06-01 19:15:48,617: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:48,619: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:48,670: INFO/command_plan] add_jobs: created Job 1248 (Test for host connectivity)
[2017-06-01 19:15:48,779: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:48,780: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:48,806: INFO/command_plan] add_jobs: created Job 1249 (Test for host connectivity)
[2017-06-01 19:15:48,835: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:15:48,884: INFO/job_scheduler] Job 1248: RunJobThread.run
[2017-06-01 19:15:48,884: INFO/job_scheduler] Job 1249: RunJobThread.run
[2017-06-01 19:15:48,979: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:48,979: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:49,034: INFO/command_plan] add_jobs: created Job 1250 (Test for host connectivity)
[2017-06-01 19:15:49,053: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:15:49,059: INFO/job_scheduler] Job 1250: RunJobThread.run
[2017-06-01 19:15:49,159: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:15:50,762: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:15:51,199: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:15:51,291: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:15:51,292: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:15:52,205: INFO/job_scheduler] Job 1247 finished 1 steps successfully
[2017-06-01 19:15:52,251: INFO/job_scheduler] Job 1247: complete_job: Updating cache
[2017-06-01 19:15:52,251: INFO/job_scheduler] Job 1247 completing (errored=False, cancelled=False)
[2017-06-01 19:15:52,325: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:15:52,546: INFO/job_scheduler] Job 1250 finished 1 steps successfully
[2017-06-01 19:15:52,547: INFO/job_scheduler] Job 1249 finished 1 steps successfully
[2017-06-01 19:15:52,592: INFO/job_scheduler] Job 1250: complete_job: Updating cache
[2017-06-01 19:15:52,592: INFO/job_scheduler] Job 1250 completing (errored=False, cancelled=False)
[2017-06-01 19:15:52,626: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:15:52,674: INFO/job_scheduler] Job 1249: complete_job: Updating cache
[2017-06-01 19:15:52,674: INFO/job_scheduler] Job 1249 completing (errored=False, cancelled=False)
[2017-06-01 19:15:52,699: INFO/job_scheduler] Job 1248 finished 1 steps successfully
[2017-06-01 19:15:52,756: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:15:52,798: INFO/job_scheduler] Job 1248: complete_job: Updating cache
[2017-06-01 19:15:52,799: INFO/job_scheduler] Job 1248 completing (errored=False, cancelled=False)
[2017-06-01 19:15:52,862: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:15:54,801: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:54,802: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:54,817: INFO/command_plan] add_jobs: created Job 1251 (Test for host connectivity)
[2017-06-01 19:15:54,882: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:15:54,897: INFO/job_scheduler] Job 1251: RunJobThread.run
[2017-06-01 19:15:54,947: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:54,947: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:54,967: INFO/command_plan] add_jobs: created Job 1252 (Test for host connectivity)
[2017-06-01 19:15:55,051: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:15:55,072: INFO/job_scheduler] Job 1252: RunJobThread.run
[2017-06-01 19:15:55,142: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:55,143: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:55,175: INFO/command_plan] add_jobs: created Job 1253 (Test for host connectivity)
[2017-06-01 19:15:55,291: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:15:55,291: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:15:55,310: INFO/command_plan] add_jobs: created Job 1254 (Test for host connectivity)
[2017-06-01 19:15:55,324: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:15:55,326: INFO/job_scheduler] Job 1253: RunJobThread.run
[2017-06-01 19:15:55,327: INFO/job_scheduler] Job 1254: RunJobThread.run
[2017-06-01 19:15:55,337: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:15:57,261: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:15:57,346: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:15:57,613: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:15:57,643: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:15:58,588: INFO/job_scheduler] Job 1251 finished 1 steps successfully
[2017-06-01 19:15:58,651: INFO/job_scheduler] Job 1251: complete_job: Updating cache
[2017-06-01 19:15:58,651: INFO/job_scheduler] Job 1251 completing (errored=False, cancelled=False)
[2017-06-01 19:15:58,706: INFO/job_scheduler] Job 1252 finished 1 steps successfully
[2017-06-01 19:15:58,776: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:15:58,855: INFO/job_scheduler] Job 1252: complete_job: Updating cache
[2017-06-01 19:15:58,855: INFO/job_scheduler] Job 1252 completing (errored=False, cancelled=False)
[2017-06-01 19:15:58,918: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:15:58,999: INFO/job_scheduler] Job 1254 finished 1 steps successfully
[2017-06-01 19:15:59,006: INFO/job_scheduler] Job 1253 finished 1 steps successfully
[2017-06-01 19:15:59,059: INFO/job_scheduler] Job 1254: complete_job: Updating cache
[2017-06-01 19:15:59,059: INFO/job_scheduler] Job 1254 completing (errored=False, cancelled=False)
[2017-06-01 19:15:59,102: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:15:59,123: INFO/job_scheduler] Job 1253: complete_job: Updating cache
[2017-06-01 19:15:59,123: INFO/job_scheduler] Job 1253 completing (errored=False, cancelled=False)
[2017-06-01 19:15:59,230: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:16:01,029: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:01,031: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:01,042: INFO/command_plan] add_jobs: created Job 1255 (Test for host connectivity)
[2017-06-01 19:16:01,055: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:16:01,058: INFO/job_scheduler] Job 1255: RunJobThread.run
[2017-06-01 19:16:01,154: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:01,154: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:01,177: INFO/command_plan] add_jobs: created Job 1256 (Test for host connectivity)
[2017-06-01 19:16:01,268: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:16:01,305: INFO/job_scheduler] Job 1256: RunJobThread.run
[2017-06-01 19:16:01,344: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:01,344: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:01,373: INFO/command_plan] add_jobs: created Job 1257 (Test for host connectivity)
[2017-06-01 19:16:01,447: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:16:01,468: INFO/job_scheduler] Job 1257: RunJobThread.run
[2017-06-01 19:16:01,550: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:01,560: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:01,630: INFO/command_plan] add_jobs: created Job 1258 (Test for host connectivity)
[2017-06-01 19:16:01,713: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:16:01,752: INFO/job_scheduler] Job 1258: RunJobThread.run
[2017-06-01 19:16:03,435: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:16:03,608: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:16:03,722: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:16:04,047: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:16:04,939: INFO/job_scheduler] Job 1255 finished 1 steps successfully
[2017-06-01 19:16:04,975: INFO/job_scheduler] Job 1255: complete_job: Updating cache
[2017-06-01 19:16:04,975: INFO/job_scheduler] Job 1255 completing (errored=False, cancelled=False)
[2017-06-01 19:16:05,057: INFO/job_scheduler] Job 1256 finished 1 steps successfully
[2017-06-01 19:16:05,057: INFO/job_scheduler] Job 1257 finished 1 steps successfully
[2017-06-01 19:16:05,075: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:16:05,100: INFO/job_scheduler] Job 1256: complete_job: Updating cache
[2017-06-01 19:16:05,100: INFO/job_scheduler] Job 1256 completing (errored=False, cancelled=False)
[2017-06-01 19:16:05,146: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:16:05,162: INFO/job_scheduler] Job 1257: complete_job: Updating cache
[2017-06-01 19:16:05,162: INFO/job_scheduler] Job 1257 completing (errored=False, cancelled=False)
[2017-06-01 19:16:05,218: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:16:05,391: INFO/job_scheduler] Job 1258 finished 1 steps successfully
[2017-06-01 19:16:05,441: INFO/job_scheduler] Job 1258: complete_job: Updating cache
[2017-06-01 19:16:05,441: INFO/job_scheduler] Job 1258 completing (errored=False, cancelled=False)
[2017-06-01 19:16:05,483: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:16:07,396: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:07,396: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:07,408: INFO/command_plan] add_jobs: created Job 1259 (Test for host connectivity)
[2017-06-01 19:16:07,464: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:16:07,477: INFO/job_scheduler] Job 1259: RunJobThread.run
[2017-06-01 19:16:07,537: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:07,547: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:07,557: INFO/command_plan] add_jobs: created Job 1260 (Test for host connectivity)
[2017-06-01 19:16:07,712: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:07,713: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:07,733: INFO/command_plan] add_jobs: created Job 1261 (Test for host connectivity)
[2017-06-01 19:16:07,782: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:16:07,808: INFO/job_scheduler] Job 1260: RunJobThread.run
[2017-06-01 19:16:07,828: INFO/job_scheduler] Job 1261: RunJobThread.run
[2017-06-01 19:16:07,838: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:16:07,880: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:07,880: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:07,892: INFO/command_plan] add_jobs: created Job 1262 (Test for host connectivity)
[2017-06-01 19:16:07,944: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:16:08,030: INFO/job_scheduler] Job 1262: RunJobThread.run
[2017-06-01 19:16:09,854: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:16:10,040: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:16:10,073: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:16:10,161: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:16:11,206: INFO/job_scheduler] Job 1259 finished 1 steps successfully
[2017-06-01 19:16:11,265: INFO/job_scheduler] Job 1259: complete_job: Updating cache
[2017-06-01 19:16:11,265: INFO/job_scheduler] Job 1259 completing (errored=False, cancelled=False)
[2017-06-01 19:16:11,301: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:16:11,381: INFO/job_scheduler] Job 1261 finished 1 steps successfully
[2017-06-01 19:16:11,431: INFO/job_scheduler] Job 1260 finished 1 steps successfully
[2017-06-01 19:16:11,445: INFO/job_scheduler] Job 1261: complete_job: Updating cache
[2017-06-01 19:16:11,445: INFO/job_scheduler] Job 1261 completing (errored=False, cancelled=False)
[2017-06-01 19:16:11,462: INFO/job_scheduler] Job 1262 finished 1 steps successfully
[2017-06-01 19:16:11,490: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:16:11,508: INFO/job_scheduler] Job 1260: complete_job: Updating cache
[2017-06-01 19:16:11,508: INFO/job_scheduler] Job 1260 completing (errored=False, cancelled=False)
[2017-06-01 19:16:11,607: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:16:11,624: INFO/job_scheduler] Job 1262: complete_job: Updating cache
[2017-06-01 19:16:11,625: INFO/job_scheduler] Job 1262 completing (errored=False, cancelled=False)
[2017-06-01 19:16:11,695: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:16:13,693: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:13,693: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:13,709: INFO/command_plan] add_jobs: created Job 1263 (Test for host connectivity)
[2017-06-01 19:16:13,778: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:16:13,803: INFO/job_scheduler] Job 1263: RunJobThread.run
[2017-06-01 19:16:13,840: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:13,840: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:13,869: INFO/command_plan] add_jobs: created Job 1264 (Test for host connectivity)
[2017-06-01 19:16:14,015: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:16:14,019: INFO/job_scheduler] Job 1264: RunJobThread.run
[2017-06-01 19:16:14,085: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:14,085: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:14,100: INFO/command_plan] add_jobs: created Job 1265 (Test for host connectivity)
[2017-06-01 19:16:14,131: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:16:14,160: INFO/job_scheduler] Job 1265: RunJobThread.run
[2017-06-01 19:16:14,195: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:14,195: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:14,236: INFO/command_plan] add_jobs: created Job 1266 (Test for host connectivity)
[2017-06-01 19:16:14,259: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:16:14,278: INFO/job_scheduler] Job 1266: RunJobThread.run
[2017-06-01 19:16:16,262: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:16:16,298: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:16:16,568: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:16:16,607: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:16:17,606: INFO/job_scheduler] Job 1263 finished 1 steps successfully
[2017-06-01 19:16:17,639: INFO/job_scheduler] Job 1264 finished 1 steps successfully
[2017-06-01 19:16:17,664: INFO/job_scheduler] Job 1263: complete_job: Updating cache
[2017-06-01 19:16:17,664: INFO/job_scheduler] Job 1263 completing (errored=False, cancelled=False)
[2017-06-01 19:16:17,748: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:16:17,771: INFO/job_scheduler] Job 1264: complete_job: Updating cache
[2017-06-01 19:16:17,771: INFO/job_scheduler] Job 1264 completing (errored=False, cancelled=False)
[2017-06-01 19:16:17,914: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:16:18,004: INFO/job_scheduler] Job 1265 finished 1 steps successfully
[2017-06-01 19:16:18,005: INFO/job_scheduler] Job 1266 finished 1 steps successfully
[2017-06-01 19:16:18,049: INFO/job_scheduler] Job 1265: complete_job: Updating cache
[2017-06-01 19:16:18,050: INFO/job_scheduler] Job 1265 completing (errored=False, cancelled=False)
[2017-06-01 19:16:18,097: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:16:18,107: INFO/job_scheduler] Job 1266: complete_job: Updating cache
[2017-06-01 19:16:18,107: INFO/job_scheduler] Job 1266 completing (errored=False, cancelled=False)
[2017-06-01 19:16:18,147: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:16:20,178: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:20,178: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:20,209: INFO/command_plan] add_jobs: created Job 1267 (Test for host connectivity)
[2017-06-01 19:16:20,268: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:16:20,292: INFO/job_scheduler] Job 1267: RunJobThread.run
[2017-06-01 19:16:20,343: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:20,344: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:20,378: INFO/command_plan] add_jobs: created Job 1268 (Test for host connectivity)
[2017-06-01 19:16:20,411: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:16:20,467: INFO/job_scheduler] Job 1268: RunJobThread.run
[2017-06-01 19:16:20,522: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:20,522: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:20,530: INFO/command_plan] add_jobs: created Job 1269 (Test for host connectivity)
[2017-06-01 19:16:20,578: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:16:20,622: INFO/job_scheduler] Job 1269: RunJobThread.run
[2017-06-01 19:16:20,720: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:20,726: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:20,818: INFO/command_plan] add_jobs: created Job 1270 (Test for host connectivity)
[2017-06-01 19:16:20,841: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:16:20,850: INFO/job_scheduler] Job 1270: RunJobThread.run
[2017-06-01 19:16:22,545: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:16:22,762: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:16:22,918: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:16:23,121: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:16:23,870: INFO/job_scheduler] Job 1267 finished 1 steps successfully
[2017-06-01 19:16:23,947: INFO/job_scheduler] Job 1267: complete_job: Updating cache
[2017-06-01 19:16:23,948: INFO/job_scheduler] Job 1267 completing (errored=False, cancelled=False)
[2017-06-01 19:16:24,021: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:16:24,133: INFO/job_scheduler] Job 1268 finished 1 steps successfully
[2017-06-01 19:16:24,197: INFO/job_scheduler] Job 1268: complete_job: Updating cache
[2017-06-01 19:16:24,199: INFO/job_scheduler] Job 1268 completing (errored=False, cancelled=False)
[2017-06-01 19:16:24,271: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:16:24,277: INFO/job_scheduler] Job 1269 finished 1 steps successfully
[2017-06-01 19:16:24,305: INFO/job_scheduler] Job 1269: complete_job: Updating cache
[2017-06-01 19:16:24,305: INFO/job_scheduler] Job 1269 completing (errored=False, cancelled=False)
[2017-06-01 19:16:24,368: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:16:24,560: INFO/job_scheduler] Job 1270 finished 1 steps successfully
[2017-06-01 19:16:24,570: INFO/job_scheduler] Job 1270: complete_job: Updating cache
[2017-06-01 19:16:24,571: INFO/job_scheduler] Job 1270 completing (errored=False, cancelled=False)
[2017-06-01 19:16:24,604: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:16:26,558: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:26,560: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:26,592: INFO/command_plan] add_jobs: created Job 1271 (Test for host connectivity)
[2017-06-01 19:16:26,649: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:16:26,656: INFO/job_scheduler] Job 1271: RunJobThread.run
[2017-06-01 19:16:26,746: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:26,748: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:26,764: INFO/command_plan] add_jobs: created Job 1272 (Test for host connectivity)
[2017-06-01 19:16:26,825: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:16:26,831: INFO/job_scheduler] Job 1272: RunJobThread.run
[2017-06-01 19:16:26,897: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:26,899: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:26,931: INFO/command_plan] add_jobs: created Job 1273 (Test for host connectivity)
[2017-06-01 19:16:26,947: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:16:26,949: INFO/job_scheduler] Job 1273: RunJobThread.run
[2017-06-01 19:16:27,088: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:27,089: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:27,117: INFO/command_plan] add_jobs: created Job 1274 (Test for host connectivity)
[2017-06-01 19:16:27,172: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:16:27,196: INFO/job_scheduler] Job 1274: RunJobThread.run
[2017-06-01 19:16:29,042: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:16:29,124: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:16:29,232: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:16:29,439: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:16:30,421: INFO/job_scheduler] Job 1271 finished 1 steps successfully
[2017-06-01 19:16:30,473: INFO/job_scheduler] Job 1271: complete_job: Updating cache
[2017-06-01 19:16:30,473: INFO/job_scheduler] Job 1271 completing (errored=False, cancelled=False)
[2017-06-01 19:16:30,479: INFO/job_scheduler] Job 1273 finished 1 steps successfully
[2017-06-01 19:16:30,506: INFO/job_scheduler] Job 1272 finished 1 steps successfully
[2017-06-01 19:16:30,612: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:16:30,645: INFO/job_scheduler] Job 1273: complete_job: Updating cache
[2017-06-01 19:16:30,647: INFO/job_scheduler] Job 1273 completing (errored=False, cancelled=False)
[2017-06-01 19:16:30,756: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:16:30,781: INFO/job_scheduler] Job 1274 finished 1 steps successfully
[2017-06-01 19:16:30,807: INFO/job_scheduler] Job 1272: complete_job: Updating cache
[2017-06-01 19:16:30,815: INFO/job_scheduler] Job 1272 completing (errored=False, cancelled=False)
[2017-06-01 19:16:30,905: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:16:30,945: INFO/job_scheduler] Job 1274: complete_job: Updating cache
[2017-06-01 19:16:30,945: INFO/job_scheduler] Job 1274 completing (errored=False, cancelled=False)
[2017-06-01 19:16:31,033: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:16:32,837: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:32,839: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:32,853: INFO/command_plan] add_jobs: created Job 1275 (Test for host connectivity)
[2017-06-01 19:16:32,922: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:16:32,945: INFO/job_scheduler] Job 1275: RunJobThread.run
[2017-06-01 19:16:32,978: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:32,979: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:33,005: INFO/command_plan] add_jobs: created Job 1276 (Test for host connectivity)
[2017-06-01 19:16:33,112: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:33,113: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:33,147: INFO/command_plan] add_jobs: created Job 1277 (Test for host connectivity)
[2017-06-01 19:16:33,173: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:16:33,219: INFO/job_scheduler] Job 1276: RunJobThread.run
[2017-06-01 19:16:33,238: INFO/job_scheduler] Job 1277: RunJobThread.run
[2017-06-01 19:16:33,297: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:33,297: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:33,330: INFO/command_plan] add_jobs: created Job 1278 (Test for host connectivity)
[2017-06-01 19:16:33,438: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:16:33,499: INFO/job_scheduler] Job 1278: RunJobThread.run
[2017-06-01 19:16:33,536: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:16:35,323: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:16:35,563: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:16:35,589: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:16:35,717: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:16:36,685: INFO/job_scheduler] Job 1275 finished 1 steps successfully
[2017-06-01 19:16:36,768: INFO/job_scheduler] Job 1275: complete_job: Updating cache
[2017-06-01 19:16:36,768: INFO/job_scheduler] Job 1275 completing (errored=False, cancelled=False)
[2017-06-01 19:16:36,845: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:16:37,036: INFO/job_scheduler] Job 1276 finished 1 steps successfully
[2017-06-01 19:16:37,060: INFO/job_scheduler] Job 1276: complete_job: Updating cache
[2017-06-01 19:16:37,060: INFO/job_scheduler] Job 1276 completing (errored=False, cancelled=False)
[2017-06-01 19:16:37,093: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:16:37,126: INFO/job_scheduler] Job 1278 finished 1 steps successfully
[2017-06-01 19:16:37,127: INFO/job_scheduler] Job 1277 finished 1 steps successfully
[2017-06-01 19:16:37,210: INFO/job_scheduler] Job 1278: complete_job: Updating cache
[2017-06-01 19:16:37,210: INFO/job_scheduler] Job 1278 completing (errored=False, cancelled=False)
[2017-06-01 19:16:37,277: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:16:37,279: INFO/job_scheduler] Job 1277: complete_job: Updating cache
[2017-06-01 19:16:37,279: INFO/job_scheduler] Job 1277 completing (errored=False, cancelled=False)
[2017-06-01 19:16:37,394: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:16:39,151: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:39,153: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:39,159: INFO/command_plan] add_jobs: created Job 1279 (Test for host connectivity)
[2017-06-01 19:16:39,215: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:16:39,238: INFO/job_scheduler] Job 1279: RunJobThread.run
[2017-06-01 19:16:39,265: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:39,265: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:39,277: INFO/command_plan] add_jobs: created Job 1280 (Test for host connectivity)
[2017-06-01 19:16:39,296: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:16:39,330: INFO/job_scheduler] Job 1280: RunJobThread.run
[2017-06-01 19:16:39,472: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:39,474: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:39,511: INFO/command_plan] add_jobs: created Job 1281 (Test for host connectivity)
[2017-06-01 19:16:39,584: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:16:39,599: INFO/job_scheduler] Job 1281: RunJobThread.run
[2017-06-01 19:16:39,638: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:39,639: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:39,659: INFO/command_plan] add_jobs: created Job 1282 (Test for host connectivity)
[2017-06-01 19:16:39,670: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:16:39,672: INFO/job_scheduler] Job 1282: RunJobThread.run
[2017-06-01 19:16:41,730: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:16:41,755: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:16:41,978: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:16:42,014: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:16:43,116: INFO/job_scheduler] Job 1280 finished 1 steps successfully
[2017-06-01 19:16:43,116: INFO/job_scheduler] Job 1279 finished 1 steps successfully
[2017-06-01 19:16:43,175: INFO/job_scheduler] Job 1280: complete_job: Updating cache
[2017-06-01 19:16:43,175: INFO/job_scheduler] Job 1280 completing (errored=False, cancelled=False)
[2017-06-01 19:16:43,210: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:16:43,278: INFO/job_scheduler] Job 1279: complete_job: Updating cache
[2017-06-01 19:16:43,279: INFO/job_scheduler] Job 1279 completing (errored=False, cancelled=False)
[2017-06-01 19:16:43,334: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:16:43,412: INFO/job_scheduler] Job 1282 finished 1 steps successfully
[2017-06-01 19:16:43,413: INFO/job_scheduler] Job 1281 finished 1 steps successfully
[2017-06-01 19:16:43,491: INFO/job_scheduler] Job 1282: complete_job: Updating cache
[2017-06-01 19:16:43,491: INFO/job_scheduler] Job 1282 completing (errored=False, cancelled=False)
[2017-06-01 19:16:43,558: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:16:43,579: INFO/job_scheduler] Job 1281: complete_job: Updating cache
[2017-06-01 19:16:43,580: INFO/job_scheduler] Job 1281 completing (errored=False, cancelled=False)
[2017-06-01 19:16:43,701: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:16:45,363: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:45,363: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:45,383: INFO/command_plan] add_jobs: created Job 1283 (Test for host connectivity)
[2017-06-01 19:16:45,423: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:16:45,432: INFO/job_scheduler] Job 1283: RunJobThread.run
[2017-06-01 19:16:45,488: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:45,489: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:45,505: INFO/command_plan] add_jobs: created Job 1284 (Test for host connectivity)
[2017-06-01 19:16:45,554: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:16:45,568: INFO/job_scheduler] Job 1284: RunJobThread.run
[2017-06-01 19:16:45,640: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:45,641: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:45,658: INFO/command_plan] add_jobs: created Job 1285 (Test for host connectivity)
[2017-06-01 19:16:45,748: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:16:45,797: INFO/job_scheduler] Job 1285: RunJobThread.run
[2017-06-01 19:16:45,871: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:45,872: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:45,886: INFO/command_plan] add_jobs: created Job 1286 (Test for host connectivity)
[2017-06-01 19:16:45,922: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:16:45,950: INFO/job_scheduler] Job 1286: RunJobThread.run
[2017-06-01 19:16:47,930: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:16:48,030: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:16:48,161: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:16:48,169: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:16:49,288: INFO/job_scheduler] Job 1284 finished 1 steps successfully
[2017-06-01 19:16:49,349: INFO/job_scheduler] Job 1284: complete_job: Updating cache
[2017-06-01 19:16:49,349: INFO/job_scheduler] Job 1284 completing (errored=False, cancelled=False)
[2017-06-01 19:16:49,373: INFO/job_scheduler] Job 1283 finished 1 steps successfully
[2017-06-01 19:16:49,382: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:16:49,450: INFO/job_scheduler] Job 1283: complete_job: Updating cache
[2017-06-01 19:16:49,451: INFO/job_scheduler] Job 1283 completing (errored=False, cancelled=False)
[2017-06-01 19:16:49,493: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:16:49,496: INFO/job_scheduler] Job 1286 finished 1 steps successfully
[2017-06-01 19:16:49,566: INFO/job_scheduler] Job 1286: complete_job: Updating cache
[2017-06-01 19:16:49,567: INFO/job_scheduler] Job 1286 completing (errored=False, cancelled=False)
[2017-06-01 19:16:49,693: INFO/job_scheduler] Job 1285 finished 1 steps successfully
[2017-06-01 19:16:49,732: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:16:49,766: INFO/job_scheduler] Job 1285: complete_job: Updating cache
[2017-06-01 19:16:49,766: INFO/job_scheduler] Job 1285 completing (errored=False, cancelled=False)
[2017-06-01 19:16:49,826: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:16:51,690: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:51,692: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:51,708: INFO/command_plan] add_jobs: created Job 1287 (Test for host connectivity)
[2017-06-01 19:16:51,749: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:16:51,772: INFO/job_scheduler] Job 1287: RunJobThread.run
[2017-06-01 19:16:51,811: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:51,865: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:51,933: INFO/command_plan] add_jobs: created Job 1288 (Test for host connectivity)
[2017-06-01 19:16:51,959: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:16:51,979: INFO/job_scheduler] Job 1288: RunJobThread.run
[2017-06-01 19:16:52,032: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:52,034: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:52,054: INFO/command_plan] add_jobs: created Job 1289 (Test for host connectivity)
[2017-06-01 19:16:52,094: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:16:52,121: INFO/job_scheduler] Job 1289: RunJobThread.run
[2017-06-01 19:16:52,200: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:52,203: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:52,280: INFO/command_plan] add_jobs: created Job 1290 (Test for host connectivity)
[2017-06-01 19:16:52,338: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:16:52,345: INFO/job_scheduler] Job 1290: RunJobThread.run
[2017-06-01 19:16:54,031: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:16:54,212: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:16:54,455: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:16:54,586: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:16:55,389: INFO/job_scheduler] Job 1287 finished 1 steps successfully
[2017-06-01 19:16:55,406: INFO/job_scheduler] Job 1287: complete_job: Updating cache
[2017-06-01 19:16:55,406: INFO/job_scheduler] Job 1287 completing (errored=False, cancelled=False)
[2017-06-01 19:16:55,443: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:16:55,572: INFO/job_scheduler] Job 1288 finished 1 steps successfully
[2017-06-01 19:16:55,614: INFO/job_scheduler] Job 1288: complete_job: Updating cache
[2017-06-01 19:16:55,614: INFO/job_scheduler] Job 1288 completing (errored=False, cancelled=False)
[2017-06-01 19:16:55,709: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:16:55,937: INFO/job_scheduler] Job 1289 finished 1 steps successfully
[2017-06-01 19:16:55,946: INFO/job_scheduler] Job 1290 finished 1 steps successfully
[2017-06-01 19:16:55,989: INFO/job_scheduler] Job 1289: complete_job: Updating cache
[2017-06-01 19:16:55,990: INFO/job_scheduler] Job 1289 completing (errored=False, cancelled=False)
[2017-06-01 19:16:56,031: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:16:56,068: INFO/job_scheduler] Job 1290: complete_job: Updating cache
[2017-06-01 19:16:56,074: INFO/job_scheduler] Job 1290 completing (errored=False, cancelled=False)
[2017-06-01 19:16:56,168: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:16:58,084: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:58,084: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:58,167: INFO/command_plan] add_jobs: created Job 1291 (Test for host connectivity)
[2017-06-01 19:16:58,178: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:16:58,181: INFO/job_scheduler] Job 1291: RunJobThread.run
[2017-06-01 19:16:58,243: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:58,243: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:58,281: INFO/command_plan] add_jobs: created Job 1292 (Test for host connectivity)
[2017-06-01 19:16:58,335: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:16:58,345: INFO/job_scheduler] Job 1292: RunJobThread.run
[2017-06-01 19:16:58,431: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:58,431: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:58,467: INFO/command_plan] add_jobs: created Job 1293 (Test for host connectivity)
[2017-06-01 19:16:58,546: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:16:58,547: INFO/job_scheduler] Job 1293: RunJobThread.run
[2017-06-01 19:16:58,656: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:16:58,657: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:16:58,738: INFO/command_plan] add_jobs: created Job 1294 (Test for host connectivity)
[2017-06-01 19:16:58,783: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:16:58,816: INFO/job_scheduler] Job 1294: RunJobThread.run
[2017-06-01 19:17:00,526: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:17:00,729: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:17:01,053: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:17:01,161: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:17:01,959: INFO/job_scheduler] Job 1291 finished 1 steps successfully
[2017-06-01 19:17:02,009: INFO/job_scheduler] Job 1291: complete_job: Updating cache
[2017-06-01 19:17:02,009: INFO/job_scheduler] Job 1291 completing (errored=False, cancelled=False)
[2017-06-01 19:17:02,072: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:17:02,566: INFO/job_scheduler] Job 1292 finished 1 steps successfully
[2017-06-01 19:17:02,600: INFO/job_scheduler] Job 1292: complete_job: Updating cache
[2017-06-01 19:17:02,600: INFO/job_scheduler] Job 1292 completing (errored=False, cancelled=False)
[2017-06-01 19:17:02,644: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:17:02,693: INFO/job_scheduler] Job 1294 finished 1 steps successfully
[2017-06-01 19:17:02,745: INFO/job_scheduler] Job 1294: complete_job: Updating cache
[2017-06-01 19:17:02,745: INFO/job_scheduler] Job 1294 completing (errored=False, cancelled=False)
[2017-06-01 19:17:02,826: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:17:02,900: INFO/job_scheduler] Job 1293 finished 1 steps successfully
[2017-06-01 19:17:02,967: INFO/job_scheduler] Job 1293: complete_job: Updating cache
[2017-06-01 19:17:02,967: INFO/job_scheduler] Job 1293 completing (errored=False, cancelled=False)
[2017-06-01 19:17:03,092: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:17:05,092: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:05,095: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:05,119: INFO/command_plan] add_jobs: created Job 1295 (Test for host connectivity)
[2017-06-01 19:17:05,139: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:17:05,146: INFO/job_scheduler] Job 1295: RunJobThread.run
[2017-06-01 19:17:05,263: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:05,263: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:05,284: INFO/command_plan] add_jobs: created Job 1296 (Test for host connectivity)
[2017-06-01 19:17:05,334: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:17:05,382: INFO/job_scheduler] Job 1296: RunJobThread.run
[2017-06-01 19:17:05,440: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:05,441: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:05,469: INFO/command_plan] add_jobs: created Job 1297 (Test for host connectivity)
[2017-06-01 19:17:05,490: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:17:05,542: INFO/job_scheduler] Job 1297: RunJobThread.run
[2017-06-01 19:17:05,623: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:05,624: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:05,644: INFO/command_plan] add_jobs: created Job 1298 (Test for host connectivity)
[2017-06-01 19:17:05,675: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:17:05,716: INFO/job_scheduler] Job 1298: RunJobThread.run
[2017-06-01 19:17:07,522: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:17:07,751: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:17:07,897: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:17:08,013: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:17:08,871: INFO/job_scheduler] Job 1295 finished 1 steps successfully
[2017-06-01 19:17:08,899: INFO/job_scheduler] Job 1295: complete_job: Updating cache
[2017-06-01 19:17:08,900: INFO/job_scheduler] Job 1295 completing (errored=False, cancelled=False)
[2017-06-01 19:17:08,966: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:17:09,156: INFO/job_scheduler] Job 1296 finished 1 steps successfully
[2017-06-01 19:17:09,209: INFO/job_scheduler] Job 1296: complete_job: Updating cache
[2017-06-01 19:17:09,210: INFO/job_scheduler] Job 1296 completing (errored=False, cancelled=False)
[2017-06-01 19:17:09,268: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:17:09,296: INFO/job_scheduler] Job 1297 finished 1 steps successfully
[2017-06-01 19:17:09,342: INFO/job_scheduler] Job 1297: complete_job: Updating cache
[2017-06-01 19:17:09,343: INFO/job_scheduler] Job 1297 completing (errored=False, cancelled=False)
[2017-06-01 19:17:09,408: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:17:09,445: INFO/job_scheduler] Job 1298 finished 1 steps successfully
[2017-06-01 19:17:09,484: INFO/job_scheduler] Job 1298: complete_job: Updating cache
[2017-06-01 19:17:09,484: INFO/job_scheduler] Job 1298 completing (errored=False, cancelled=False)
[2017-06-01 19:17:09,534: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:17:11,354: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:11,355: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:11,374: INFO/command_plan] add_jobs: created Job 1299 (Test for host connectivity)
[2017-06-01 19:17:11,437: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:17:11,454: INFO/job_scheduler] Job 1299: RunJobThread.run
[2017-06-01 19:17:11,516: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:11,517: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:11,549: INFO/command_plan] add_jobs: created Job 1300 (Test for host connectivity)
[2017-06-01 19:17:11,570: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:17:11,658: INFO/job_scheduler] Job 1300: RunJobThread.run
[2017-06-01 19:17:11,721: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:11,723: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:11,749: INFO/command_plan] add_jobs: created Job 1301 (Test for host connectivity)
[2017-06-01 19:17:11,813: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:17:11,861: INFO/job_scheduler] Job 1301: RunJobThread.run
[2017-06-01 19:17:11,949: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:11,950: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:11,970: INFO/command_plan] add_jobs: created Job 1302 (Test for host connectivity)
[2017-06-01 19:17:12,052: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:17:12,086: INFO/job_scheduler] Job 1302: RunJobThread.run
[2017-06-01 19:17:13,805: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:17:13,992: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:17:14,198: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:17:14,326: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:17:15,131: INFO/job_scheduler] Job 1299 finished 1 steps successfully
[2017-06-01 19:17:15,149: INFO/job_scheduler] Job 1299: complete_job: Updating cache
[2017-06-01 19:17:15,149: INFO/job_scheduler] Job 1299 completing (errored=False, cancelled=False)
[2017-06-01 19:17:15,183: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:17:15,341: INFO/job_scheduler] Job 1300 finished 1 steps successfully
[2017-06-01 19:17:15,374: INFO/job_scheduler] Job 1300: complete_job: Updating cache
[2017-06-01 19:17:15,377: INFO/job_scheduler] Job 1300 completing (errored=False, cancelled=False)
[2017-06-01 19:17:15,442: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:17:15,690: INFO/job_scheduler] Job 1301 finished 1 steps successfully
[2017-06-01 19:17:15,724: INFO/job_scheduler] Job 1301: complete_job: Updating cache
[2017-06-01 19:17:15,724: INFO/job_scheduler] Job 1301 completing (errored=False, cancelled=False)
[2017-06-01 19:17:15,763: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:17:15,896: INFO/job_scheduler] Job 1302 finished 1 steps successfully
[2017-06-01 19:17:15,962: INFO/job_scheduler] Job 1302: complete_job: Updating cache
[2017-06-01 19:17:15,962: INFO/job_scheduler] Job 1302 completing (errored=False, cancelled=False)
[2017-06-01 19:17:15,995: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:17:17,630: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:17,632: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:17,674: INFO/command_plan] add_jobs: created Job 1303 (Test for host connectivity)
[2017-06-01 19:17:17,725: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:17:17,793: INFO/job_scheduler] Job 1303: RunJobThread.run
[2017-06-01 19:17:17,868: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:17,868: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:17,889: INFO/command_plan] add_jobs: created Job 1304 (Test for host connectivity)
[2017-06-01 19:17:17,997: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:17:18,031: INFO/job_scheduler] Job 1304: RunJobThread.run
[2017-06-01 19:17:18,095: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:18,097: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:18,120: INFO/command_plan] add_jobs: created Job 1305 (Test for host connectivity)
[2017-06-01 19:17:18,142: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:17:18,144: INFO/job_scheduler] Job 1305: RunJobThread.run
[2017-06-01 19:17:18,297: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:18,298: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:18,319: INFO/command_plan] add_jobs: created Job 1306 (Test for host connectivity)
[2017-06-01 19:17:18,370: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:17:18,448: INFO/job_scheduler] Job 1306: RunJobThread.run
[2017-06-01 19:17:20,127: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:17:20,369: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:17:20,499: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:17:20,694: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:17:21,449: INFO/job_scheduler] Job 1303 finished 1 steps successfully
[2017-06-01 19:17:21,544: INFO/job_scheduler] Job 1303: complete_job: Updating cache
[2017-06-01 19:17:21,545: INFO/job_scheduler] Job 1303 completing (errored=False, cancelled=False)
[2017-06-01 19:17:21,603: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:17:21,743: INFO/job_scheduler] Job 1304 finished 1 steps successfully
[2017-06-01 19:17:21,798: INFO/job_scheduler] Job 1304: complete_job: Updating cache
[2017-06-01 19:17:21,798: INFO/job_scheduler] Job 1304 completing (errored=False, cancelled=False)
[2017-06-01 19:17:21,887: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:17:21,889: INFO/job_scheduler] Job 1305 finished 1 steps successfully
[2017-06-01 19:17:21,912: INFO/job_scheduler] Job 1305: complete_job: Updating cache
[2017-06-01 19:17:21,913: INFO/job_scheduler] Job 1305 completing (errored=False, cancelled=False)
[2017-06-01 19:17:21,981: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:17:22,176: INFO/job_scheduler] Job 1306 finished 1 steps successfully
[2017-06-01 19:17:22,227: INFO/job_scheduler] Job 1306: complete_job: Updating cache
[2017-06-01 19:17:22,227: INFO/job_scheduler] Job 1306 completing (errored=False, cancelled=False)
[2017-06-01 19:17:22,252: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:17:24,036: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:24,036: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:24,055: INFO/command_plan] add_jobs: created Job 1307 (Test for host connectivity)
[2017-06-01 19:17:24,084: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:17:24,095: INFO/job_scheduler] Job 1307: RunJobThread.run
[2017-06-01 19:17:24,151: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:24,152: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:24,176: INFO/command_plan] add_jobs: created Job 1308 (Test for host connectivity)
[2017-06-01 19:17:24,250: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:17:24,284: INFO/job_scheduler] Job 1308: RunJobThread.run
[2017-06-01 19:17:24,321: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:24,322: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:24,371: INFO/command_plan] add_jobs: created Job 1309 (Test for host connectivity)
[2017-06-01 19:17:24,412: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:17:24,446: INFO/job_scheduler] Job 1309: RunJobThread.run
[2017-06-01 19:17:24,539: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:24,540: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:24,552: INFO/command_plan] add_jobs: created Job 1310 (Test for host connectivity)
[2017-06-01 19:17:24,672: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:17:24,686: INFO/job_scheduler] Job 1310: RunJobThread.run
[2017-06-01 19:17:26,534: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:17:26,591: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:17:26,737: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:17:26,950: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:17:27,863: INFO/job_scheduler] Job 1308 finished 1 steps successfully
[2017-06-01 19:17:27,917: INFO/job_scheduler] Job 1308: complete_job: Updating cache
[2017-06-01 19:17:27,918: INFO/job_scheduler] Job 1308 completing (errored=False, cancelled=False)
[2017-06-01 19:17:27,962: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:17:28,015: INFO/job_scheduler] Job 1307 finished 1 steps successfully
[2017-06-01 19:17:28,034: INFO/job_scheduler] Job 1307: complete_job: Updating cache
[2017-06-01 19:17:28,035: INFO/job_scheduler] Job 1307 completing (errored=False, cancelled=False)
[2017-06-01 19:17:28,136: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:17:28,193: INFO/job_scheduler] Job 1309 finished 1 steps successfully
[2017-06-01 19:17:28,235: INFO/job_scheduler] Job 1309: complete_job: Updating cache
[2017-06-01 19:17:28,235: INFO/job_scheduler] Job 1309 completing (errored=False, cancelled=False)
[2017-06-01 19:17:28,302: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:17:28,410: INFO/job_scheduler] Job 1310 finished 1 steps successfully
[2017-06-01 19:17:28,451: INFO/job_scheduler] Job 1310: complete_job: Updating cache
[2017-06-01 19:17:28,451: INFO/job_scheduler] Job 1310 completing (errored=False, cancelled=False)
[2017-06-01 19:17:28,485: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:17:30,294: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:30,296: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:30,309: INFO/command_plan] add_jobs: created Job 1311 (Test for host connectivity)
[2017-06-01 19:17:30,368: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:30,369: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:30,419: INFO/command_plan] add_jobs: created Job 1312 (Test for host connectivity)
[2017-06-01 19:17:30,446: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:17:30,462: INFO/job_scheduler] Job 1312: RunJobThread.run
[2017-06-01 19:17:30,463: INFO/job_scheduler] Job 1311: RunJobThread.run
[2017-06-01 19:17:30,468: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:17:30,568: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:30,570: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:30,588: INFO/command_plan] add_jobs: created Job 1313 (Test for host connectivity)
[2017-06-01 19:17:30,662: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:17:30,703: INFO/job_scheduler] Job 1313: RunJobThread.run
[2017-06-01 19:17:30,747: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:30,748: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:30,767: INFO/command_plan] add_jobs: created Job 1314 (Test for host connectivity)
[2017-06-01 19:17:30,902: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:17:30,909: INFO/job_scheduler] Job 1314: RunJobThread.run
[2017-06-01 19:17:32,793: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:17:32,882: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:17:33,049: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:17:33,292: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:17:34,117: INFO/job_scheduler] Job 1311 finished 1 steps successfully
[2017-06-01 19:17:34,158: INFO/job_scheduler] Job 1311: complete_job: Updating cache
[2017-06-01 19:17:34,158: INFO/job_scheduler] Job 1311 completing (errored=False, cancelled=False)
[2017-06-01 19:17:34,192: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:17:34,235: INFO/job_scheduler] Job 1312 finished 1 steps successfully
[2017-06-01 19:17:34,325: INFO/job_scheduler] Job 1312: complete_job: Updating cache
[2017-06-01 19:17:34,326: INFO/job_scheduler] Job 1312 completing (errored=False, cancelled=False)
[2017-06-01 19:17:34,393: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:17:34,416: INFO/job_scheduler] Job 1313 finished 1 steps successfully
[2017-06-01 19:17:34,435: INFO/job_scheduler] Job 1313: complete_job: Updating cache
[2017-06-01 19:17:34,453: INFO/job_scheduler] Job 1313 completing (errored=False, cancelled=False)
[2017-06-01 19:17:34,568: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:17:34,786: INFO/job_scheduler] Job 1314 finished 1 steps successfully
[2017-06-01 19:17:34,833: INFO/job_scheduler] Job 1314: complete_job: Updating cache
[2017-06-01 19:17:34,833: INFO/job_scheduler] Job 1314 completing (errored=False, cancelled=False)
[2017-06-01 19:17:34,859: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:17:36,529: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:36,529: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:36,542: INFO/command_plan] add_jobs: created Job 1315 (Test for host connectivity)
[2017-06-01 19:17:36,634: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:17:36,662: INFO/job_scheduler] Job 1315: RunJobThread.run
[2017-06-01 19:17:36,712: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:36,712: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:36,737: INFO/command_plan] add_jobs: created Job 1316 (Test for host connectivity)
[2017-06-01 19:17:36,808: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:17:36,849: INFO/job_scheduler] Job 1316: RunJobThread.run
[2017-06-01 19:17:36,880: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:36,880: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:36,901: INFO/command_plan] add_jobs: created Job 1317 (Test for host connectivity)
[2017-06-01 19:17:36,933: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:17:36,972: INFO/job_scheduler] Job 1317: RunJobThread.run
[2017-06-01 19:17:37,063: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:37,089: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:37,134: INFO/command_plan] add_jobs: created Job 1318 (Test for host connectivity)
[2017-06-01 19:17:37,191: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:17:37,243: INFO/job_scheduler] Job 1318: RunJobThread.run
[2017-06-01 19:17:39,056: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:17:39,098: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:17:39,440: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:17:39,652: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:17:40,384: INFO/job_scheduler] Job 1315 finished 1 steps successfully
[2017-06-01 19:17:40,415: INFO/job_scheduler] Job 1315: complete_job: Updating cache
[2017-06-01 19:17:40,415: INFO/job_scheduler] Job 1315 completing (errored=False, cancelled=False)
[2017-06-01 19:17:40,483: INFO/job_scheduler] Job 1316 finished 1 steps successfully
[2017-06-01 19:17:40,499: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:17:40,526: INFO/job_scheduler] Job 1316: complete_job: Updating cache
[2017-06-01 19:17:40,529: INFO/job_scheduler] Job 1316 completing (errored=False, cancelled=False)
[2017-06-01 19:17:40,670: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:17:40,904: INFO/job_scheduler] Job 1317 finished 1 steps successfully
[2017-06-01 19:17:40,920: INFO/job_scheduler] Job 1318 finished 1 steps successfully
[2017-06-01 19:17:40,948: INFO/job_scheduler] Job 1317: complete_job: Updating cache
[2017-06-01 19:17:40,948: INFO/job_scheduler] Job 1317 completing (errored=False, cancelled=False)
[2017-06-01 19:17:41,001: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:17:41,015: INFO/job_scheduler] Job 1318: complete_job: Updating cache
[2017-06-01 19:17:41,016: INFO/job_scheduler] Job 1318 completing (errored=False, cancelled=False)
[2017-06-01 19:17:41,100: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:17:42,799: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:42,801: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:42,815: INFO/command_plan] add_jobs: created Job 1319 (Test for host connectivity)
[2017-06-01 19:17:42,912: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:42,912: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:42,948: INFO/command_plan] add_jobs: created Job 1320 (Test for host connectivity)
[2017-06-01 19:17:43,012: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:17:43,026: INFO/job_scheduler] Job 1320: RunJobThread.run
[2017-06-01 19:17:43,033: INFO/job_scheduler] Job 1319: RunJobThread.run
[2017-06-01 19:17:43,040: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:17:43,083: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:43,085: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:43,098: INFO/command_plan] add_jobs: created Job 1321 (Test for host connectivity)
[2017-06-01 19:17:43,124: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:17:43,196: INFO/job_scheduler] Job 1321: RunJobThread.run
[2017-06-01 19:17:43,285: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:43,286: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:43,306: INFO/command_plan] add_jobs: created Job 1322 (Test for host connectivity)
[2017-06-01 19:17:43,330: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:17:43,363: INFO/job_scheduler] Job 1322: RunJobThread.run
[2017-06-01 19:17:45,251: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:17:45,350: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:17:45,570: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:17:45,719: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:17:46,614: INFO/job_scheduler] Job 1320 finished 1 steps successfully
[2017-06-01 19:17:46,676: INFO/job_scheduler] Job 1320: complete_job: Updating cache
[2017-06-01 19:17:46,676: INFO/job_scheduler] Job 1320 completing (errored=False, cancelled=False)
[2017-06-01 19:17:46,708: INFO/job_scheduler] Job 1319 finished 1 steps successfully
[2017-06-01 19:17:46,732: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:17:46,757: INFO/job_scheduler] Job 1319: complete_job: Updating cache
[2017-06-01 19:17:46,758: INFO/job_scheduler] Job 1319 completing (errored=False, cancelled=False)
[2017-06-01 19:17:46,882: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:17:47,010: INFO/job_scheduler] Job 1321 finished 1 steps successfully
[2017-06-01 19:17:47,130: INFO/job_scheduler] Job 1321: complete_job: Updating cache
[2017-06-01 19:17:47,130: INFO/job_scheduler] Job 1321 completing (errored=False, cancelled=False)
[2017-06-01 19:17:47,165: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:17:47,214: INFO/job_scheduler] Job 1322 finished 1 steps successfully
[2017-06-01 19:17:47,255: INFO/job_scheduler] Job 1322: complete_job: Updating cache
[2017-06-01 19:17:47,255: INFO/job_scheduler] Job 1322 completing (errored=False, cancelled=False)
[2017-06-01 19:17:47,362: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:17:48,949: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:48,950: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:48,963: INFO/command_plan] add_jobs: created Job 1323 (Test for host connectivity)
[2017-06-01 19:17:49,076: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:49,077: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:49,097: INFO/command_plan] add_jobs: created Job 1324 (Test for host connectivity)
[2017-06-01 19:17:49,132: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:17:49,147: INFO/job_scheduler] Job 1323: RunJobThread.run
[2017-06-01 19:17:49,171: INFO/job_scheduler] Job 1324: RunJobThread.run
[2017-06-01 19:17:49,222: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:49,228: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:49,262: INFO/command_plan] add_jobs: created Job 1325 (Test for host connectivity)
[2017-06-01 19:17:49,298: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:17:49,349: INFO/job_scheduler] Job 1325: RunJobThread.run
[2017-06-01 19:17:49,456: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:49,457: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:49,481: INFO/command_plan] add_jobs: created Job 1326 (Test for host connectivity)
[2017-06-01 19:17:49,497: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:17:49,538: INFO/job_scheduler] Job 1326: RunJobThread.run
[2017-06-01 19:17:49,583: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:17:51,464: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:17:51,572: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:17:51,659: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:17:51,929: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:17:52,855: INFO/job_scheduler] Job 1323 finished 1 steps successfully
[2017-06-01 19:17:52,896: INFO/job_scheduler] Job 1323: complete_job: Updating cache
[2017-06-01 19:17:52,896: INFO/job_scheduler] Job 1323 completing (errored=False, cancelled=False)
[2017-06-01 19:17:52,916: INFO/job_scheduler] Job 1324 finished 1 steps successfully
[2017-06-01 19:17:52,947: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:17:52,972: INFO/job_scheduler] Job 1324: complete_job: Updating cache
[2017-06-01 19:17:52,974: INFO/job_scheduler] Job 1324 completing (errored=False, cancelled=False)
[2017-06-01 19:17:53,019: INFO/job_scheduler] Job 1325 finished 1 steps successfully
[2017-06-01 19:17:53,103: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:17:53,161: INFO/job_scheduler] Job 1325: complete_job: Updating cache
[2017-06-01 19:17:53,161: INFO/job_scheduler] Job 1325 completing (errored=False, cancelled=False)
[2017-06-01 19:17:53,241: INFO/job_scheduler] Job 1326 finished 1 steps successfully
[2017-06-01 19:17:53,250: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:17:53,286: INFO/job_scheduler] Job 1326: complete_job: Updating cache
[2017-06-01 19:17:53,286: INFO/job_scheduler] Job 1326 completing (errored=False, cancelled=False)
[2017-06-01 19:17:53,382: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:17:55,131: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:55,131: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:55,146: INFO/command_plan] add_jobs: created Job 1327 (Test for host connectivity)
[2017-06-01 19:17:55,203: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:55,203: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:55,231: INFO/command_plan] add_jobs: created Job 1328 (Test for host connectivity)
[2017-06-01 19:17:55,252: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:17:55,262: INFO/job_scheduler] Job 1328: RunJobThread.run
[2017-06-01 19:17:55,312: INFO/job_scheduler] Job 1327: RunJobThread.run
[2017-06-01 19:17:55,325: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:17:55,385: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:55,385: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:55,395: INFO/command_plan] add_jobs: created Job 1329 (Test for host connectivity)
[2017-06-01 19:17:55,452: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:17:55,505: INFO/job_scheduler] Job 1329: RunJobThread.run
[2017-06-01 19:17:55,622: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:17:55,624: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:17:55,651: INFO/command_plan] add_jobs: created Job 1330 (Test for host connectivity)
[2017-06-01 19:17:55,685: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:17:55,739: INFO/job_scheduler] Job 1330: RunJobThread.run
[2017-06-01 19:17:57,685: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:17:57,707: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:17:57,860: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:17:58,010: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:17:59,037: INFO/job_scheduler] Job 1327 finished 1 steps successfully
[2017-06-01 19:17:59,111: INFO/job_scheduler] Job 1327: complete_job: Updating cache
[2017-06-01 19:17:59,111: INFO/job_scheduler] Job 1327 completing (errored=False, cancelled=False)
[2017-06-01 19:17:59,116: INFO/job_scheduler] Job 1328 finished 1 steps successfully
[2017-06-01 19:17:59,172: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:17:59,200: INFO/job_scheduler] Job 1328: complete_job: Updating cache
[2017-06-01 19:17:59,200: INFO/job_scheduler] Job 1328 completing (errored=False, cancelled=False)
[2017-06-01 19:17:59,294: INFO/job_scheduler] Job 1329 finished 1 steps successfully
[2017-06-01 19:17:59,386: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:17:59,397: INFO/job_scheduler] Job 1330 finished 1 steps successfully
[2017-06-01 19:17:59,426: INFO/job_scheduler] Job 1329: complete_job: Updating cache
[2017-06-01 19:17:59,426: INFO/job_scheduler] Job 1329 completing (errored=False, cancelled=False)
[2017-06-01 19:17:59,503: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:17:59,520: INFO/job_scheduler] Job 1330: complete_job: Updating cache
[2017-06-01 19:17:59,520: INFO/job_scheduler] Job 1330 completing (errored=False, cancelled=False)
[2017-06-01 19:17:59,562: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:18:01,306: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:01,306: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:01,319: INFO/command_plan] add_jobs: created Job 1331 (Test for host connectivity)
[2017-06-01 19:18:01,426: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:01,432: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:01,444: INFO/command_plan] add_jobs: created Job 1332 (Test for host connectivity)
[2017-06-01 19:18:01,456: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:18:01,496: INFO/job_scheduler] Job 1331: RunJobThread.run
[2017-06-01 19:18:01,526: INFO/job_scheduler] Job 1332: RunJobThread.run
[2017-06-01 19:18:01,549: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:18:01,608: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:01,610: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:01,769: INFO/command_plan] add_jobs: created Job 1333 (Test for host connectivity)
[2017-06-01 19:18:01,804: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:18:01,812: INFO/job_scheduler] Job 1333: RunJobThread.run
[2017-06-01 19:18:01,854: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:01,854: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:01,877: INFO/command_plan] add_jobs: created Job 1334 (Test for host connectivity)
[2017-06-01 19:18:01,888: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:18:01,926: INFO/job_scheduler] Job 1334: RunJobThread.run
[2017-06-01 19:18:03,894: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:18:03,930: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:18:04,320: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:18:04,320: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:18:05,240: INFO/job_scheduler] Job 1332 finished 1 steps successfully
[2017-06-01 19:18:05,266: INFO/job_scheduler] Job 1331 finished 1 steps successfully
[2017-06-01 19:18:05,295: INFO/job_scheduler] Job 1332: complete_job: Updating cache
[2017-06-01 19:18:05,295: INFO/job_scheduler] Job 1332 completing (errored=False, cancelled=False)
[2017-06-01 19:18:05,411: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:18:05,462: INFO/job_scheduler] Job 1331: complete_job: Updating cache
[2017-06-01 19:18:05,463: INFO/job_scheduler] Job 1331 completing (errored=False, cancelled=False)
[2017-06-01 19:18:05,544: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:18:05,788: INFO/job_scheduler] Job 1334 finished 1 steps successfully
[2017-06-01 19:18:05,810: INFO/job_scheduler] Job 1334: complete_job: Updating cache
[2017-06-01 19:18:05,810: INFO/job_scheduler] Job 1334 completing (errored=False, cancelled=False)
[2017-06-01 19:18:05,843: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:18:05,876: INFO/job_scheduler] Job 1333 finished 1 steps successfully
[2017-06-01 19:18:05,893: INFO/job_scheduler] Job 1333: complete_job: Updating cache
[2017-06-01 19:18:05,897: INFO/job_scheduler] Job 1333 completing (errored=False, cancelled=False)
[2017-06-01 19:18:05,953: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:18:07,525: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:07,527: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:07,536: INFO/command_plan] add_jobs: created Job 1335 (Test for host connectivity)
[2017-06-01 19:18:07,641: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:18:07,645: INFO/job_scheduler] Job 1335: RunJobThread.run
[2017-06-01 19:18:07,688: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:07,688: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:07,702: INFO/command_plan] add_jobs: created Job 1336 (Test for host connectivity)
[2017-06-01 19:18:07,756: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:18:07,781: INFO/job_scheduler] Job 1336: RunJobThread.run
[2017-06-01 19:18:07,837: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:07,840: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:07,876: INFO/command_plan] add_jobs: created Job 1337 (Test for host connectivity)
[2017-06-01 19:18:07,963: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:18:07,973: INFO/job_scheduler] Job 1337: RunJobThread.run
[2017-06-01 19:18:08,052: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:08,159: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:08,217: INFO/command_plan] add_jobs: created Job 1338 (Test for host connectivity)
[2017-06-01 19:18:08,247: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:18:08,248: INFO/job_scheduler] Job 1338: RunJobThread.run
[2017-06-01 19:18:09,935: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:18:10,079: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:18:10,267: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:18:10,399: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:18:11,258: INFO/job_scheduler] Job 1335 finished 1 steps successfully
[2017-06-01 19:18:11,283: INFO/job_scheduler] Job 1335: complete_job: Updating cache
[2017-06-01 19:18:11,284: INFO/job_scheduler] Job 1335 completing (errored=False, cancelled=False)
[2017-06-01 19:18:11,351: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:18:11,414: INFO/job_scheduler] Job 1336 finished 1 steps successfully
[2017-06-01 19:18:11,472: INFO/job_scheduler] Job 1336: complete_job: Updating cache
[2017-06-01 19:18:11,474: INFO/job_scheduler] Job 1336 completing (errored=False, cancelled=False)
[2017-06-01 19:18:11,536: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:18:11,647: INFO/job_scheduler] Job 1337 finished 1 steps successfully
[2017-06-01 19:18:11,708: INFO/job_scheduler] Job 1337: complete_job: Updating cache
[2017-06-01 19:18:11,709: INFO/job_scheduler] Job 1337 completing (errored=False, cancelled=False)
[2017-06-01 19:18:11,776: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:18:11,856: INFO/job_scheduler] Job 1338 finished 1 steps successfully
[2017-06-01 19:18:11,901: INFO/job_scheduler] Job 1338: complete_job: Updating cache
[2017-06-01 19:18:11,903: INFO/job_scheduler] Job 1338 completing (errored=False, cancelled=False)
[2017-06-01 19:18:11,942: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:18:13,869: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:13,869: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:13,892: INFO/command_plan] add_jobs: created Job 1339 (Test for host connectivity)
[2017-06-01 19:18:13,913: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:18:13,933: INFO/job_scheduler] Job 1339: RunJobThread.run
[2017-06-01 19:18:14,015: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:14,016: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:14,036: INFO/command_plan] add_jobs: created Job 1340 (Test for host connectivity)
[2017-06-01 19:18:14,090: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:18:14,146: INFO/job_scheduler] Job 1340: RunJobThread.run
[2017-06-01 19:18:14,177: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:14,178: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:14,319: INFO/command_plan] add_jobs: created Job 1341 (Test for host connectivity)
[2017-06-01 19:18:14,374: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:14,374: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:14,427: INFO/command_plan] add_jobs: created Job 1342 (Test for host connectivity)
[2017-06-01 19:18:14,446: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:18:14,496: INFO/job_scheduler] Job 1341: RunJobThread.run
[2017-06-01 19:18:14,557: INFO/job_scheduler] Job 1342: RunJobThread.run
[2017-06-01 19:18:14,572: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:18:16,193: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:18:16,359: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:18:16,823: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:18:16,854: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:18:17,513: INFO/job_scheduler] Job 1339 finished 1 steps successfully
[2017-06-01 19:18:17,568: INFO/job_scheduler] Job 1339: complete_job: Updating cache
[2017-06-01 19:18:17,568: INFO/job_scheduler] Job 1339 completing (errored=False, cancelled=False)
[2017-06-01 19:18:17,685: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:18:17,705: INFO/job_scheduler] Job 1340 finished 1 steps successfully
[2017-06-01 19:18:17,764: INFO/job_scheduler] Job 1340: complete_job: Updating cache
[2017-06-01 19:18:17,765: INFO/job_scheduler] Job 1340 completing (errored=False, cancelled=False)
[2017-06-01 19:18:17,851: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:18:18,259: INFO/job_scheduler] Job 1342 finished 1 steps successfully
[2017-06-01 19:18:18,310: INFO/job_scheduler] Job 1342: complete_job: Updating cache
[2017-06-01 19:18:18,310: INFO/job_scheduler] Job 1342 completing (errored=False, cancelled=False)
[2017-06-01 19:18:18,320: INFO/job_scheduler] Job 1341 finished 1 steps successfully
[2017-06-01 19:18:18,343: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:18:18,369: INFO/job_scheduler] Job 1341: complete_job: Updating cache
[2017-06-01 19:18:18,379: INFO/job_scheduler] Job 1341 completing (errored=False, cancelled=False)
[2017-06-01 19:18:18,461: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:18:20,101: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:20,103: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:20,111: INFO/command_plan] add_jobs: created Job 1343 (Test for host connectivity)
[2017-06-01 19:18:20,130: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:18:20,178: INFO/job_scheduler] Job 1343: RunJobThread.run
[2017-06-01 19:18:20,243: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:20,244: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:20,259: INFO/command_plan] add_jobs: created Job 1344 (Test for host connectivity)
[2017-06-01 19:18:20,308: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:18:20,346: INFO/job_scheduler] Job 1344: RunJobThread.run
[2017-06-01 19:18:20,387: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:20,388: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:20,426: INFO/command_plan] add_jobs: created Job 1345 (Test for host connectivity)
[2017-06-01 19:18:20,480: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:18:20,510: INFO/job_scheduler] Job 1345: RunJobThread.run
[2017-06-01 19:18:20,586: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:20,587: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:20,643: INFO/command_plan] add_jobs: created Job 1346 (Test for host connectivity)
[2017-06-01 19:18:20,673: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:18:20,689: INFO/job_scheduler] Job 1346: RunJobThread.run
[2017-06-01 19:18:22,487: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:18:22,782: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:18:22,785: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:18:23,017: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:18:23,780: INFO/job_scheduler] Job 1343 finished 1 steps successfully
[2017-06-01 19:18:23,817: INFO/job_scheduler] Job 1343: complete_job: Updating cache
[2017-06-01 19:18:23,817: INFO/job_scheduler] Job 1343 completing (errored=False, cancelled=False)
[2017-06-01 19:18:23,984: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:18:24,204: INFO/job_scheduler] Job 1345 finished 1 steps successfully
[2017-06-01 19:18:24,239: INFO/job_scheduler] Job 1344 finished 1 steps successfully
[2017-06-01 19:18:24,258: INFO/job_scheduler] Job 1345: complete_job: Updating cache
[2017-06-01 19:18:24,258: INFO/job_scheduler] Job 1345 completing (errored=False, cancelled=False)
[2017-06-01 19:18:24,333: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:18:24,355: INFO/job_scheduler] Job 1344: complete_job: Updating cache
[2017-06-01 19:18:24,356: INFO/job_scheduler] Job 1344 completing (errored=False, cancelled=False)
[2017-06-01 19:18:24,418: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:18:24,429: INFO/job_scheduler] Job 1346 finished 1 steps successfully
[2017-06-01 19:18:24,493: INFO/job_scheduler] Job 1346: complete_job: Updating cache
[2017-06-01 19:18:24,493: INFO/job_scheduler] Job 1346 completing (errored=False, cancelled=False)
[2017-06-01 19:18:24,536: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:18:26,334: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:26,334: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:26,360: INFO/command_plan] add_jobs: created Job 1347 (Test for host connectivity)
[2017-06-01 19:18:26,410: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:18:26,419: INFO/job_scheduler] Job 1347: RunJobThread.run
[2017-06-01 19:18:26,487: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:26,487: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:26,502: INFO/command_plan] add_jobs: created Job 1348 (Test for host connectivity)
[2017-06-01 19:18:26,544: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:18:26,557: INFO/job_scheduler] Job 1348: RunJobThread.run
[2017-06-01 19:18:26,679: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:26,680: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:26,708: INFO/command_plan] add_jobs: created Job 1349 (Test for host connectivity)
[2017-06-01 19:18:26,763: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:18:26,795: INFO/job_scheduler] Job 1349: RunJobThread.run
[2017-06-01 19:18:26,849: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:26,854: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:26,885: INFO/command_plan] add_jobs: created Job 1350 (Test for host connectivity)
[2017-06-01 19:18:26,979: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:18:27,012: INFO/job_scheduler] Job 1350: RunJobThread.run
[2017-06-01 19:18:28,958: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:18:29,000: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:18:29,081: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:18:29,317: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:18:30,296: INFO/job_scheduler] Job 1347 finished 1 steps successfully
[2017-06-01 19:18:30,348: INFO/job_scheduler] Job 1348 finished 1 steps successfully
[2017-06-01 19:18:30,366: INFO/job_scheduler] Job 1347: complete_job: Updating cache
[2017-06-01 19:18:30,366: INFO/job_scheduler] Job 1347 completing (errored=False, cancelled=False)
[2017-06-01 19:18:30,408: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:18:30,421: INFO/job_scheduler] Job 1349 finished 1 steps successfully
[2017-06-01 19:18:30,424: INFO/job_scheduler] Job 1348: complete_job: Updating cache
[2017-06-01 19:18:30,426: INFO/job_scheduler] Job 1348 completing (errored=False, cancelled=False)
[2017-06-01 19:18:30,574: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:18:30,592: INFO/job_scheduler] Job 1349: complete_job: Updating cache
[2017-06-01 19:18:30,592: INFO/job_scheduler] Job 1349 completing (errored=False, cancelled=False)
[2017-06-01 19:18:30,691: INFO/job_scheduler] Job 1350 finished 1 steps successfully
[2017-06-01 19:18:30,721: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:18:30,751: INFO/job_scheduler] Job 1350: complete_job: Updating cache
[2017-06-01 19:18:30,755: INFO/job_scheduler] Job 1350 completing (errored=False, cancelled=False)
[2017-06-01 19:18:30,834: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:18:32,644: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:32,644: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:32,711: INFO/command_plan] add_jobs: created Job 1351 (Test for host connectivity)
[2017-06-01 19:18:32,775: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:18:32,804: INFO/job_scheduler] Job 1351: RunJobThread.run
[2017-06-01 19:18:32,850: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:32,858: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:32,875: INFO/command_plan] add_jobs: created Job 1352 (Test for host connectivity)
[2017-06-01 19:18:32,935: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:18:32,966: INFO/job_scheduler] Job 1352: RunJobThread.run
[2017-06-01 19:18:33,015: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:33,026: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:33,132: INFO/command_plan] add_jobs: created Job 1353 (Test for host connectivity)
[2017-06-01 19:18:33,176: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:18:33,188: INFO/job_scheduler] Job 1353: RunJobThread.run
[2017-06-01 19:18:33,244: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:33,246: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:33,279: INFO/command_plan] add_jobs: created Job 1354 (Test for host connectivity)
[2017-06-01 19:18:33,341: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:18:33,414: INFO/job_scheduler] Job 1354: RunJobThread.run
[2017-06-01 19:18:35,079: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:18:35,186: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:18:35,487: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:18:35,628: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:18:36,384: INFO/job_scheduler] Job 1351 finished 1 steps successfully
[2017-06-01 19:18:36,448: INFO/job_scheduler] Job 1351: complete_job: Updating cache
[2017-06-01 19:18:36,448: INFO/job_scheduler] Job 1351 completing (errored=False, cancelled=False)
[2017-06-01 19:18:36,482: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:18:36,509: INFO/job_scheduler] Job 1352 finished 1 steps successfully
[2017-06-01 19:18:36,547: INFO/job_scheduler] Job 1352: complete_job: Updating cache
[2017-06-01 19:18:36,557: INFO/job_scheduler] Job 1352 completing (errored=False, cancelled=False)
[2017-06-01 19:18:36,639: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:18:37,014: INFO/job_scheduler] Job 1353 finished 1 steps successfully
[2017-06-01 19:18:37,072: INFO/job_scheduler] Job 1353: complete_job: Updating cache
[2017-06-01 19:18:37,073: INFO/job_scheduler] Job 1353 completing (errored=False, cancelled=False)
[2017-06-01 19:18:37,075: INFO/job_scheduler] Job 1354 finished 1 steps successfully
[2017-06-01 19:18:37,125: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:18:37,141: INFO/job_scheduler] Job 1354: complete_job: Updating cache
[2017-06-01 19:18:37,141: INFO/job_scheduler] Job 1354 completing (errored=False, cancelled=False)
[2017-06-01 19:18:37,173: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:18:38,553: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:38,555: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:38,581: INFO/command_plan] add_jobs: created Job 1355 (Test for host connectivity)
[2017-06-01 19:18:38,645: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:18:38,655: INFO/job_scheduler] Job 1355: RunJobThread.run
[2017-06-01 19:18:38,695: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:38,695: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:38,711: INFO/command_plan] add_jobs: created Job 1356 (Test for host connectivity)
[2017-06-01 19:18:38,822: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:18:38,828: INFO/job_scheduler] Job 1356: RunJobThread.run
[2017-06-01 19:18:38,859: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:38,860: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:38,894: INFO/command_plan] add_jobs: created Job 1357 (Test for host connectivity)
[2017-06-01 19:18:38,959: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:18:39,038: INFO/job_scheduler] Job 1357: RunJobThread.run
[2017-06-01 19:18:39,078: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:39,080: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:39,101: INFO/command_plan] add_jobs: created Job 1358 (Test for host connectivity)
[2017-06-01 19:18:39,202: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:18:39,213: INFO/job_scheduler] Job 1358: RunJobThread.run
[2017-06-01 19:18:40,930: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:18:41,440: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:18:41,449: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:18:41,490: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:18:42,259: INFO/job_scheduler] Job 1355 finished 1 steps successfully
[2017-06-01 19:18:42,271: INFO/job_scheduler] Job 1355: complete_job: Updating cache
[2017-06-01 19:18:42,271: INFO/job_scheduler] Job 1355 completing (errored=False, cancelled=False)
[2017-06-01 19:18:42,401: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:18:42,953: INFO/job_scheduler] Job 1358 finished 1 steps successfully
[2017-06-01 19:18:42,965: INFO/job_scheduler] Job 1356 finished 1 steps successfully
[2017-06-01 19:18:42,970: INFO/job_scheduler] Job 1357 finished 1 steps successfully
[2017-06-01 19:18:43,013: INFO/job_scheduler] Job 1358: complete_job: Updating cache
[2017-06-01 19:18:43,013: INFO/job_scheduler] Job 1358 completing (errored=False, cancelled=False)
[2017-06-01 19:18:43,048: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:18:43,064: INFO/job_scheduler] Job 1356: complete_job: Updating cache
[2017-06-01 19:18:43,064: INFO/job_scheduler] Job 1356 completing (errored=False, cancelled=False)
[2017-06-01 19:18:43,121: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:18:43,157: INFO/job_scheduler] Job 1357: complete_job: Updating cache
[2017-06-01 19:18:43,158: INFO/job_scheduler] Job 1357 completing (errored=False, cancelled=False)
[2017-06-01 19:18:43,215: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:18:44,884: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:44,884: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:44,908: INFO/command_plan] add_jobs: created Job 1359 (Test for host connectivity)
[2017-06-01 19:18:44,942: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:18:44,966: INFO/job_scheduler] Job 1359: RunJobThread.run
[2017-06-01 19:18:45,104: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:45,105: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:45,138: INFO/command_plan] add_jobs: created Job 1360 (Test for host connectivity)
[2017-06-01 19:18:45,235: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:18:45,240: INFO/job_scheduler] Job 1360: RunJobThread.run
[2017-06-01 19:18:45,281: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:45,281: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:45,328: INFO/command_plan] add_jobs: created Job 1361 (Test for host connectivity)
[2017-06-01 19:18:45,361: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:18:45,402: INFO/job_scheduler] Job 1361: RunJobThread.run
[2017-06-01 19:18:45,455: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:45,455: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:45,473: INFO/command_plan] add_jobs: created Job 1362 (Test for host connectivity)
[2017-06-01 19:18:45,514: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:18:45,580: INFO/job_scheduler] Job 1362: RunJobThread.run
[2017-06-01 19:18:47,315: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:18:47,506: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:18:47,726: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:18:47,874: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:18:48,650: INFO/job_scheduler] Job 1359 finished 1 steps successfully
[2017-06-01 19:18:48,687: INFO/job_scheduler] Job 1359: complete_job: Updating cache
[2017-06-01 19:18:48,687: INFO/job_scheduler] Job 1359 completing (errored=False, cancelled=False)
[2017-06-01 19:18:48,789: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:18:48,832: INFO/job_scheduler] Job 1360 finished 1 steps successfully
[2017-06-01 19:18:48,862: INFO/job_scheduler] Job 1360: complete_job: Updating cache
[2017-06-01 19:18:48,862: INFO/job_scheduler] Job 1360 completing (errored=False, cancelled=False)
[2017-06-01 19:18:48,951: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:18:49,105: INFO/job_scheduler] Job 1361 finished 1 steps successfully
[2017-06-01 19:18:49,139: INFO/job_scheduler] Job 1361: complete_job: Updating cache
[2017-06-01 19:18:49,139: INFO/job_scheduler] Job 1361 completing (errored=False, cancelled=False)
[2017-06-01 19:18:49,196: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:18:49,311: INFO/job_scheduler] Job 1362 finished 1 steps successfully
[2017-06-01 19:18:49,337: INFO/job_scheduler] Job 1362: complete_job: Updating cache
[2017-06-01 19:18:49,337: INFO/job_scheduler] Job 1362 completing (errored=False, cancelled=False)
[2017-06-01 19:18:49,387: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:18:51,384: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:51,384: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:51,429: INFO/command_plan] add_jobs: created Job 1363 (Test for host connectivity)
[2017-06-01 19:18:51,466: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:18:51,469: INFO/job_scheduler] Job 1363: RunJobThread.run
[2017-06-01 19:18:51,520: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:51,521: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:51,548: INFO/command_plan] add_jobs: created Job 1364 (Test for host connectivity)
[2017-06-01 19:18:51,619: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:18:51,689: INFO/job_scheduler] Job 1364: RunJobThread.run
[2017-06-01 19:18:51,723: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:51,724: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:51,753: INFO/command_plan] add_jobs: created Job 1365 (Test for host connectivity)
[2017-06-01 19:18:51,833: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:18:51,893: INFO/job_scheduler] Job 1365: RunJobThread.run
[2017-06-01 19:18:51,949: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:51,949: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:51,962: INFO/command_plan] add_jobs: created Job 1366 (Test for host connectivity)
[2017-06-01 19:18:52,008: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:18:52,023: INFO/job_scheduler] Job 1366: RunJobThread.run
[2017-06-01 19:18:53,928: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:18:54,072: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:18:54,211: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:18:54,289: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:18:55,308: INFO/job_scheduler] Job 1363 finished 1 steps successfully
[2017-06-01 19:18:55,377: INFO/job_scheduler] Job 1363: complete_job: Updating cache
[2017-06-01 19:18:55,377: INFO/job_scheduler] Job 1363 completing (errored=False, cancelled=False)
[2017-06-01 19:18:55,401: INFO/job_scheduler] Job 1364 finished 1 steps successfully
[2017-06-01 19:18:55,427: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:18:55,492: INFO/job_scheduler] Job 1364: complete_job: Updating cache
[2017-06-01 19:18:55,492: INFO/job_scheduler] Job 1364 completing (errored=False, cancelled=False)
[2017-06-01 19:18:55,546: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:18:55,645: INFO/job_scheduler] Job 1366 finished 1 steps successfully
[2017-06-01 19:18:55,650: INFO/job_scheduler] Job 1365 finished 1 steps successfully
[2017-06-01 19:18:55,669: INFO/job_scheduler] Job 1366: complete_job: Updating cache
[2017-06-01 19:18:55,670: INFO/job_scheduler] Job 1366 completing (errored=False, cancelled=False)
[2017-06-01 19:18:55,794: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:18:55,862: INFO/job_scheduler] Job 1365: complete_job: Updating cache
[2017-06-01 19:18:55,863: INFO/job_scheduler] Job 1365 completing (errored=False, cancelled=False)
[2017-06-01 19:18:55,929: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:18:57,809: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:57,810: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:57,835: INFO/command_plan] add_jobs: created Job 1367 (Test for host connectivity)
[2017-06-01 19:18:57,848: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:18:57,924: INFO/job_scheduler] Job 1367: RunJobThread.run
[2017-06-01 19:18:57,994: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:57,995: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:58,019: INFO/command_plan] add_jobs: created Job 1368 (Test for host connectivity)
[2017-06-01 19:18:58,088: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:18:58,111: INFO/job_scheduler] Job 1368: RunJobThread.run
[2017-06-01 19:18:58,174: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:58,175: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:58,226: INFO/command_plan] add_jobs: created Job 1369 (Test for host connectivity)
[2017-06-01 19:18:58,265: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:18:58,277: INFO/job_scheduler] Job 1369: RunJobThread.run
[2017-06-01 19:18:58,406: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:18:58,406: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:18:58,441: INFO/command_plan] add_jobs: created Job 1370 (Test for host connectivity)
[2017-06-01 19:18:58,524: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:18:58,591: INFO/job_scheduler] Job 1370: RunJobThread.run
[2017-06-01 19:19:00,245: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:19:00,455: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:19:00,770: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:19:00,787: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:19:01,577: INFO/job_scheduler] Job 1367 finished 1 steps successfully
[2017-06-01 19:19:01,618: INFO/job_scheduler] Job 1367: complete_job: Updating cache
[2017-06-01 19:19:01,618: INFO/job_scheduler] Job 1367 completing (errored=False, cancelled=False)
[2017-06-01 19:19:01,661: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:19:01,816: INFO/job_scheduler] Job 1368 finished 1 steps successfully
[2017-06-01 19:19:01,878: INFO/job_scheduler] Job 1368: complete_job: Updating cache
[2017-06-01 19:19:01,878: INFO/job_scheduler] Job 1368 completing (errored=False, cancelled=False)
[2017-06-01 19:19:02,002: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:19:02,257: INFO/job_scheduler] Job 1370 finished 1 steps successfully
[2017-06-01 19:19:02,284: INFO/job_scheduler] Job 1370: complete_job: Updating cache
[2017-06-01 19:19:02,284: INFO/job_scheduler] Job 1370 completing (errored=False, cancelled=False)
[2017-06-01 19:19:02,305: INFO/job_scheduler] Job 1369 finished 1 steps successfully
[2017-06-01 19:19:02,378: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:19:02,405: INFO/job_scheduler] Job 1369: complete_job: Updating cache
[2017-06-01 19:19:02,420: INFO/job_scheduler] Job 1369 completing (errored=False, cancelled=False)
[2017-06-01 19:19:02,528: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:19:04,219: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:04,219: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:04,334: INFO/command_plan] add_jobs: created Job 1371 (Test for host connectivity)
[2017-06-01 19:19:04,403: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:19:04,415: INFO/job_scheduler] Job 1371: RunJobThread.run
[2017-06-01 19:19:04,455: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:04,459: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:04,478: INFO/command_plan] add_jobs: created Job 1372 (Test for host connectivity)
[2017-06-01 19:19:04,527: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:19:04,569: INFO/job_scheduler] Job 1372: RunJobThread.run
[2017-06-01 19:19:04,636: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:04,637: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:04,667: INFO/command_plan] add_jobs: created Job 1373 (Test for host connectivity)
[2017-06-01 19:19:04,773: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:19:04,776: INFO/job_scheduler] Job 1373: RunJobThread.run
[2017-06-01 19:19:04,820: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:04,820: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:04,842: INFO/command_plan] add_jobs: created Job 1374 (Test for host connectivity)
[2017-06-01 19:19:04,965: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:19:05,019: INFO/job_scheduler] Job 1374: RunJobThread.run
[2017-06-01 19:19:06,689: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:19:06,950: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:19:07,040: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:19:07,243: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:19:08,033: INFO/job_scheduler] Job 1371 finished 1 steps successfully
[2017-06-01 19:19:08,050: INFO/job_scheduler] Job 1371: complete_job: Updating cache
[2017-06-01 19:19:08,050: INFO/job_scheduler] Job 1371 completing (errored=False, cancelled=False)
[2017-06-01 19:19:08,101: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:19:08,364: INFO/job_scheduler] Job 1372 finished 1 steps successfully
[2017-06-01 19:19:08,433: INFO/job_scheduler] Job 1372: complete_job: Updating cache
[2017-06-01 19:19:08,433: INFO/job_scheduler] Job 1372 completing (errored=False, cancelled=False)
[2017-06-01 19:19:08,495: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:19:08,522: INFO/job_scheduler] Job 1373 finished 1 steps successfully
[2017-06-01 19:19:08,558: INFO/job_scheduler] Job 1373: complete_job: Updating cache
[2017-06-01 19:19:08,558: INFO/job_scheduler] Job 1373 completing (errored=False, cancelled=False)
[2017-06-01 19:19:08,639: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:19:08,675: INFO/job_scheduler] Job 1374 finished 1 steps successfully
[2017-06-01 19:19:08,725: INFO/job_scheduler] Job 1374: complete_job: Updating cache
[2017-06-01 19:19:08,725: INFO/job_scheduler] Job 1374 completing (errored=False, cancelled=False)
[2017-06-01 19:19:08,783: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:19:10,672: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:10,672: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:10,685: INFO/command_plan] add_jobs: created Job 1375 (Test for host connectivity)
[2017-06-01 19:19:10,778: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:19:10,802: INFO/job_scheduler] Job 1375: RunJobThread.run
[2017-06-01 19:19:10,840: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:10,841: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:10,876: INFO/command_plan] add_jobs: created Job 1376 (Test for host connectivity)
[2017-06-01 19:19:10,957: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:10,957: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:11,049: INFO/command_plan] add_jobs: created Job 1377 (Test for host connectivity)
[2017-06-01 19:19:11,107: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:19:11,124: INFO/job_scheduler] Job 1376: RunJobThread.run
[2017-06-01 19:19:11,130: INFO/job_scheduler] Job 1377: RunJobThread.run
[2017-06-01 19:19:11,142: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:19:11,180: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:11,186: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:11,337: INFO/command_plan] add_jobs: created Job 1378 (Test for host connectivity)
[2017-06-01 19:19:11,365: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:19:11,382: INFO/job_scheduler] Job 1378: RunJobThread.run
[2017-06-01 19:19:13,101: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:19:13,312: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:19:13,344: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:19:13,608: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:19:14,437: INFO/job_scheduler] Job 1375 finished 1 steps successfully
[2017-06-01 19:19:14,523: INFO/job_scheduler] Job 1375: complete_job: Updating cache
[2017-06-01 19:19:14,524: INFO/job_scheduler] Job 1375 completing (errored=False, cancelled=False)
[2017-06-01 19:19:14,646: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:19:14,706: INFO/job_scheduler] Job 1377 finished 1 steps successfully
[2017-06-01 19:19:14,708: INFO/job_scheduler] Job 1376 finished 1 steps successfully
[2017-06-01 19:19:14,775: INFO/job_scheduler] Job 1377: complete_job: Updating cache
[2017-06-01 19:19:14,776: INFO/job_scheduler] Job 1377 completing (errored=False, cancelled=False)
[2017-06-01 19:19:14,823: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:19:14,869: INFO/job_scheduler] Job 1376: complete_job: Updating cache
[2017-06-01 19:19:14,869: INFO/job_scheduler] Job 1376 completing (errored=False, cancelled=False)
[2017-06-01 19:19:15,016: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:19:15,062: INFO/job_scheduler] Job 1378 finished 1 steps successfully
[2017-06-01 19:19:15,098: INFO/job_scheduler] Job 1378: complete_job: Updating cache
[2017-06-01 19:19:15,098: INFO/job_scheduler] Job 1378 completing (errored=False, cancelled=False)
[2017-06-01 19:19:15,141: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:19:17,105: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:17,107: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:17,123: INFO/command_plan] add_jobs: created Job 1379 (Test for host connectivity)
[2017-06-01 19:19:17,143: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:19:17,145: INFO/job_scheduler] Job 1379: RunJobThread.run
[2017-06-01 19:19:17,206: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:17,207: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:17,218: INFO/command_plan] add_jobs: created Job 1380 (Test for host connectivity)
[2017-06-01 19:19:17,253: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:19:17,261: INFO/job_scheduler] Job 1380: RunJobThread.run
[2017-06-01 19:19:17,358: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:17,358: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:17,378: INFO/command_plan] add_jobs: created Job 1381 (Test for host connectivity)
[2017-06-01 19:19:17,494: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:19:17,504: INFO/job_scheduler] Job 1381: RunJobThread.run
[2017-06-01 19:19:17,534: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:17,534: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:17,556: INFO/command_plan] add_jobs: created Job 1382 (Test for host connectivity)
[2017-06-01 19:19:17,623: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:19:17,643: INFO/job_scheduler] Job 1382: RunJobThread.run
[2017-06-01 19:19:19,450: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:19:19,638: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:19:19,788: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:19:19,954: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:19:20,773: INFO/job_scheduler] Job 1379 finished 1 steps successfully
[2017-06-01 19:19:20,791: INFO/job_scheduler] Job 1379: complete_job: Updating cache
[2017-06-01 19:19:20,791: INFO/job_scheduler] Job 1379 completing (errored=False, cancelled=False)
[2017-06-01 19:19:20,846: INFO/job_scheduler] Job 1380 finished 1 steps successfully
[2017-06-01 19:19:20,850: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:19:20,875: INFO/job_scheduler] Job 1380: complete_job: Updating cache
[2017-06-01 19:19:20,875: INFO/job_scheduler] Job 1380 completing (errored=False, cancelled=False)
[2017-06-01 19:19:21,035: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:19:21,321: INFO/job_scheduler] Job 1381 finished 1 steps successfully
[2017-06-01 19:19:21,340: INFO/job_scheduler] Job 1381: complete_job: Updating cache
[2017-06-01 19:19:21,340: INFO/job_scheduler] Job 1381 completing (errored=False, cancelled=False)
[2017-06-01 19:19:21,368: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:19:21,465: INFO/job_scheduler] Job 1382 finished 1 steps successfully
[2017-06-01 19:19:21,517: INFO/job_scheduler] Job 1382: complete_job: Updating cache
[2017-06-01 19:19:21,520: INFO/job_scheduler] Job 1382 completing (errored=False, cancelled=False)
[2017-06-01 19:19:21,585: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:19:23,352: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:23,353: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:23,365: INFO/command_plan] add_jobs: created Job 1383 (Test for host connectivity)
[2017-06-01 19:19:23,406: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:19:23,413: INFO/job_scheduler] Job 1383: RunJobThread.run
[2017-06-01 19:19:23,516: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:23,523: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:23,542: INFO/command_plan] add_jobs: created Job 1384 (Test for host connectivity)
[2017-06-01 19:19:23,641: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:19:23,654: INFO/job_scheduler] Job 1384: RunJobThread.run
[2017-06-01 19:19:23,690: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:23,690: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:23,701: INFO/command_plan] add_jobs: created Job 1385 (Test for host connectivity)
[2017-06-01 19:19:23,729: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:19:23,768: INFO/job_scheduler] Job 1385: RunJobThread.run
[2017-06-01 19:19:23,883: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:23,885: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:23,916: INFO/command_plan] add_jobs: created Job 1386 (Test for host connectivity)
[2017-06-01 19:19:23,964: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:19:23,980: INFO/job_scheduler] Job 1386: RunJobThread.run
[2017-06-01 19:19:25,879: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:19:25,923: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:19:26,015: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:19:26,210: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:19:27,356: INFO/job_scheduler] Job 1383 finished 1 steps successfully
[2017-06-01 19:19:27,387: INFO/job_scheduler] Job 1384 finished 1 steps successfully
[2017-06-01 19:19:27,407: INFO/job_scheduler] Job 1383: complete_job: Updating cache
[2017-06-01 19:19:27,407: INFO/job_scheduler] Job 1383 completing (errored=False, cancelled=False)
[2017-06-01 19:19:27,484: INFO/job_scheduler] Job 1385 finished 1 steps successfully
[2017-06-01 19:19:27,502: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:19:27,534: INFO/job_scheduler] Job 1384: complete_job: Updating cache
[2017-06-01 19:19:27,535: INFO/job_scheduler] Job 1384 completing (errored=False, cancelled=False)
[2017-06-01 19:19:27,561: INFO/job_scheduler] Job 1386 finished 1 steps successfully
[2017-06-01 19:19:27,632: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:19:27,649: INFO/job_scheduler] Job 1385: complete_job: Updating cache
[2017-06-01 19:19:27,650: INFO/job_scheduler] Job 1385 completing (errored=False, cancelled=False)
[2017-06-01 19:19:27,723: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:19:27,759: INFO/job_scheduler] Job 1386: complete_job: Updating cache
[2017-06-01 19:19:27,759: INFO/job_scheduler] Job 1386 completing (errored=False, cancelled=False)
[2017-06-01 19:19:27,839: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:19:29,733: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:29,733: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:29,755: INFO/command_plan] add_jobs: created Job 1387 (Test for host connectivity)
[2017-06-01 19:19:29,854: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:29,855: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:29,879: INFO/command_plan] add_jobs: created Job 1388 (Test for host connectivity)
[2017-06-01 19:19:29,898: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:19:29,925: INFO/job_scheduler] Job 1387: RunJobThread.run
[2017-06-01 19:19:29,962: INFO/job_scheduler] Job 1388: RunJobThread.run
[2017-06-01 19:19:29,973: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:19:30,134: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:30,135: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:30,164: INFO/command_plan] add_jobs: created Job 1389 (Test for host connectivity)
[2017-06-01 19:19:30,226: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:19:30,256: INFO/job_scheduler] Job 1389: RunJobThread.run
[2017-06-01 19:19:30,288: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:30,288: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:30,313: INFO/command_plan] add_jobs: created Job 1390 (Test for host connectivity)
[2017-06-01 19:19:30,377: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:19:30,422: INFO/job_scheduler] Job 1390: RunJobThread.run
[2017-06-01 19:19:32,297: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:19:32,352: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:19:32,595: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:19:32,684: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:19:33,637: INFO/job_scheduler] Job 1387 finished 1 steps successfully
[2017-06-01 19:19:33,655: INFO/job_scheduler] Job 1387: complete_job: Updating cache
[2017-06-01 19:19:33,655: INFO/job_scheduler] Job 1387 completing (errored=False, cancelled=False)
[2017-06-01 19:19:33,732: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:19:33,752: INFO/job_scheduler] Job 1388 finished 1 steps successfully
[2017-06-01 19:19:33,838: INFO/job_scheduler] Job 1388: complete_job: Updating cache
[2017-06-01 19:19:33,839: INFO/job_scheduler] Job 1388 completing (errored=False, cancelled=False)
[2017-06-01 19:19:33,946: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:19:34,044: INFO/job_scheduler] Job 1390 finished 1 steps successfully
[2017-06-01 19:19:34,045: INFO/job_scheduler] Job 1389 finished 1 steps successfully
[2017-06-01 19:19:34,055: INFO/job_scheduler] Job 1390: complete_job: Updating cache
[2017-06-01 19:19:34,055: INFO/job_scheduler] Job 1390 completing (errored=False, cancelled=False)
[2017-06-01 19:19:34,115: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:19:34,139: INFO/job_scheduler] Job 1389: complete_job: Updating cache
[2017-06-01 19:19:34,139: INFO/job_scheduler] Job 1389 completing (errored=False, cancelled=False)
[2017-06-01 19:19:34,193: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:19:36,213: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:36,215: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:36,221: INFO/command_plan] add_jobs: created Job 1391 (Test for host connectivity)
[2017-06-01 19:19:36,315: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:36,315: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:36,332: INFO/command_plan] add_jobs: created Job 1392 (Test for host connectivity)
[2017-06-01 19:19:36,347: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:19:36,372: INFO/job_scheduler] Job 1392: RunJobThread.run
[2017-06-01 19:19:36,379: INFO/job_scheduler] Job 1391: RunJobThread.run
[2017-06-01 19:19:36,392: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:19:36,492: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:36,493: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:36,573: INFO/command_plan] add_jobs: created Job 1393 (Test for host connectivity)
[2017-06-01 19:19:36,609: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:19:36,652: INFO/job_scheduler] Job 1393: RunJobThread.run
[2017-06-01 19:19:36,693: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:36,693: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:36,704: INFO/command_plan] add_jobs: created Job 1394 (Test for host connectivity)
[2017-06-01 19:19:36,808: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:19:36,815: INFO/job_scheduler] Job 1394: RunJobThread.run
[2017-06-01 19:19:38,673: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:19:38,769: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:19:38,944: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:19:39,048: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:19:39,999: INFO/job_scheduler] Job 1391 finished 1 steps successfully
[2017-06-01 19:19:40,050: INFO/job_scheduler] Job 1391: complete_job: Updating cache
[2017-06-01 19:19:40,050: INFO/job_scheduler] Job 1391 completing (errored=False, cancelled=False)
[2017-06-01 19:19:40,120: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:19:40,128: INFO/job_scheduler] Job 1392 finished 1 steps successfully
[2017-06-01 19:19:40,208: INFO/job_scheduler] Job 1392: complete_job: Updating cache
[2017-06-01 19:19:40,208: INFO/job_scheduler] Job 1392 completing (errored=False, cancelled=False)
[2017-06-01 19:19:40,282: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:19:40,334: INFO/job_scheduler] Job 1393 finished 1 steps successfully
[2017-06-01 19:19:40,380: INFO/job_scheduler] Job 1393: complete_job: Updating cache
[2017-06-01 19:19:40,381: INFO/job_scheduler] Job 1393 completing (errored=False, cancelled=False)
[2017-06-01 19:19:40,412: INFO/job_scheduler] Job 1394 finished 1 steps successfully
[2017-06-01 19:19:40,505: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:19:40,584: INFO/job_scheduler] Job 1394: complete_job: Updating cache
[2017-06-01 19:19:40,584: INFO/job_scheduler] Job 1394 completing (errored=False, cancelled=False)
[2017-06-01 19:19:40,630: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:19:42,551: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:42,552: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:42,575: INFO/command_plan] add_jobs: created Job 1395 (Test for host connectivity)
[2017-06-01 19:19:42,598: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:19:42,602: INFO/job_scheduler] Job 1395: RunJobThread.run
[2017-06-01 19:19:42,728: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:42,729: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:42,861: INFO/command_plan] add_jobs: created Job 1396 (Test for host connectivity)
[2017-06-01 19:19:42,937: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:42,937: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:42,981: INFO/command_plan] add_jobs: created Job 1397 (Test for host connectivity)
[2017-06-01 19:19:43,103: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:19:43,106: INFO/job_scheduler] Job 1396: RunJobThread.run
[2017-06-01 19:19:43,107: INFO/job_scheduler] Job 1397: RunJobThread.run
[2017-06-01 19:19:43,144: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:19:43,171: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:43,172: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:43,216: INFO/command_plan] add_jobs: created Job 1398 (Test for host connectivity)
[2017-06-01 19:19:43,296: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:19:43,360: INFO/job_scheduler] Job 1398: RunJobThread.run
[2017-06-01 19:19:44,974: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:19:45,430: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:19:45,487: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:19:45,647: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:19:46,347: INFO/job_scheduler] Job 1395 finished 1 steps successfully
[2017-06-01 19:19:46,369: INFO/job_scheduler] Job 1395: complete_job: Updating cache
[2017-06-01 19:19:46,369: INFO/job_scheduler] Job 1395 completing (errored=False, cancelled=False)
[2017-06-01 19:19:46,405: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:19:46,970: INFO/job_scheduler] Job 1397 finished 1 steps successfully
[2017-06-01 19:19:46,979: INFO/job_scheduler] Job 1396 finished 1 steps successfully
[2017-06-01 19:19:47,010: INFO/job_scheduler] Job 1397: complete_job: Updating cache
[2017-06-01 19:19:47,011: INFO/job_scheduler] Job 1397 completing (errored=False, cancelled=False)
[2017-06-01 19:19:47,050: INFO/job_scheduler] Job 1398 finished 1 steps successfully
[2017-06-01 19:19:47,069: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:19:47,085: INFO/job_scheduler] Job 1396: complete_job: Updating cache
[2017-06-01 19:19:47,086: INFO/job_scheduler] Job 1396 completing (errored=False, cancelled=False)
[2017-06-01 19:19:47,194: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:19:47,261: INFO/job_scheduler] Job 1398: complete_job: Updating cache
[2017-06-01 19:19:47,261: INFO/job_scheduler] Job 1398 completing (errored=False, cancelled=False)
[2017-06-01 19:19:47,344: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:19:48,986: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:48,986: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:49,007: INFO/command_plan] add_jobs: created Job 1399 (Test for host connectivity)
[2017-06-01 19:19:49,090: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:19:49,110: INFO/job_scheduler] Job 1399: RunJobThread.run
[2017-06-01 19:19:49,154: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:49,155: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:49,187: INFO/command_plan] add_jobs: created Job 1400 (Test for host connectivity)
[2017-06-01 19:19:49,213: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:19:49,267: INFO/job_scheduler] Job 1400: RunJobThread.run
[2017-06-01 19:19:49,347: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:49,349: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:49,362: INFO/command_plan] add_jobs: created Job 1401 (Test for host connectivity)
[2017-06-01 19:19:49,437: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:19:49,463: INFO/job_scheduler] Job 1401: RunJobThread.run
[2017-06-01 19:19:49,528: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:49,529: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:49,552: INFO/command_plan] add_jobs: created Job 1402 (Test for host connectivity)
[2017-06-01 19:19:49,667: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:19:49,683: INFO/job_scheduler] Job 1402: RunJobThread.run
[2017-06-01 19:19:51,540: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:19:51,543: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:19:51,806: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:19:52,007: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:19:52,908: INFO/job_scheduler] Job 1399 finished 1 steps successfully
[2017-06-01 19:19:52,967: INFO/job_scheduler] Job 1399: complete_job: Updating cache
[2017-06-01 19:19:52,968: INFO/job_scheduler] Job 1399 completing (errored=False, cancelled=False)
[2017-06-01 19:19:52,978: INFO/job_scheduler] Job 1400 finished 1 steps successfully
[2017-06-01 19:19:53,036: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:19:53,054: INFO/job_scheduler] Job 1400: complete_job: Updating cache
[2017-06-01 19:19:53,054: INFO/job_scheduler] Job 1400 completing (errored=False, cancelled=False)
[2017-06-01 19:19:53,145: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:19:53,186: INFO/job_scheduler] Job 1401 finished 1 steps successfully
[2017-06-01 19:19:53,245: INFO/job_scheduler] Job 1401: complete_job: Updating cache
[2017-06-01 19:19:53,245: INFO/job_scheduler] Job 1401 completing (errored=False, cancelled=False)
[2017-06-01 19:19:53,330: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:19:53,404: INFO/job_scheduler] Job 1402 finished 1 steps successfully
[2017-06-01 19:19:53,460: INFO/job_scheduler] Job 1402: complete_job: Updating cache
[2017-06-01 19:19:53,462: INFO/job_scheduler] Job 1402 completing (errored=False, cancelled=False)
[2017-06-01 19:19:53,501: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:19:55,383: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:55,384: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:55,411: INFO/command_plan] add_jobs: created Job 1403 (Test for host connectivity)
[2017-06-01 19:19:55,424: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:19:55,437: INFO/job_scheduler] Job 1403: RunJobThread.run
[2017-06-01 19:19:55,512: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:55,512: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:55,568: INFO/command_plan] add_jobs: created Job 1404 (Test for host connectivity)
[2017-06-01 19:19:55,645: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:19:55,689: INFO/job_scheduler] Job 1404: RunJobThread.run
[2017-06-01 19:19:55,713: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:55,714: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:55,736: INFO/command_plan] add_jobs: created Job 1405 (Test for host connectivity)
[2017-06-01 19:19:55,772: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:19:55,795: INFO/job_scheduler] Job 1405: RunJobThread.run
[2017-06-01 19:19:55,904: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:19:55,905: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:19:55,945: INFO/command_plan] add_jobs: created Job 1406 (Test for host connectivity)
[2017-06-01 19:19:55,998: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:19:56,007: INFO/job_scheduler] Job 1406: RunJobThread.run
[2017-06-01 19:19:57,741: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:19:57,881: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:19:58,051: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:19:58,253: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:19:59,087: INFO/job_scheduler] Job 1403 finished 1 steps successfully
[2017-06-01 19:19:59,132: INFO/job_scheduler] Job 1403: complete_job: Updating cache
[2017-06-01 19:19:59,134: INFO/job_scheduler] Job 1403 completing (errored=False, cancelled=False)
[2017-06-01 19:19:59,190: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:19:59,270: INFO/job_scheduler] Job 1404 finished 1 steps successfully
[2017-06-01 19:19:59,314: INFO/job_scheduler] Job 1404: complete_job: Updating cache
[2017-06-01 19:19:59,315: INFO/job_scheduler] Job 1404 completing (errored=False, cancelled=False)
[2017-06-01 19:19:59,414: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:19:59,473: INFO/job_scheduler] Job 1405 finished 1 steps successfully
[2017-06-01 19:19:59,538: INFO/job_scheduler] Job 1405: complete_job: Updating cache
[2017-06-01 19:19:59,539: INFO/job_scheduler] Job 1405 completing (errored=False, cancelled=False)
[2017-06-01 19:19:59,606: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:19:59,694: INFO/job_scheduler] Job 1406 finished 1 steps successfully
[2017-06-01 19:19:59,761: INFO/job_scheduler] Job 1406: complete_job: Updating cache
[2017-06-01 19:19:59,761: INFO/job_scheduler] Job 1406 completing (errored=False, cancelled=False)
[2017-06-01 19:19:59,844: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:20:01,717: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:01,718: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:01,746: INFO/command_plan] add_jobs: created Job 1407 (Test for host connectivity)
[2017-06-01 19:20:01,821: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:20:01,850: INFO/job_scheduler] Job 1407: RunJobThread.run
[2017-06-01 19:20:01,890: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:01,890: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:01,912: INFO/command_plan] add_jobs: created Job 1408 (Test for host connectivity)
[2017-06-01 19:20:01,954: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:20:02,007: INFO/job_scheduler] Job 1408: RunJobThread.run
[2017-06-01 19:20:02,072: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:02,073: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:02,118: INFO/command_plan] add_jobs: created Job 1409 (Test for host connectivity)
[2017-06-01 19:20:02,174: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:20:02,207: INFO/job_scheduler] Job 1409: RunJobThread.run
[2017-06-01 19:20:02,256: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:02,257: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:02,274: INFO/command_plan] add_jobs: created Job 1410 (Test for host connectivity)
[2017-06-01 19:20:02,378: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:20:02,419: INFO/job_scheduler] Job 1410: RunJobThread.run
[2017-06-01 19:20:04,202: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:20:04,323: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:20:04,492: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:20:04,634: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:20:05,540: INFO/job_scheduler] Job 1407 finished 1 steps successfully
[2017-06-01 19:20:05,622: INFO/job_scheduler] Job 1407: complete_job: Updating cache
[2017-06-01 19:20:05,624: INFO/job_scheduler] Job 1407 completing (errored=False, cancelled=False)
[2017-06-01 19:20:05,692: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:20:05,940: INFO/job_scheduler] Job 1409 finished 1 steps successfully
[2017-06-01 19:20:05,993: INFO/job_scheduler] Job 1409: complete_job: Updating cache
[2017-06-01 19:20:05,993: INFO/job_scheduler] Job 1409 completing (errored=False, cancelled=False)
[2017-06-01 19:20:06,036: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:20:06,281: INFO/job_scheduler] Job 1408 finished 1 steps successfully
[2017-06-01 19:20:06,349: INFO/job_scheduler] Job 1408: complete_job: Updating cache
[2017-06-01 19:20:06,349: INFO/job_scheduler] Job 1408 completing (errored=False, cancelled=False)
[2017-06-01 19:20:06,384: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:20:06,472: INFO/job_scheduler] Job 1410 finished 1 steps successfully
[2017-06-01 19:20:06,547: INFO/job_scheduler] Job 1410: complete_job: Updating cache
[2017-06-01 19:20:06,547: INFO/job_scheduler] Job 1410 completing (errored=False, cancelled=False)
[2017-06-01 19:20:06,606: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:20:08,828: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:08,830: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:08,841: INFO/command_plan] add_jobs: created Job 1411 (Test for host connectivity)
[2017-06-01 19:20:08,893: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:20:08,925: INFO/job_scheduler] Job 1411: RunJobThread.run
[2017-06-01 19:20:08,989: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:08,989: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:08,999: INFO/command_plan] add_jobs: created Job 1412 (Test for host connectivity)
[2017-06-01 19:20:09,051: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:20:09,126: INFO/job_scheduler] Job 1412: RunJobThread.run
[2017-06-01 19:20:09,190: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:09,191: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:09,296: INFO/command_plan] add_jobs: created Job 1413 (Test for host connectivity)
[2017-06-01 19:20:09,355: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:09,355: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:09,366: INFO/command_plan] add_jobs: created Job 1414 (Test for host connectivity)
[2017-06-01 19:20:09,396: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:20:09,433: INFO/job_scheduler] Job 1413: RunJobThread.run
[2017-06-01 19:20:09,464: INFO/job_scheduler] Job 1414: RunJobThread.run
[2017-06-01 19:20:09,477: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:20:11,312: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:20:11,399: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:20:11,767: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:20:11,775: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:20:12,692: INFO/job_scheduler] Job 1411 finished 1 steps successfully
[2017-06-01 19:20:12,739: INFO/job_scheduler] Job 1411: complete_job: Updating cache
[2017-06-01 19:20:12,740: INFO/job_scheduler] Job 1411 completing (errored=False, cancelled=False)
[2017-06-01 19:20:12,774: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:20:12,978: INFO/job_scheduler] Job 1412 finished 1 steps successfully
[2017-06-01 19:20:13,016: INFO/job_scheduler] Job 1412: complete_job: Updating cache
[2017-06-01 19:20:13,017: INFO/job_scheduler] Job 1412 completing (errored=False, cancelled=False)
[2017-06-01 19:20:13,098: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:20:13,220: INFO/job_scheduler] Job 1414 finished 1 steps successfully
[2017-06-01 19:20:13,264: INFO/job_scheduler] Job 1414: complete_job: Updating cache
[2017-06-01 19:20:13,265: INFO/job_scheduler] Job 1414 completing (errored=False, cancelled=False)
[2017-06-01 19:20:13,320: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:20:13,363: INFO/job_scheduler] Job 1413 finished 1 steps successfully
[2017-06-01 19:20:13,381: INFO/job_scheduler] Job 1413: complete_job: Updating cache
[2017-06-01 19:20:13,381: INFO/job_scheduler] Job 1413 completing (errored=False, cancelled=False)
[2017-06-01 19:20:13,434: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:20:15,120: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:15,120: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:15,131: INFO/command_plan] add_jobs: created Job 1415 (Test for host connectivity)
[2017-06-01 19:20:15,157: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:20:15,195: INFO/job_scheduler] Job 1415: RunJobThread.run
[2017-06-01 19:20:15,279: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:15,279: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:15,298: INFO/command_plan] add_jobs: created Job 1416 (Test for host connectivity)
[2017-06-01 19:20:15,372: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:20:15,401: INFO/job_scheduler] Job 1416: RunJobThread.run
[2017-06-01 19:20:15,543: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:15,543: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:15,556: INFO/command_plan] add_jobs: created Job 1417 (Test for host connectivity)
[2017-06-01 19:20:15,608: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:20:15,611: INFO/job_scheduler] Job 1417: RunJobThread.run
[2017-06-01 19:20:15,671: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:15,671: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:15,697: INFO/command_plan] add_jobs: created Job 1418 (Test for host connectivity)
[2017-06-01 19:20:15,767: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:20:15,793: INFO/job_scheduler] Job 1418: RunJobThread.run
[2017-06-01 19:20:17,717: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:20:17,736: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:20:18,094: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:20:18,095: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:20:19,126: INFO/job_scheduler] Job 1416 finished 1 steps successfully
[2017-06-01 19:20:19,144: INFO/job_scheduler] Job 1415 finished 1 steps successfully
[2017-06-01 19:20:19,175: INFO/job_scheduler] Job 1416: complete_job: Updating cache
[2017-06-01 19:20:19,175: INFO/job_scheduler] Job 1416 completing (errored=False, cancelled=False)
[2017-06-01 19:20:19,263: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:20:19,280: INFO/job_scheduler] Job 1415: complete_job: Updating cache
[2017-06-01 19:20:19,281: INFO/job_scheduler] Job 1415 completing (errored=False, cancelled=False)
[2017-06-01 19:20:19,343: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:20:19,499: INFO/job_scheduler] Job 1417 finished 1 steps successfully
[2017-06-01 19:20:19,501: INFO/job_scheduler] Job 1418 finished 1 steps successfully
[2017-06-01 19:20:19,530: INFO/job_scheduler] Job 1417: complete_job: Updating cache
[2017-06-01 19:20:19,530: INFO/job_scheduler] Job 1417 completing (errored=False, cancelled=False)
[2017-06-01 19:20:19,581: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:20:19,625: INFO/job_scheduler] Job 1418: complete_job: Updating cache
[2017-06-01 19:20:19,650: INFO/job_scheduler] Job 1418 completing (errored=False, cancelled=False)
[2017-06-01 19:20:19,743: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:20:21,551: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:21,553: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:21,564: INFO/command_plan] add_jobs: created Job 1419 (Test for host connectivity)
[2017-06-01 19:20:21,632: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:21,632: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:21,645: INFO/command_plan] add_jobs: created Job 1420 (Test for host connectivity)
[2017-06-01 19:20:21,667: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:20:21,692: INFO/job_scheduler] Job 1419: RunJobThread.run
[2017-06-01 19:20:21,700: INFO/job_scheduler] Job 1420: RunJobThread.run
[2017-06-01 19:20:21,713: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:20:21,799: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:21,800: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:21,912: INFO/command_plan] add_jobs: created Job 1421 (Test for host connectivity)
[2017-06-01 19:20:21,977: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:20:22,006: INFO/job_scheduler] Job 1421: RunJobThread.run
[2017-06-01 19:20:22,068: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:22,071: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:22,104: INFO/command_plan] add_jobs: created Job 1422 (Test for host connectivity)
[2017-06-01 19:20:22,220: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:20:22,249: INFO/job_scheduler] Job 1422: RunJobThread.run
[2017-06-01 19:20:24,118: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:20:24,174: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:20:24,487: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:20:24,542: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:20:25,498: INFO/job_scheduler] Job 1419 finished 1 steps successfully
[2017-06-01 19:20:25,562: INFO/job_scheduler] Job 1419: complete_job: Updating cache
[2017-06-01 19:20:25,563: INFO/job_scheduler] Job 1419 completing (errored=False, cancelled=False)
[2017-06-01 19:20:25,599: INFO/job_scheduler] Job 1420 finished 1 steps successfully
[2017-06-01 19:20:25,676: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:20:25,726: INFO/job_scheduler] Job 1420: complete_job: Updating cache
[2017-06-01 19:20:25,727: INFO/job_scheduler] Job 1420 completing (errored=False, cancelled=False)
[2017-06-01 19:20:25,804: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:20:25,906: INFO/job_scheduler] Job 1421 finished 1 steps successfully
[2017-06-01 19:20:25,916: INFO/job_scheduler] Job 1422 finished 1 steps successfully
[2017-06-01 19:20:25,962: INFO/job_scheduler] Job 1421: complete_job: Updating cache
[2017-06-01 19:20:25,962: INFO/job_scheduler] Job 1421 completing (errored=False, cancelled=False)
[2017-06-01 19:20:25,996: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:20:26,012: INFO/job_scheduler] Job 1422: complete_job: Updating cache
[2017-06-01 19:20:26,013: INFO/job_scheduler] Job 1422 completing (errored=False, cancelled=False)
[2017-06-01 19:20:26,154: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:20:27,946: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:27,946: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:27,978: INFO/command_plan] add_jobs: created Job 1423 (Test for host connectivity)
[2017-06-01 19:20:28,075: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:28,075: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:28,095: INFO/command_plan] add_jobs: created Job 1424 (Test for host connectivity)
[2017-06-01 19:20:28,104: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:20:28,136: INFO/job_scheduler] Job 1424: RunJobThread.run
[2017-06-01 19:20:28,191: INFO/job_scheduler] Job 1423: RunJobThread.run
[2017-06-01 19:20:28,197: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:20:28,267: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:28,268: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:28,289: INFO/command_plan] add_jobs: created Job 1425 (Test for host connectivity)
[2017-06-01 19:20:28,380: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:28,380: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:28,386: INFO/command_plan] add_jobs: created Job 1426 (Test for host connectivity)
[2017-06-01 19:20:28,429: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:20:28,441: INFO/job_scheduler] Job 1425: RunJobThread.run
[2017-06-01 19:20:28,475: INFO/job_scheduler] Job 1426: RunJobThread.run
[2017-06-01 19:20:28,500: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:20:30,536: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:20:30,555: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:20:30,707: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:20:30,792: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:20:31,897: INFO/job_scheduler] Job 1423 finished 1 steps successfully
[2017-06-01 19:20:31,933: INFO/job_scheduler] Job 1424 finished 1 steps successfully
[2017-06-01 19:20:31,960: INFO/job_scheduler] Job 1423: complete_job: Updating cache
[2017-06-01 19:20:31,961: INFO/job_scheduler] Job 1423 completing (errored=False, cancelled=False)
[2017-06-01 19:20:32,040: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:20:32,061: INFO/job_scheduler] Job 1424: complete_job: Updating cache
[2017-06-01 19:20:32,061: INFO/job_scheduler] Job 1424 completing (errored=False, cancelled=False)
[2017-06-01 19:20:32,115: INFO/job_scheduler] Job 1425 finished 1 steps successfully
[2017-06-01 19:20:32,119: INFO/job_scheduler] Job 1426 finished 1 steps successfully
[2017-06-01 19:20:32,173: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:20:32,207: INFO/job_scheduler] Job 1425: complete_job: Updating cache
[2017-06-01 19:20:32,208: INFO/job_scheduler] Job 1425 completing (errored=False, cancelled=False)
[2017-06-01 19:20:32,290: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:20:32,335: INFO/job_scheduler] Job 1426: complete_job: Updating cache
[2017-06-01 19:20:32,336: INFO/job_scheduler] Job 1426 completing (errored=False, cancelled=False)
[2017-06-01 19:20:32,415: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:20:34,177: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:34,179: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:34,185: INFO/command_plan] add_jobs: created Job 1427 (Test for host connectivity)
[2017-06-01 19:20:34,256: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:20:34,264: INFO/job_scheduler] Job 1427: RunJobThread.run
[2017-06-01 19:20:34,324: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:34,325: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:34,348: INFO/command_plan] add_jobs: created Job 1428 (Test for host connectivity)
[2017-06-01 19:20:34,374: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:20:34,389: INFO/job_scheduler] Job 1428: RunJobThread.run
[2017-06-01 19:20:34,493: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:34,494: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:34,520: INFO/command_plan] add_jobs: created Job 1429 (Test for host connectivity)
[2017-06-01 19:20:34,658: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:34,661: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:34,723: INFO/command_plan] add_jobs: created Job 1430 (Test for host connectivity)
[2017-06-01 19:20:34,762: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:20:34,780: INFO/job_scheduler] Job 1429: RunJobThread.run
[2017-06-01 19:20:34,824: INFO/job_scheduler] Job 1430: RunJobThread.run
[2017-06-01 19:20:34,839: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:20:36,613: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:20:36,890: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:20:37,114: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:20:37,203: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:20:37,984: INFO/job_scheduler] Job 1427 finished 1 steps successfully
[2017-06-01 19:20:38,018: INFO/job_scheduler] Job 1427: complete_job: Updating cache
[2017-06-01 19:20:38,018: INFO/job_scheduler] Job 1427 completing (errored=False, cancelled=False)
[2017-06-01 19:20:38,130: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:20:38,356: INFO/job_scheduler] Job 1428 finished 1 steps successfully
[2017-06-01 19:20:38,392: INFO/job_scheduler] Job 1428: complete_job: Updating cache
[2017-06-01 19:20:38,392: INFO/job_scheduler] Job 1428 completing (errored=False, cancelled=False)
[2017-06-01 19:20:38,452: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:20:38,508: INFO/job_scheduler] Job 1430 finished 1 steps successfully
[2017-06-01 19:20:38,578: INFO/job_scheduler] Job 1430: complete_job: Updating cache
[2017-06-01 19:20:38,578: INFO/job_scheduler] Job 1430 completing (errored=False, cancelled=False)
[2017-06-01 19:20:38,662: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:20:38,713: INFO/job_scheduler] Job 1429 finished 1 steps successfully
[2017-06-01 19:20:38,751: INFO/job_scheduler] Job 1429: complete_job: Updating cache
[2017-06-01 19:20:38,751: INFO/job_scheduler] Job 1429 completing (errored=False, cancelled=False)
[2017-06-01 19:20:38,811: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:20:40,582: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:40,583: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:40,603: INFO/command_plan] add_jobs: created Job 1431 (Test for host connectivity)
[2017-06-01 19:20:40,649: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:20:40,652: INFO/job_scheduler] Job 1431: RunJobThread.run
[2017-06-01 19:20:40,708: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:40,709: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:40,740: INFO/command_plan] add_jobs: created Job 1432 (Test for host connectivity)
[2017-06-01 19:20:40,827: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:20:40,854: INFO/job_scheduler] Job 1432: RunJobThread.run
[2017-06-01 19:20:40,932: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:40,933: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:41,011: INFO/command_plan] add_jobs: created Job 1433 (Test for host connectivity)
[2017-06-01 19:20:41,054: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:20:41,083: INFO/job_scheduler] Job 1433: RunJobThread.run
[2017-06-01 19:20:41,135: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:41,135: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:41,153: INFO/command_plan] add_jobs: created Job 1434 (Test for host connectivity)
[2017-06-01 19:20:41,176: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:20:41,190: INFO/job_scheduler] Job 1434: RunJobThread.run
[2017-06-01 19:20:43,010: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:20:43,141: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:20:43,355: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:20:43,492: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:20:44,404: INFO/job_scheduler] Job 1431 finished 1 steps successfully
[2017-06-01 19:20:44,475: INFO/job_scheduler] Job 1431: complete_job: Updating cache
[2017-06-01 19:20:44,475: INFO/job_scheduler] Job 1431 completing (errored=False, cancelled=False)
[2017-06-01 19:20:44,567: INFO/job_scheduler] Job 1432 finished 1 steps successfully
[2017-06-01 19:20:44,593: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:20:44,617: INFO/job_scheduler] Job 1432: complete_job: Updating cache
[2017-06-01 19:20:44,617: INFO/job_scheduler] Job 1432 completing (errored=False, cancelled=False)
[2017-06-01 19:20:44,701: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:20:44,754: INFO/job_scheduler] Job 1433 finished 1 steps successfully
[2017-06-01 19:20:44,792: INFO/job_scheduler] Job 1433: complete_job: Updating cache
[2017-06-01 19:20:44,792: INFO/job_scheduler] Job 1433 completing (errored=False, cancelled=False)
[2017-06-01 19:20:44,845: INFO/job_scheduler] Job 1434 finished 1 steps successfully
[2017-06-01 19:20:44,875: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:20:44,899: INFO/job_scheduler] Job 1434: complete_job: Updating cache
[2017-06-01 19:20:44,899: INFO/job_scheduler] Job 1434 completing (errored=False, cancelled=False)
[2017-06-01 19:20:44,935: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:20:46,892: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:46,894: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:46,908: INFO/command_plan] add_jobs: created Job 1435 (Test for host connectivity)
[2017-06-01 19:20:46,938: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:20:46,957: INFO/job_scheduler] Job 1435: RunJobThread.run
[2017-06-01 19:20:47,010: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:47,011: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:47,027: INFO/command_plan] add_jobs: created Job 1436 (Test for host connectivity)
[2017-06-01 19:20:47,065: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:20:47,079: INFO/job_scheduler] Job 1436: RunJobThread.run
[2017-06-01 19:20:47,177: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:47,178: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:47,209: INFO/command_plan] add_jobs: created Job 1437 (Test for host connectivity)
[2017-06-01 19:20:47,265: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:20:47,305: INFO/job_scheduler] Job 1437: RunJobThread.run
[2017-06-01 19:20:47,378: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:47,379: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:47,432: INFO/command_plan] add_jobs: created Job 1438 (Test for host connectivity)
[2017-06-01 19:20:47,513: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:20:47,531: INFO/job_scheduler] Job 1438: RunJobThread.run
[2017-06-01 19:20:49,321: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:20:49,486: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:20:49,634: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:20:49,716: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:20:50,645: INFO/job_scheduler] Job 1435 finished 1 steps successfully
[2017-06-01 19:20:50,722: INFO/job_scheduler] Job 1435: complete_job: Updating cache
[2017-06-01 19:20:50,723: INFO/job_scheduler] Job 1435 completing (errored=False, cancelled=False)
[2017-06-01 19:20:50,865: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:20:50,872: INFO/job_scheduler] Job 1436 finished 1 steps successfully
[2017-06-01 19:20:50,929: INFO/job_scheduler] Job 1436: complete_job: Updating cache
[2017-06-01 19:20:50,929: INFO/job_scheduler] Job 1436 completing (errored=False, cancelled=False)
[2017-06-01 19:20:51,024: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:20:51,034: INFO/job_scheduler] Job 1437 finished 1 steps successfully
[2017-06-01 19:20:51,070: INFO/job_scheduler] Job 1437: complete_job: Updating cache
[2017-06-01 19:20:51,071: INFO/job_scheduler] Job 1437 completing (errored=False, cancelled=False)
[2017-06-01 19:20:51,074: INFO/job_scheduler] Job 1438 finished 1 steps successfully
[2017-06-01 19:20:51,162: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:20:51,189: INFO/job_scheduler] Job 1438: complete_job: Updating cache
[2017-06-01 19:20:51,189: INFO/job_scheduler] Job 1438 completing (errored=False, cancelled=False)
[2017-06-01 19:20:51,259: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:20:53,217: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:53,217: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:53,223: INFO/command_plan] add_jobs: created Job 1439 (Test for host connectivity)
[2017-06-01 19:20:53,298: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:20:53,311: INFO/job_scheduler] Job 1439: RunJobThread.run
[2017-06-01 19:20:53,367: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:53,367: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:53,401: INFO/command_plan] add_jobs: created Job 1440 (Test for host connectivity)
[2017-06-01 19:20:53,453: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:20:53,486: INFO/job_scheduler] Job 1440: RunJobThread.run
[2017-06-01 19:20:53,590: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:53,595: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:53,631: INFO/command_plan] add_jobs: created Job 1441 (Test for host connectivity)
[2017-06-01 19:20:53,684: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:20:53,734: INFO/job_scheduler] Job 1441: RunJobThread.run
[2017-06-01 19:20:53,775: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:53,777: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:53,792: INFO/command_plan] add_jobs: created Job 1442 (Test for host connectivity)
[2017-06-01 19:20:53,855: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:20:53,884: INFO/job_scheduler] Job 1442: RunJobThread.run
[2017-06-01 19:20:55,767: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:20:55,785: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:20:56,097: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:20:56,196: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:20:57,093: INFO/job_scheduler] Job 1439 finished 1 steps successfully
[2017-06-01 19:20:57,127: INFO/job_scheduler] Job 1440 finished 1 steps successfully
[2017-06-01 19:20:57,139: INFO/job_scheduler] Job 1439: complete_job: Updating cache
[2017-06-01 19:20:57,139: INFO/job_scheduler] Job 1439 completing (errored=False, cancelled=False)
[2017-06-01 19:20:57,195: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:20:57,242: INFO/job_scheduler] Job 1440: complete_job: Updating cache
[2017-06-01 19:20:57,242: INFO/job_scheduler] Job 1440 completing (errored=False, cancelled=False)
[2017-06-01 19:20:57,364: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:20:57,450: INFO/job_scheduler] Job 1441 finished 1 steps successfully
[2017-06-01 19:20:57,620: INFO/job_scheduler] Job 1442 finished 1 steps successfully
[2017-06-01 19:20:57,631: INFO/job_scheduler] Job 1441: complete_job: Updating cache
[2017-06-01 19:20:57,631: INFO/job_scheduler] Job 1441 completing (errored=False, cancelled=False)
[2017-06-01 19:20:57,682: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:20:57,698: INFO/job_scheduler] Job 1442: complete_job: Updating cache
[2017-06-01 19:20:57,700: INFO/job_scheduler] Job 1442 completing (errored=False, cancelled=False)
[2017-06-01 19:20:57,822: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:20:59,528: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:59,529: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:59,559: INFO/command_plan] add_jobs: created Job 1443 (Test for host connectivity)
[2017-06-01 19:20:59,661: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:20:59,694: INFO/job_scheduler] Job 1443: RunJobThread.run
[2017-06-01 19:20:59,732: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:59,734: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:59,743: INFO/command_plan] add_jobs: created Job 1444 (Test for host connectivity)
[2017-06-01 19:20:59,797: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:20:59,861: INFO/job_scheduler] Job 1444: RunJobThread.run
[2017-06-01 19:20:59,912: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:20:59,915: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:20:59,935: INFO/command_plan] add_jobs: created Job 1445 (Test for host connectivity)
[2017-06-01 19:20:59,979: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:20:59,988: INFO/job_scheduler] Job 1445: RunJobThread.run
[2017-06-01 19:21:00,077: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:00,077: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:00,102: INFO/command_plan] add_jobs: created Job 1446 (Test for host connectivity)
[2017-06-01 19:21:00,192: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:21:00,195: INFO/job_scheduler] Job 1446: RunJobThread.run
[2017-06-01 19:21:01,981: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:21:02,128: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:21:02,428: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:21:02,445: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:21:03,283: INFO/job_scheduler] Job 1443 finished 1 steps successfully
[2017-06-01 19:21:03,325: INFO/job_scheduler] Job 1443: complete_job: Updating cache
[2017-06-01 19:21:03,325: INFO/job_scheduler] Job 1443 completing (errored=False, cancelled=False)
[2017-06-01 19:21:03,408: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:21:03,490: INFO/job_scheduler] Job 1444 finished 1 steps successfully
[2017-06-01 19:21:03,544: INFO/job_scheduler] Job 1444: complete_job: Updating cache
[2017-06-01 19:21:03,544: INFO/job_scheduler] Job 1444 completing (errored=False, cancelled=False)
[2017-06-01 19:21:03,710: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:21:03,919: INFO/job_scheduler] Job 1446 finished 1 steps successfully
[2017-06-01 19:21:03,975: INFO/job_scheduler] Job 1446: complete_job: Updating cache
[2017-06-01 19:21:03,975: INFO/job_scheduler] Job 1446 completing (errored=False, cancelled=False)
[2017-06-01 19:21:04,000: INFO/job_scheduler] Job 1445 finished 1 steps successfully
[2017-06-01 19:21:04,036: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:21:04,069: INFO/job_scheduler] Job 1445: complete_job: Updating cache
[2017-06-01 19:21:04,070: INFO/job_scheduler] Job 1445 completing (errored=False, cancelled=False)
[2017-06-01 19:21:04,153: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:21:05,885: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:05,886: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:05,911: INFO/command_plan] add_jobs: created Job 1447 (Test for host connectivity)
[2017-06-01 19:21:05,940: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:21:05,952: INFO/job_scheduler] Job 1447: RunJobThread.run
[2017-06-01 19:21:06,042: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:06,043: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:06,061: INFO/command_plan] add_jobs: created Job 1448 (Test for host connectivity)
[2017-06-01 19:21:06,142: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:21:06,154: INFO/job_scheduler] Job 1448: RunJobThread.run
[2017-06-01 19:21:06,242: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:06,244: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:06,268: INFO/command_plan] add_jobs: created Job 1449 (Test for host connectivity)
[2017-06-01 19:21:06,340: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:21:06,367: INFO/job_scheduler] Job 1449: RunJobThread.run
[2017-06-01 19:21:06,428: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:06,435: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:06,465: INFO/command_plan] add_jobs: created Job 1450 (Test for host connectivity)
[2017-06-01 19:21:06,535: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:21:06,547: INFO/job_scheduler] Job 1450: RunJobThread.run
[2017-06-01 19:21:08,189: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:21:08,396: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:21:08,700: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:21:08,816: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:21:09,538: INFO/job_scheduler] Job 1447 finished 1 steps successfully
[2017-06-01 19:21:09,574: INFO/job_scheduler] Job 1447: complete_job: Updating cache
[2017-06-01 19:21:09,574: INFO/job_scheduler] Job 1447 completing (errored=False, cancelled=False)
[2017-06-01 19:21:09,608: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:21:09,768: INFO/job_scheduler] Job 1448 finished 1 steps successfully
[2017-06-01 19:21:09,790: INFO/job_scheduler] Job 1448: complete_job: Updating cache
[2017-06-01 19:21:09,791: INFO/job_scheduler] Job 1448 completing (errored=False, cancelled=False)
[2017-06-01 19:21:09,891: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:21:10,149: INFO/job_scheduler] Job 1449 finished 1 steps successfully
[2017-06-01 19:21:10,165: INFO/job_scheduler] Job 1450 finished 1 steps successfully
[2017-06-01 19:21:10,207: INFO/job_scheduler] Job 1449: complete_job: Updating cache
[2017-06-01 19:21:10,207: INFO/job_scheduler] Job 1449 completing (errored=False, cancelled=False)
[2017-06-01 19:21:10,266: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:21:10,392: INFO/job_scheduler] Job 1450: complete_job: Updating cache
[2017-06-01 19:21:10,393: INFO/job_scheduler] Job 1450 completing (errored=False, cancelled=False)
[2017-06-01 19:21:10,485: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:21:12,310: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:12,312: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:12,333: INFO/command_plan] add_jobs: created Job 1451 (Test for host connectivity)
[2017-06-01 19:21:12,431: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:12,432: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:12,442: INFO/command_plan] add_jobs: created Job 1452 (Test for host connectivity)
[2017-06-01 19:21:12,453: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:21:12,474: INFO/job_scheduler] Job 1451: RunJobThread.run
[2017-06-01 19:21:12,518: INFO/job_scheduler] Job 1452: RunJobThread.run
[2017-06-01 19:21:12,543: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:21:12,626: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:12,628: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:12,646: INFO/command_plan] add_jobs: created Job 1453 (Test for host connectivity)
[2017-06-01 19:21:12,723: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:21:12,766: INFO/job_scheduler] Job 1453: RunJobThread.run
[2017-06-01 19:21:12,838: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:12,839: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:12,858: INFO/command_plan] add_jobs: created Job 1454 (Test for host connectivity)
[2017-06-01 19:21:12,883: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:21:12,937: INFO/job_scheduler] Job 1454: RunJobThread.run
[2017-06-01 19:21:14,806: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:21:14,934: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:21:14,958: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:21:15,229: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:21:16,104: INFO/job_scheduler] Job 1451 finished 1 steps successfully
[2017-06-01 19:21:16,156: INFO/job_scheduler] Job 1451: complete_job: Updating cache
[2017-06-01 19:21:16,156: INFO/job_scheduler] Job 1451 completing (errored=False, cancelled=False)
[2017-06-01 19:21:16,197: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:21:16,249: INFO/job_scheduler] Job 1453 finished 1 steps successfully
[2017-06-01 19:21:16,267: INFO/job_scheduler] Job 1452 finished 1 steps successfully
[2017-06-01 19:21:16,331: INFO/job_scheduler] Job 1453: complete_job: Updating cache
[2017-06-01 19:21:16,331: INFO/job_scheduler] Job 1453 completing (errored=False, cancelled=False)
[2017-06-01 19:21:16,367: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:21:16,397: INFO/job_scheduler] Job 1452: complete_job: Updating cache
[2017-06-01 19:21:16,398: INFO/job_scheduler] Job 1452 completing (errored=False, cancelled=False)
[2017-06-01 19:21:16,489: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:21:16,615: INFO/job_scheduler] Job 1454 finished 1 steps successfully
[2017-06-01 19:21:16,655: INFO/job_scheduler] Job 1454: complete_job: Updating cache
[2017-06-01 19:21:16,656: INFO/job_scheduler] Job 1454 completing (errored=False, cancelled=False)
[2017-06-01 19:21:16,714: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:21:18,711: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:18,713: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:18,747: INFO/command_plan] add_jobs: created Job 1455 (Test for host connectivity)
[2017-06-01 19:21:18,785: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:21:18,788: INFO/job_scheduler] Job 1455: RunJobThread.run
[2017-06-01 19:21:18,867: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:18,868: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:18,880: INFO/command_plan] add_jobs: created Job 1456 (Test for host connectivity)
[2017-06-01 19:21:18,901: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:21:18,928: INFO/job_scheduler] Job 1456: RunJobThread.run
[2017-06-01 19:21:19,001: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:19,001: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:19,014: INFO/command_plan] add_jobs: created Job 1457 (Test for host connectivity)
[2017-06-01 19:21:19,050: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:21:19,065: INFO/job_scheduler] Job 1457: RunJobThread.run
[2017-06-01 19:21:19,128: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:19,128: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:19,151: INFO/command_plan] add_jobs: created Job 1458 (Test for host connectivity)
[2017-06-01 19:21:19,210: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:21:19,246: INFO/job_scheduler] Job 1458: RunJobThread.run
[2017-06-01 19:21:21,090: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:21:21,223: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:21:21,415: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:21:21,509: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:21:22,403: INFO/job_scheduler] Job 1455 finished 1 steps successfully
[2017-06-01 19:21:22,454: INFO/job_scheduler] Job 1455: complete_job: Updating cache
[2017-06-01 19:21:22,455: INFO/job_scheduler] Job 1455 completing (errored=False, cancelled=False)
[2017-06-01 19:21:22,514: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:21:22,584: INFO/job_scheduler] Job 1456 finished 1 steps successfully
[2017-06-01 19:21:22,655: INFO/job_scheduler] Job 1456: complete_job: Updating cache
[2017-06-01 19:21:22,656: INFO/job_scheduler] Job 1456 completing (errored=False, cancelled=False)
[2017-06-01 19:21:22,733: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:21:22,799: INFO/job_scheduler] Job 1457 finished 1 steps successfully
[2017-06-01 19:21:22,830: INFO/job_scheduler] Job 1457: complete_job: Updating cache
[2017-06-01 19:21:22,831: INFO/job_scheduler] Job 1457 completing (errored=False, cancelled=False)
[2017-06-01 19:21:22,932: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:21:23,075: INFO/job_scheduler] Job 1458 finished 1 steps successfully
[2017-06-01 19:21:23,130: INFO/job_scheduler] Job 1458: complete_job: Updating cache
[2017-06-01 19:21:23,131: INFO/job_scheduler] Job 1458 completing (errored=False, cancelled=False)
[2017-06-01 19:21:23,163: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:21:24,912: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:24,914: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:24,929: INFO/command_plan] add_jobs: created Job 1459 (Test for host connectivity)
[2017-06-01 19:21:25,000: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:21:25,007: INFO/job_scheduler] Job 1459: RunJobThread.run
[2017-06-01 19:21:25,031: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:25,031: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:25,065: INFO/command_plan] add_jobs: created Job 1460 (Test for host connectivity)
[2017-06-01 19:21:25,188: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:25,188: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:25,222: INFO/command_plan] add_jobs: created Job 1461 (Test for host connectivity)
[2017-06-01 19:21:25,230: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:21:25,253: INFO/job_scheduler] Job 1460: RunJobThread.run
[2017-06-01 19:21:25,303: INFO/job_scheduler] Job 1461: RunJobThread.run
[2017-06-01 19:21:25,372: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:25,372: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:25,405: INFO/command_plan] add_jobs: created Job 1462 (Test for host connectivity)
[2017-06-01 19:21:25,531: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:21:25,536: INFO/job_scheduler] Job 1462: RunJobThread.run
[2017-06-01 19:21:25,591: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:21:27,550: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:21:27,609: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:21:27,689: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:21:28,054: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:21:28,933: INFO/job_scheduler] Job 1461 finished 1 steps successfully
[2017-06-01 19:21:28,955: INFO/job_scheduler] Job 1461: complete_job: Updating cache
[2017-06-01 19:21:28,955: INFO/job_scheduler] Job 1461 completing (errored=False, cancelled=False)
[2017-06-01 19:21:28,999: INFO/job_scheduler] Job 1459 finished 1 steps successfully
[2017-06-01 19:21:29,009: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:21:29,013: INFO/job_scheduler] Job 1460 finished 1 steps successfully
[2017-06-01 19:21:29,023: INFO/job_scheduler] Job 1459: complete_job: Updating cache
[2017-06-01 19:21:29,027: INFO/job_scheduler] Job 1459 completing (errored=False, cancelled=False)
[2017-06-01 19:21:29,147: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:21:29,173: INFO/job_scheduler] Job 1460: complete_job: Updating cache
[2017-06-01 19:21:29,173: INFO/job_scheduler] Job 1460 completing (errored=False, cancelled=False)
[2017-06-01 19:21:29,298: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:21:29,601: INFO/job_scheduler] Job 1462 finished 1 steps successfully
[2017-06-01 19:21:29,655: INFO/job_scheduler] Job 1462: complete_job: Updating cache
[2017-06-01 19:21:29,655: INFO/job_scheduler] Job 1462 completing (errored=False, cancelled=False)
[2017-06-01 19:21:29,707: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:21:31,260: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:31,260: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:31,271: INFO/command_plan] add_jobs: created Job 1463 (Test for host connectivity)
[2017-06-01 19:21:31,341: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:21:31,363: INFO/job_scheduler] Job 1463: RunJobThread.run
[2017-06-01 19:21:31,397: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:31,403: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:31,422: INFO/command_plan] add_jobs: created Job 1464 (Test for host connectivity)
[2017-06-01 19:21:31,589: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:31,590: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:31,605: INFO/command_plan] add_jobs: created Job 1465 (Test for host connectivity)
[2017-06-01 19:21:31,692: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:21:31,715: INFO/job_scheduler] Job 1464: RunJobThread.run
[2017-06-01 19:21:31,717: INFO/job_scheduler] Job 1465: RunJobThread.run
[2017-06-01 19:21:31,733: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:21:31,796: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:31,797: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:31,833: INFO/command_plan] add_jobs: created Job 1466 (Test for host connectivity)
[2017-06-01 19:21:31,945: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:21:31,971: INFO/job_scheduler] Job 1466: RunJobThread.run
[2017-06-01 19:21:33,829: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:21:33,998: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:21:34,144: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:21:34,231: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:21:35,173: INFO/job_scheduler] Job 1463 finished 1 steps successfully
[2017-06-01 19:21:35,226: INFO/job_scheduler] Job 1463: complete_job: Updating cache
[2017-06-01 19:21:35,227: INFO/job_scheduler] Job 1463 completing (errored=False, cancelled=False)
[2017-06-01 19:21:35,272: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:21:35,347: INFO/job_scheduler] Job 1464 finished 1 steps successfully
[2017-06-01 19:21:35,404: INFO/job_scheduler] Job 1464: complete_job: Updating cache
[2017-06-01 19:21:35,406: INFO/job_scheduler] Job 1464 completing (errored=False, cancelled=False)
[2017-06-01 19:21:35,458: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:21:35,555: INFO/job_scheduler] Job 1465 finished 1 steps successfully
[2017-06-01 19:21:35,605: INFO/job_scheduler] Job 1465: complete_job: Updating cache
[2017-06-01 19:21:35,605: INFO/job_scheduler] Job 1465 completing (errored=False, cancelled=False)
[2017-06-01 19:21:35,638: INFO/job_scheduler] Job 1466 finished 1 steps successfully
[2017-06-01 19:21:35,663: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:21:35,679: INFO/job_scheduler] Job 1466: complete_job: Updating cache
[2017-06-01 19:21:35,679: INFO/job_scheduler] Job 1466 completing (errored=False, cancelled=False)
[2017-06-01 19:21:35,738: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:21:38,224: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:38,224: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:38,238: INFO/command_plan] add_jobs: created Job 1467 (Test for host connectivity)
[2017-06-01 19:21:38,364: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:38,367: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:38,387: INFO/command_plan] add_jobs: created Job 1468 (Test for host connectivity)
[2017-06-01 19:21:38,404: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:21:38,424: INFO/job_scheduler] Job 1467: RunJobThread.run
[2017-06-01 19:21:38,445: INFO/job_scheduler] Job 1468: RunJobThread.run
[2017-06-01 19:21:38,454: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:21:38,571: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:38,571: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:38,590: INFO/command_plan] add_jobs: created Job 1469 (Test for host connectivity)
[2017-06-01 19:21:38,659: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:21:38,693: INFO/job_scheduler] Job 1469: RunJobThread.run
[2017-06-01 19:21:38,761: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:38,767: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:38,784: INFO/command_plan] add_jobs: created Job 1470 (Test for host connectivity)
[2017-06-01 19:21:38,896: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:21:38,905: INFO/job_scheduler] Job 1470: RunJobThread.run
[2017-06-01 19:21:40,957: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:21:41,000: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:21:41,396: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:21:41,437: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:21:42,325: INFO/job_scheduler] Job 1468 finished 1 steps successfully
[2017-06-01 19:21:42,344: INFO/job_scheduler] Job 1468: complete_job: Updating cache
[2017-06-01 19:21:42,344: INFO/job_scheduler] Job 1468 completing (errored=False, cancelled=False)
[2017-06-01 19:21:42,359: INFO/job_scheduler] Job 1467 finished 1 steps successfully
[2017-06-01 19:21:42,378: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:21:42,394: INFO/job_scheduler] Job 1467: complete_job: Updating cache
[2017-06-01 19:21:42,397: INFO/job_scheduler] Job 1467 completing (errored=False, cancelled=False)
[2017-06-01 19:21:42,488: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:21:42,883: INFO/job_scheduler] Job 1470 finished 1 steps successfully
[2017-06-01 19:21:42,902: INFO/job_scheduler] Job 1470: complete_job: Updating cache
[2017-06-01 19:21:42,902: INFO/job_scheduler] Job 1470 completing (errored=False, cancelled=False)
[2017-06-01 19:21:42,939: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:21:42,943: INFO/job_scheduler] Job 1469 finished 1 steps successfully
[2017-06-01 19:21:43,018: INFO/job_scheduler] Job 1469: complete_job: Updating cache
[2017-06-01 19:21:43,018: INFO/job_scheduler] Job 1469 completing (errored=False, cancelled=False)
[2017-06-01 19:21:43,106: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:21:44,546: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:44,546: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:44,561: INFO/command_plan] add_jobs: created Job 1471 (Test for host connectivity)
[2017-06-01 19:21:44,577: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:21:44,580: INFO/job_scheduler] Job 1471: RunJobThread.run
[2017-06-01 19:21:44,670: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:44,670: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:44,711: INFO/command_plan] add_jobs: created Job 1472 (Test for host connectivity)
[2017-06-01 19:21:44,756: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:21:44,798: INFO/job_scheduler] Job 1472: RunJobThread.run
[2017-06-01 19:21:44,917: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:44,919: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:44,935: INFO/command_plan] add_jobs: created Job 1473 (Test for host connectivity)
[2017-06-01 19:21:45,092: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:45,094: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:45,111: INFO/command_plan] add_jobs: created Job 1474 (Test for host connectivity)
[2017-06-01 19:21:45,155: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:21:45,205: INFO/job_scheduler] Job 1473: RunJobThread.run
[2017-06-01 19:21:45,244: INFO/job_scheduler] Job 1474: RunJobThread.run
[2017-06-01 19:21:45,264: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:21:46,996: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:21:47,188: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:21:47,500: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:21:47,662: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:21:48,364: INFO/job_scheduler] Job 1471 finished 1 steps successfully
[2017-06-01 19:21:48,384: INFO/job_scheduler] Job 1471: complete_job: Updating cache
[2017-06-01 19:21:48,385: INFO/job_scheduler] Job 1471 completing (errored=False, cancelled=False)
[2017-06-01 19:21:48,421: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:21:48,530: INFO/job_scheduler] Job 1472 finished 1 steps successfully
[2017-06-01 19:21:48,596: INFO/job_scheduler] Job 1472: complete_job: Updating cache
[2017-06-01 19:21:48,596: INFO/job_scheduler] Job 1472 completing (errored=False, cancelled=False)
[2017-06-01 19:21:48,661: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:21:49,023: INFO/job_scheduler] Job 1474 finished 1 steps successfully
[2017-06-01 19:21:49,043: INFO/job_scheduler] Job 1474: complete_job: Updating cache
[2017-06-01 19:21:49,043: INFO/job_scheduler] Job 1474 completing (errored=False, cancelled=False)
[2017-06-01 19:21:49,066: INFO/job_scheduler] Job 1473 finished 1 steps successfully
[2017-06-01 19:21:49,078: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:21:49,102: INFO/job_scheduler] Job 1473: complete_job: Updating cache
[2017-06-01 19:21:49,102: INFO/job_scheduler] Job 1473 completing (errored=False, cancelled=False)
[2017-06-01 19:21:49,178: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:21:50,750: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:50,762: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:50,777: INFO/command_plan] add_jobs: created Job 1475 (Test for host connectivity)
[2017-06-01 19:21:50,853: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:21:50,862: INFO/job_scheduler] Job 1475: RunJobThread.run
[2017-06-01 19:21:50,906: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:50,908: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:50,927: INFO/command_plan] add_jobs: created Job 1476 (Test for host connectivity)
[2017-06-01 19:21:50,966: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:21:50,972: INFO/job_scheduler] Job 1476: RunJobThread.run
[2017-06-01 19:21:51,124: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:51,126: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:51,143: INFO/command_plan] add_jobs: created Job 1477 (Test for host connectivity)
[2017-06-01 19:21:51,187: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:21:51,199: INFO/job_scheduler] Job 1477: RunJobThread.run
[2017-06-01 19:21:51,237: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:51,238: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:51,264: INFO/command_plan] add_jobs: created Job 1478 (Test for host connectivity)
[2017-06-01 19:21:51,284: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:21:51,303: INFO/job_scheduler] Job 1478: RunJobThread.run
[2017-06-01 19:21:53,154: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:21:53,343: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:21:53,598: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:21:53,665: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:21:54,474: INFO/job_scheduler] Job 1475 finished 1 steps successfully
[2017-06-01 19:21:54,508: INFO/job_scheduler] Job 1475: complete_job: Updating cache
[2017-06-01 19:21:54,508: INFO/job_scheduler] Job 1475 completing (errored=False, cancelled=False)
[2017-06-01 19:21:54,542: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:21:54,663: INFO/job_scheduler] Job 1476 finished 1 steps successfully
[2017-06-01 19:21:54,745: INFO/job_scheduler] Job 1476: complete_job: Updating cache
[2017-06-01 19:21:54,746: INFO/job_scheduler] Job 1476 completing (errored=False, cancelled=False)
[2017-06-01 19:21:54,885: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:21:55,061: INFO/job_scheduler] Job 1477 finished 1 steps successfully
[2017-06-01 19:21:55,141: INFO/job_scheduler] Job 1477: complete_job: Updating cache
[2017-06-01 19:21:55,142: INFO/job_scheduler] Job 1477 completing (errored=False, cancelled=False)
[2017-06-01 19:21:55,189: INFO/job_scheduler] Job 1478 finished 1 steps successfully
[2017-06-01 19:21:55,217: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:21:55,233: INFO/job_scheduler] Job 1478: complete_job: Updating cache
[2017-06-01 19:21:55,234: INFO/job_scheduler] Job 1478 completing (errored=False, cancelled=False)
[2017-06-01 19:21:55,308: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:21:56,905: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:56,905: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:56,925: INFO/command_plan] add_jobs: created Job 1479 (Test for host connectivity)
[2017-06-01 19:21:56,975: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:21:57,017: INFO/job_scheduler] Job 1479: RunJobThread.run
[2017-06-01 19:21:57,087: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:57,090: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:57,108: INFO/command_plan] add_jobs: created Job 1480 (Test for host connectivity)
[2017-06-01 19:21:57,140: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:21:57,189: INFO/job_scheduler] Job 1480: RunJobThread.run
[2017-06-01 19:21:57,293: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:57,295: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:57,310: INFO/command_plan] add_jobs: created Job 1481 (Test for host connectivity)
[2017-06-01 19:21:57,342: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:21:57,351: INFO/job_scheduler] Job 1481: RunJobThread.run
[2017-06-01 19:21:57,490: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:21:57,491: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:21:57,500: INFO/command_plan] add_jobs: created Job 1482 (Test for host connectivity)
[2017-06-01 19:21:57,533: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:21:57,545: INFO/job_scheduler] Job 1482: RunJobThread.run
[2017-06-01 19:21:59,466: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:21:59,596: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:21:59,677: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:21:59,844: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:22:00,786: INFO/job_scheduler] Job 1479 finished 1 steps successfully
[2017-06-01 19:22:00,815: INFO/job_scheduler] Job 1479: complete_job: Updating cache
[2017-06-01 19:22:00,815: INFO/job_scheduler] Job 1479 completing (errored=False, cancelled=False)
[2017-06-01 19:22:00,857: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:22:00,948: INFO/job_scheduler] Job 1480 finished 1 steps successfully
[2017-06-01 19:22:00,980: INFO/job_scheduler] Job 1480: complete_job: Updating cache
[2017-06-01 19:22:00,982: INFO/job_scheduler] Job 1480 completing (errored=False, cancelled=False)
[2017-06-01 19:22:00,986: INFO/job_scheduler] Job 1481 finished 1 steps successfully
[2017-06-01 19:22:01,032: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:22:01,041: INFO/job_scheduler] Job 1481: complete_job: Updating cache
[2017-06-01 19:22:01,041: INFO/job_scheduler] Job 1481 completing (errored=False, cancelled=False)
[2017-06-01 19:22:01,126: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:22:01,281: INFO/job_scheduler] Job 1482 finished 1 steps successfully
[2017-06-01 19:22:01,350: INFO/job_scheduler] Job 1482: complete_job: Updating cache
[2017-06-01 19:22:01,350: INFO/job_scheduler] Job 1482 completing (errored=False, cancelled=False)
[2017-06-01 19:22:01,388: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:22:03,173: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:03,176: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:03,190: INFO/command_plan] add_jobs: created Job 1483 (Test for host connectivity)
[2017-06-01 19:22:03,309: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:22:03,325: INFO/job_scheduler] Job 1483: RunJobThread.run
[2017-06-01 19:22:03,364: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:03,364: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:03,375: INFO/command_plan] add_jobs: created Job 1484 (Test for host connectivity)
[2017-06-01 19:22:03,430: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:22:03,485: INFO/job_scheduler] Job 1484: RunJobThread.run
[2017-06-01 19:22:03,556: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:03,558: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:03,664: INFO/command_plan] add_jobs: created Job 1485 (Test for host connectivity)
[2017-06-01 19:22:03,766: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:03,767: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:03,799: INFO/command_plan] add_jobs: created Job 1486 (Test for host connectivity)
[2017-06-01 19:22:03,850: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:22:03,865: INFO/job_scheduler] Job 1485: RunJobThread.run
[2017-06-01 19:22:03,899: INFO/job_scheduler] Job 1486: RunJobThread.run
[2017-06-01 19:22:03,915: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:22:05,768: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:22:05,819: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:22:06,251: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:22:06,333: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:22:07,078: INFO/job_scheduler] Job 1483 finished 1 steps successfully
[2017-06-01 19:22:07,122: INFO/job_scheduler] Job 1483: complete_job: Updating cache
[2017-06-01 19:22:07,123: INFO/job_scheduler] Job 1483 completing (errored=False, cancelled=False)
[2017-06-01 19:22:07,183: INFO/job_scheduler] Job 1484 finished 1 steps successfully
[2017-06-01 19:22:07,190: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:22:07,231: INFO/job_scheduler] Job 1484: complete_job: Updating cache
[2017-06-01 19:22:07,238: INFO/job_scheduler] Job 1484 completing (errored=False, cancelled=False)
[2017-06-01 19:22:07,339: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:22:07,693: INFO/job_scheduler] Job 1486 finished 1 steps successfully
[2017-06-01 19:22:07,706: INFO/job_scheduler] Job 1486: complete_job: Updating cache
[2017-06-01 19:22:07,706: INFO/job_scheduler] Job 1486 completing (errored=False, cancelled=False)
[2017-06-01 19:22:07,738: INFO/job_scheduler] Job 1485 finished 1 steps successfully
[2017-06-01 19:22:07,740: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:22:07,765: INFO/job_scheduler] Job 1485: complete_job: Updating cache
[2017-06-01 19:22:07,767: INFO/job_scheduler] Job 1485 completing (errored=False, cancelled=False)
[2017-06-01 19:22:07,891: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:22:09,497: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:09,498: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:09,514: INFO/command_plan] add_jobs: created Job 1487 (Test for host connectivity)
[2017-06-01 19:22:09,561: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:22:09,591: INFO/job_scheduler] Job 1487: RunJobThread.run
[2017-06-01 19:22:09,649: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:09,649: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:09,685: INFO/command_plan] add_jobs: created Job 1488 (Test for host connectivity)
[2017-06-01 19:22:09,766: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:22:09,804: INFO/job_scheduler] Job 1488: RunJobThread.run
[2017-06-01 19:22:09,871: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:09,872: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:09,889: INFO/command_plan] add_jobs: created Job 1489 (Test for host connectivity)
[2017-06-01 19:22:09,930: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:22:09,942: INFO/job_scheduler] Job 1489: RunJobThread.run
[2017-06-01 19:22:10,023: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:10,033: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:10,069: INFO/command_plan] add_jobs: created Job 1490 (Test for host connectivity)
[2017-06-01 19:22:10,125: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:22:10,202: INFO/job_scheduler] Job 1490: RunJobThread.run
[2017-06-01 19:22:12,012: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:22:12,126: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:22:12,323: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:22:12,496: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:22:13,341: INFO/job_scheduler] Job 1487 finished 1 steps successfully
[2017-06-01 19:22:13,398: INFO/job_scheduler] Job 1487: complete_job: Updating cache
[2017-06-01 19:22:13,398: INFO/job_scheduler] Job 1487 completing (errored=False, cancelled=False)
[2017-06-01 19:22:13,473: INFO/job_scheduler] Job 1488 finished 1 steps successfully
[2017-06-01 19:22:13,473: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:22:13,500: INFO/job_scheduler] Job 1488: complete_job: Updating cache
[2017-06-01 19:22:13,503: INFO/job_scheduler] Job 1488 completing (errored=False, cancelled=False)
[2017-06-01 19:22:13,614: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:22:13,704: INFO/job_scheduler] Job 1489 finished 1 steps successfully
[2017-06-01 19:22:13,804: INFO/job_scheduler] Job 1489: complete_job: Updating cache
[2017-06-01 19:22:13,805: INFO/job_scheduler] Job 1489 completing (errored=False, cancelled=False)
[2017-06-01 19:22:13,856: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:22:13,908: INFO/job_scheduler] Job 1490 finished 1 steps successfully
[2017-06-01 19:22:13,935: INFO/job_scheduler] Job 1490: complete_job: Updating cache
[2017-06-01 19:22:13,935: INFO/job_scheduler] Job 1490 completing (errored=False, cancelled=False)
[2017-06-01 19:22:13,998: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:22:15,753: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:15,755: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:15,762: INFO/command_plan] add_jobs: created Job 1491 (Test for host connectivity)
[2017-06-01 19:22:15,830: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:22:15,844: INFO/job_scheduler] Job 1491: RunJobThread.run
[2017-06-01 19:22:15,889: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:15,893: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:15,915: INFO/command_plan] add_jobs: created Job 1492 (Test for host connectivity)
[2017-06-01 19:22:15,963: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:22:16,023: INFO/job_scheduler] Job 1492: RunJobThread.run
[2017-06-01 19:22:16,093: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:16,094: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:16,113: INFO/command_plan] add_jobs: created Job 1493 (Test for host connectivity)
[2017-06-01 19:22:16,131: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:22:16,150: INFO/job_scheduler] Job 1493: RunJobThread.run
[2017-06-01 19:22:16,304: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:16,305: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:16,321: INFO/command_plan] add_jobs: created Job 1494 (Test for host connectivity)
[2017-06-01 19:22:16,411: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:22:16,447: INFO/job_scheduler] Job 1494: RunJobThread.run
[2017-06-01 19:22:18,213: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:22:18,376: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:22:18,576: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:22:18,742: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:22:19,527: INFO/job_scheduler] Job 1491 finished 1 steps successfully
[2017-06-01 19:22:19,595: INFO/job_scheduler] Job 1491: complete_job: Updating cache
[2017-06-01 19:22:19,595: INFO/job_scheduler] Job 1491 completing (errored=False, cancelled=False)
[2017-06-01 19:22:19,684: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:22:19,719: INFO/job_scheduler] Job 1492 finished 1 steps successfully
[2017-06-01 19:22:19,858: INFO/job_scheduler] Job 1492: complete_job: Updating cache
[2017-06-01 19:22:19,858: INFO/job_scheduler] Job 1492 completing (errored=False, cancelled=False)
[2017-06-01 19:22:19,965: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:22:20,000: INFO/job_scheduler] Job 1493 finished 1 steps successfully
[2017-06-01 19:22:20,030: INFO/job_scheduler] Job 1493: complete_job: Updating cache
[2017-06-01 19:22:20,030: INFO/job_scheduler] Job 1493 completing (errored=False, cancelled=False)
[2017-06-01 19:22:20,087: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:22:20,165: INFO/job_scheduler] Job 1494 finished 1 steps successfully
[2017-06-01 19:22:20,195: INFO/job_scheduler] Job 1494: complete_job: Updating cache
[2017-06-01 19:22:20,199: INFO/job_scheduler] Job 1494 completing (errored=False, cancelled=False)
[2017-06-01 19:22:20,262: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:22:21,964: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:21,964: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:21,997: INFO/command_plan] add_jobs: created Job 1495 (Test for host connectivity)
[2017-06-01 19:22:22,111: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:22,111: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:22,137: INFO/command_plan] add_jobs: created Job 1496 (Test for host connectivity)
[2017-06-01 19:22:22,192: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:22:22,207: INFO/job_scheduler] Job 1496: RunJobThread.run
[2017-06-01 19:22:22,214: INFO/job_scheduler] Job 1495: RunJobThread.run
[2017-06-01 19:22:22,223: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:22:22,294: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:22,295: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:22,318: INFO/command_plan] add_jobs: created Job 1497 (Test for host connectivity)
[2017-06-01 19:22:22,423: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:22:22,470: INFO/job_scheduler] Job 1497: RunJobThread.run
[2017-06-01 19:22:22,509: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:22,512: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:22,528: INFO/command_plan] add_jobs: created Job 1498 (Test for host connectivity)
[2017-06-01 19:22:22,566: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:22:22,597: INFO/job_scheduler] Job 1498: RunJobThread.run
[2017-06-01 19:22:24,565: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:22:24,577: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:22:24,809: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:22:24,906: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:22:26,024: INFO/job_scheduler] Job 1495 finished 1 steps successfully
[2017-06-01 19:22:26,056: INFO/job_scheduler] Job 1496 finished 1 steps successfully
[2017-06-01 19:22:26,069: INFO/job_scheduler] Job 1495: complete_job: Updating cache
[2017-06-01 19:22:26,069: INFO/job_scheduler] Job 1495 completing (errored=False, cancelled=False)
[2017-06-01 19:22:26,137: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:22:26,192: INFO/job_scheduler] Job 1497 finished 1 steps successfully
[2017-06-01 19:22:26,194: INFO/job_scheduler] Job 1496: complete_job: Updating cache
[2017-06-01 19:22:26,194: INFO/job_scheduler] Job 1496 completing (errored=False, cancelled=False)
[2017-06-01 19:22:26,265: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:22:26,266: INFO/job_scheduler] Job 1498 finished 1 steps successfully
[2017-06-01 19:22:26,334: INFO/job_scheduler] Job 1497: complete_job: Updating cache
[2017-06-01 19:22:26,334: INFO/job_scheduler] Job 1497 completing (errored=False, cancelled=False)
[2017-06-01 19:22:26,404: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:22:26,432: INFO/job_scheduler] Job 1498: complete_job: Updating cache
[2017-06-01 19:22:26,434: INFO/job_scheduler] Job 1498 completing (errored=False, cancelled=False)
[2017-06-01 19:22:26,518: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:22:28,162: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:28,165: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:28,176: INFO/command_plan] add_jobs: created Job 1499 (Test for host connectivity)
[2017-06-01 19:22:28,241: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:28,241: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:28,257: INFO/command_plan] add_jobs: created Job 1500 (Test for host connectivity)
[2017-06-01 19:22:28,295: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:22:28,317: INFO/job_scheduler] Job 1499: RunJobThread.run
[2017-06-01 19:22:28,326: INFO/job_scheduler] Job 1500: RunJobThread.run
[2017-06-01 19:22:28,335: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:22:28,404: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:28,404: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:28,428: INFO/command_plan] add_jobs: created Job 1501 (Test for host connectivity)
[2017-06-01 19:22:28,530: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:22:28,552: INFO/job_scheduler] Job 1501: RunJobThread.run
[2017-06-01 19:22:28,634: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:28,634: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:28,670: INFO/command_plan] add_jobs: created Job 1502 (Test for host connectivity)
[2017-06-01 19:22:28,718: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:22:28,723: INFO/job_scheduler] Job 1502: RunJobThread.run
[2017-06-01 19:22:30,807: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:22:30,811: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:22:30,984: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:22:31,060: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:22:32,166: INFO/job_scheduler] Job 1500 finished 1 steps successfully
[2017-06-01 19:22:32,224: INFO/job_scheduler] Job 1500: complete_job: Updating cache
[2017-06-01 19:22:32,225: INFO/job_scheduler] Job 1500 completing (errored=False, cancelled=False)
[2017-06-01 19:22:32,258: INFO/job_scheduler] Job 1499 finished 1 steps successfully
[2017-06-01 19:22:32,260: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:22:32,284: INFO/job_scheduler] Job 1499: complete_job: Updating cache
[2017-06-01 19:22:32,285: INFO/job_scheduler] Job 1499 completing (errored=False, cancelled=False)
[2017-06-01 19:22:32,336: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:22:32,444: INFO/job_scheduler] Job 1502 finished 1 steps successfully
[2017-06-01 19:22:32,447: INFO/job_scheduler] Job 1501 finished 1 steps successfully
[2017-06-01 19:22:32,477: INFO/job_scheduler] Job 1502: complete_job: Updating cache
[2017-06-01 19:22:32,477: INFO/job_scheduler] Job 1502 completing (errored=False, cancelled=False)
[2017-06-01 19:22:32,534: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:22:32,549: INFO/job_scheduler] Job 1501: complete_job: Updating cache
[2017-06-01 19:22:32,550: INFO/job_scheduler] Job 1501 completing (errored=False, cancelled=False)
[2017-06-01 19:22:32,621: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:22:34,296: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:34,298: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:34,317: INFO/command_plan] add_jobs: created Job 1503 (Test for host connectivity)
[2017-06-01 19:22:34,371: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:22:34,387: INFO/job_scheduler] Job 1503: RunJobThread.run
[2017-06-01 19:22:34,434: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:34,434: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:34,452: INFO/command_plan] add_jobs: created Job 1504 (Test for host connectivity)
[2017-06-01 19:22:34,494: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:22:34,500: INFO/job_scheduler] Job 1504: RunJobThread.run
[2017-06-01 19:22:34,601: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:34,605: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:34,616: INFO/command_plan] add_jobs: created Job 1505 (Test for host connectivity)
[2017-06-01 19:22:34,638: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:22:34,658: INFO/job_scheduler] Job 1505: RunJobThread.run
[2017-06-01 19:22:34,768: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:34,768: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:34,790: INFO/command_plan] add_jobs: created Job 1506 (Test for host connectivity)
[2017-06-01 19:22:34,877: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:22:34,925: INFO/job_scheduler] Job 1506: RunJobThread.run
[2017-06-01 19:22:36,652: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:22:36,967: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:22:37,057: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:22:37,305: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:22:38,000: INFO/job_scheduler] Job 1503 finished 1 steps successfully
[2017-06-01 19:22:38,049: INFO/job_scheduler] Job 1503: complete_job: Updating cache
[2017-06-01 19:22:38,049: INFO/job_scheduler] Job 1503 completing (errored=False, cancelled=False)
[2017-06-01 19:22:38,109: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:22:38,272: INFO/job_scheduler] Job 1504 finished 1 steps successfully
[2017-06-01 19:22:38,322: INFO/job_scheduler] Job 1504: complete_job: Updating cache
[2017-06-01 19:22:38,323: INFO/job_scheduler] Job 1504 completing (errored=False, cancelled=False)
[2017-06-01 19:22:38,387: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:22:38,463: INFO/job_scheduler] Job 1505 finished 1 steps successfully
[2017-06-01 19:22:38,534: INFO/job_scheduler] Job 1505: complete_job: Updating cache
[2017-06-01 19:22:38,534: INFO/job_scheduler] Job 1505 completing (errored=False, cancelled=False)
[2017-06-01 19:22:38,574: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:22:38,734: INFO/job_scheduler] Job 1506 finished 1 steps successfully
[2017-06-01 19:22:38,765: INFO/job_scheduler] Job 1506: complete_job: Updating cache
[2017-06-01 19:22:38,765: INFO/job_scheduler] Job 1506 completing (errored=False, cancelled=False)
[2017-06-01 19:22:38,791: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:22:40,473: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:40,473: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:40,508: INFO/command_plan] add_jobs: created Job 1507 (Test for host connectivity)
[2017-06-01 19:22:40,561: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:22:40,591: INFO/job_scheduler] Job 1507: RunJobThread.run
[2017-06-01 19:22:40,633: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:40,633: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:40,649: INFO/command_plan] add_jobs: created Job 1508 (Test for host connectivity)
[2017-06-01 19:22:40,701: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:22:40,752: INFO/job_scheduler] Job 1508: RunJobThread.run
[2017-06-01 19:22:40,793: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:40,794: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:40,815: INFO/command_plan] add_jobs: created Job 1509 (Test for host connectivity)
[2017-06-01 19:22:40,878: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:22:40,947: INFO/job_scheduler] Job 1509: RunJobThread.run
[2017-06-01 19:22:40,998: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:40,999: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:41,020: INFO/command_plan] add_jobs: created Job 1510 (Test for host connectivity)
[2017-06-01 19:22:41,067: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:22:41,079: INFO/job_scheduler] Job 1510: RunJobThread.run
[2017-06-01 19:22:43,007: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:22:43,114: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:22:43,225: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:22:43,340: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:22:44,428: INFO/job_scheduler] Job 1507 finished 1 steps successfully
[2017-06-01 19:22:44,454: INFO/job_scheduler] Job 1508 finished 1 steps successfully
[2017-06-01 19:22:44,490: INFO/job_scheduler] Job 1507: complete_job: Updating cache
[2017-06-01 19:22:44,490: INFO/job_scheduler] Job 1507 completing (errored=False, cancelled=False)
[2017-06-01 19:22:44,542: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:22:44,558: INFO/job_scheduler] Job 1508: complete_job: Updating cache
[2017-06-01 19:22:44,558: INFO/job_scheduler] Job 1508 completing (errored=False, cancelled=False)
[2017-06-01 19:22:44,622: INFO/job_scheduler] Job 1509 finished 1 steps successfully
[2017-06-01 19:22:44,634: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:22:44,740: INFO/job_scheduler] Job 1509: complete_job: Updating cache
[2017-06-01 19:22:44,742: INFO/job_scheduler] Job 1509 completing (errored=False, cancelled=False)
[2017-06-01 19:22:44,894: INFO/job_scheduler] Job 1510 finished 1 steps successfully
[2017-06-01 19:22:44,948: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:22:44,966: INFO/job_scheduler] Job 1510: complete_job: Updating cache
[2017-06-01 19:22:44,966: INFO/job_scheduler] Job 1510 completing (errored=False, cancelled=False)
[2017-06-01 19:22:45,017: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:22:46,686: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:46,686: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:46,698: INFO/command_plan] add_jobs: created Job 1511 (Test for host connectivity)
[2017-06-01 19:22:46,790: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:22:46,804: INFO/job_scheduler] Job 1511: RunJobThread.run
[2017-06-01 19:22:46,841: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:46,841: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:46,870: INFO/command_plan] add_jobs: created Job 1512 (Test for host connectivity)
[2017-06-01 19:22:46,892: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:22:46,908: INFO/job_scheduler] Job 1512: RunJobThread.run
[2017-06-01 19:22:47,010: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:47,013: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:47,033: INFO/command_plan] add_jobs: created Job 1513 (Test for host connectivity)
[2017-06-01 19:22:47,121: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:22:47,132: INFO/job_scheduler] Job 1513: RunJobThread.run
[2017-06-01 19:22:47,187: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:47,188: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:47,214: INFO/command_plan] add_jobs: created Job 1514 (Test for host connectivity)
[2017-06-01 19:22:47,232: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:22:47,310: INFO/job_scheduler] Job 1514: RunJobThread.run
[2017-06-01 19:22:49,157: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:22:49,303: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:22:49,385: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:22:49,651: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:22:50,485: INFO/job_scheduler] Job 1511 finished 1 steps successfully
[2017-06-01 19:22:50,522: INFO/job_scheduler] Job 1511: complete_job: Updating cache
[2017-06-01 19:22:50,523: INFO/job_scheduler] Job 1511 completing (errored=False, cancelled=False)
[2017-06-01 19:22:50,573: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:22:50,699: INFO/job_scheduler] Job 1513 finished 1 steps successfully
[2017-06-01 19:22:50,731: INFO/job_scheduler] Job 1513: complete_job: Updating cache
[2017-06-01 19:22:50,732: INFO/job_scheduler] Job 1513 completing (errored=False, cancelled=False)
[2017-06-01 19:22:50,733: INFO/job_scheduler] Job 1512 finished 1 steps successfully
[2017-06-01 19:22:50,857: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:22:50,904: INFO/job_scheduler] Job 1512: complete_job: Updating cache
[2017-06-01 19:22:50,904: INFO/job_scheduler] Job 1512 completing (errored=False, cancelled=False)
[2017-06-01 19:22:51,000: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:22:51,076: INFO/job_scheduler] Job 1514 finished 1 steps successfully
[2017-06-01 19:22:51,111: INFO/job_scheduler] Job 1514: complete_job: Updating cache
[2017-06-01 19:22:51,111: INFO/job_scheduler] Job 1514 completing (errored=False, cancelled=False)
[2017-06-01 19:22:51,175: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:22:52,822: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:52,822: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:52,843: INFO/command_plan] add_jobs: created Job 1515 (Test for host connectivity)
[2017-06-01 19:22:52,928: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:22:52,938: INFO/job_scheduler] Job 1515: RunJobThread.run
[2017-06-01 19:22:52,966: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:52,967: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:52,980: INFO/command_plan] add_jobs: created Job 1516 (Test for host connectivity)
[2017-06-01 19:22:53,027: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:22:53,056: INFO/job_scheduler] Job 1516: RunJobThread.run
[2017-06-01 19:22:53,140: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:53,141: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:53,157: INFO/command_plan] add_jobs: created Job 1517 (Test for host connectivity)
[2017-06-01 19:22:53,257: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:22:53,259: INFO/job_scheduler] Job 1517: RunJobThread.run
[2017-06-01 19:22:53,377: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:53,387: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:53,407: INFO/command_plan] add_jobs: created Job 1518 (Test for host connectivity)
[2017-06-01 19:22:53,478: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:22:53,508: INFO/job_scheduler] Job 1518: RunJobThread.run
[2017-06-01 19:22:55,173: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:22:55,483: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:22:55,599: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:22:55,781: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:22:56,522: INFO/job_scheduler] Job 1515 finished 1 steps successfully
[2017-06-01 19:22:56,547: INFO/job_scheduler] Job 1515: complete_job: Updating cache
[2017-06-01 19:22:56,547: INFO/job_scheduler] Job 1515 completing (errored=False, cancelled=False)
[2017-06-01 19:22:56,598: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:22:56,899: INFO/job_scheduler] Job 1516 finished 1 steps successfully
[2017-06-01 19:22:56,947: INFO/job_scheduler] Job 1516: complete_job: Updating cache
[2017-06-01 19:22:56,947: INFO/job_scheduler] Job 1516 completing (errored=False, cancelled=False)
[2017-06-01 19:22:56,982: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:22:57,091: INFO/job_scheduler] Job 1517 finished 1 steps successfully
[2017-06-01 19:22:57,138: INFO/job_scheduler] Job 1517: complete_job: Updating cache
[2017-06-01 19:22:57,139: INFO/job_scheduler] Job 1517 completing (errored=False, cancelled=False)
[2017-06-01 19:22:57,192: INFO/job_scheduler] Job 1518 finished 1 steps successfully
[2017-06-01 19:22:57,213: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:22:57,224: INFO/job_scheduler] Job 1518: complete_job: Updating cache
[2017-06-01 19:22:57,225: INFO/job_scheduler] Job 1518 completing (errored=False, cancelled=False)
[2017-06-01 19:22:57,349: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:22:59,026: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:59,026: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:59,038: INFO/command_plan] add_jobs: created Job 1519 (Test for host connectivity)
[2017-06-01 19:22:59,069: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:22:59,072: INFO/job_scheduler] Job 1519: RunJobThread.run
[2017-06-01 19:22:59,133: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:59,133: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:59,149: INFO/command_plan] add_jobs: created Job 1520 (Test for host connectivity)
[2017-06-01 19:22:59,230: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:22:59,254: INFO/job_scheduler] Job 1520: RunJobThread.run
[2017-06-01 19:22:59,317: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:59,317: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:59,366: INFO/command_plan] add_jobs: created Job 1521 (Test for host connectivity)
[2017-06-01 19:22:59,415: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:22:59,440: INFO/job_scheduler] Job 1521: RunJobThread.run
[2017-06-01 19:22:59,537: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:22:59,538: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:22:59,557: INFO/command_plan] add_jobs: created Job 1522 (Test for host connectivity)
[2017-06-01 19:22:59,582: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:22:59,623: INFO/job_scheduler] Job 1522: RunJobThread.run
[2017-06-01 19:23:01,635: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:23:01,639: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:23:01,861: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:23:02,024: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:23:03,007: INFO/job_scheduler] Job 1520 finished 1 steps successfully
[2017-06-01 19:23:03,041: INFO/job_scheduler] Job 1519 finished 1 steps successfully
[2017-06-01 19:23:03,056: INFO/job_scheduler] Job 1520: complete_job: Updating cache
[2017-06-01 19:23:03,057: INFO/job_scheduler] Job 1520 completing (errored=False, cancelled=False)
[2017-06-01 19:23:03,107: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:23:03,155: INFO/job_scheduler] Job 1519: complete_job: Updating cache
[2017-06-01 19:23:03,155: INFO/job_scheduler] Job 1519 completing (errored=False, cancelled=False)
[2017-06-01 19:23:03,211: INFO/job_scheduler] Job 1521 finished 1 steps successfully
[2017-06-01 19:23:03,246: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:23:03,371: INFO/job_scheduler] Job 1521: complete_job: Updating cache
[2017-06-01 19:23:03,372: INFO/job_scheduler] Job 1521 completing (errored=False, cancelled=False)
[2017-06-01 19:23:03,415: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:23:03,444: INFO/job_scheduler] Job 1522 finished 1 steps successfully
[2017-06-01 19:23:03,480: INFO/job_scheduler] Job 1522: complete_job: Updating cache
[2017-06-01 19:23:03,481: INFO/job_scheduler] Job 1522 completing (errored=False, cancelled=False)
[2017-06-01 19:23:03,547: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:23:05,147: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:05,149: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:05,163: INFO/command_plan] add_jobs: created Job 1523 (Test for host connectivity)
[2017-06-01 19:23:05,238: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:23:05,284: INFO/job_scheduler] Job 1523: RunJobThread.run
[2017-06-01 19:23:05,336: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:05,346: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:05,365: INFO/command_plan] add_jobs: created Job 1524 (Test for host connectivity)
[2017-06-01 19:23:05,428: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:23:05,469: INFO/job_scheduler] Job 1524: RunJobThread.run
[2017-06-01 19:23:05,526: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:05,527: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:05,540: INFO/command_plan] add_jobs: created Job 1525 (Test for host connectivity)
[2017-06-01 19:23:05,553: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:23:05,572: INFO/job_scheduler] Job 1525: RunJobThread.run
[2017-06-01 19:23:05,664: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:05,665: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:05,691: INFO/command_plan] add_jobs: created Job 1526 (Test for host connectivity)
[2017-06-01 19:23:05,762: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:23:05,820: INFO/job_scheduler] Job 1526: RunJobThread.run
[2017-06-01 19:23:07,754: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:23:07,845: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:23:07,939: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:23:08,204: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:23:09,069: INFO/job_scheduler] Job 1523 finished 1 steps successfully
[2017-06-01 19:23:09,096: INFO/job_scheduler] Job 1523: complete_job: Updating cache
[2017-06-01 19:23:09,096: INFO/job_scheduler] Job 1523 completing (errored=False, cancelled=False)
[2017-06-01 19:23:09,129: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:23:09,194: INFO/job_scheduler] Job 1524 finished 1 steps successfully
[2017-06-01 19:23:09,235: INFO/job_scheduler] Job 1524: complete_job: Updating cache
[2017-06-01 19:23:09,236: INFO/job_scheduler] Job 1524 completing (errored=False, cancelled=False)
[2017-06-01 19:23:09,310: INFO/job_scheduler] Job 1525 finished 1 steps successfully
[2017-06-01 19:23:09,371: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:23:09,416: INFO/job_scheduler] Job 1525: complete_job: Updating cache
[2017-06-01 19:23:09,416: INFO/job_scheduler] Job 1525 completing (errored=False, cancelled=False)
[2017-06-01 19:23:09,579: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:23:09,720: INFO/job_scheduler] Job 1526 finished 1 steps successfully
[2017-06-01 19:23:09,779: INFO/job_scheduler] Job 1526: complete_job: Updating cache
[2017-06-01 19:23:09,779: INFO/job_scheduler] Job 1526 completing (errored=False, cancelled=False)
[2017-06-01 19:23:09,812: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:23:10,957: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:10,958: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:10,973: INFO/command_plan] add_jobs: created Job 1527 (Test for host connectivity)
[2017-06-01 19:23:11,024: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:11,024: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:11,038: INFO/command_plan] add_jobs: created Job 1528 (Test for host connectivity)
[2017-06-01 19:23:11,072: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:23:11,078: INFO/job_scheduler] Job 1528: RunJobThread.run
[2017-06-01 19:23:11,101: INFO/job_scheduler] Job 1527: RunJobThread.run
[2017-06-01 19:23:11,215: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:11,216: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:11,245: INFO/command_plan] add_jobs: created Job 1529 (Test for host connectivity)
[2017-06-01 19:23:11,292: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:23:11,357: INFO/job_scheduler] Job 1529: RunJobThread.run
[2017-06-01 19:23:11,400: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:23:11,480: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:11,481: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:11,501: INFO/command_plan] add_jobs: created Job 1530 (Test for host connectivity)
[2017-06-01 19:23:11,522: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:23:11,564: INFO/job_scheduler] Job 1530: RunJobThread.run
[2017-06-01 19:23:13,521: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:23:13,554: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:23:13,695: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:23:13,942: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:23:14,885: INFO/job_scheduler] Job 1527 finished 1 steps successfully
[2017-06-01 19:23:14,908: INFO/job_scheduler] Job 1528 finished 1 steps successfully
[2017-06-01 19:23:14,916: INFO/job_scheduler] Job 1527: complete_job: Updating cache
[2017-06-01 19:23:14,916: INFO/job_scheduler] Job 1527 completing (errored=False, cancelled=False)
[2017-06-01 19:23:14,984: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:23:15,006: INFO/job_scheduler] Job 1528: complete_job: Updating cache
[2017-06-01 19:23:15,006: INFO/job_scheduler] Job 1528 completing (errored=False, cancelled=False)
[2017-06-01 19:23:15,060: INFO/job_scheduler] Job 1529 finished 1 steps successfully
[2017-06-01 19:23:15,095: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:23:15,112: INFO/job_scheduler] Job 1529: complete_job: Updating cache
[2017-06-01 19:23:15,112: INFO/job_scheduler] Job 1529 completing (errored=False, cancelled=False)
[2017-06-01 19:23:15,173: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:23:15,352: INFO/job_scheduler] Job 1530 finished 1 steps successfully
[2017-06-01 19:23:15,372: INFO/job_scheduler] Job 1530: complete_job: Updating cache
[2017-06-01 19:23:15,372: INFO/job_scheduler] Job 1530 completing (errored=False, cancelled=False)
[2017-06-01 19:23:15,403: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:23:17,201: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:17,202: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:17,212: INFO/command_plan] add_jobs: created Job 1531 (Test for host connectivity)
[2017-06-01 19:23:17,229: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:23:17,251: INFO/job_scheduler] Job 1531: RunJobThread.run
[2017-06-01 19:23:17,344: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:17,345: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:17,387: INFO/command_plan] add_jobs: created Job 1532 (Test for host connectivity)
[2017-06-01 19:23:17,451: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:23:17,477: INFO/job_scheduler] Job 1532: RunJobThread.run
[2017-06-01 19:23:17,513: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:17,520: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:17,532: INFO/command_plan] add_jobs: created Job 1533 (Test for host connectivity)
[2017-06-01 19:23:17,597: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:23:17,630: INFO/job_scheduler] Job 1533: RunJobThread.run
[2017-06-01 19:23:17,709: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:17,710: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:17,742: INFO/command_plan] add_jobs: created Job 1534 (Test for host connectivity)
[2017-06-01 19:23:17,781: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:23:17,840: INFO/job_scheduler] Job 1534: RunJobThread.run
[2017-06-01 19:23:19,728: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:23:19,893: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:23:20,058: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:23:20,099: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:23:21,027: INFO/job_scheduler] Job 1531 finished 1 steps successfully
[2017-06-01 19:23:21,064: INFO/job_scheduler] Job 1531: complete_job: Updating cache
[2017-06-01 19:23:21,065: INFO/job_scheduler] Job 1531 completing (errored=False, cancelled=False)
[2017-06-01 19:23:21,116: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:23:21,238: INFO/job_scheduler] Job 1532 finished 1 steps successfully
[2017-06-01 19:23:21,381: INFO/job_scheduler] Job 1532: complete_job: Updating cache
[2017-06-01 19:23:21,382: INFO/job_scheduler] Job 1532 completing (errored=False, cancelled=False)
[2017-06-01 19:23:21,434: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:23:21,518: INFO/job_scheduler] Job 1534 finished 1 steps successfully
[2017-06-01 19:23:21,534: INFO/job_scheduler] Job 1533 finished 1 steps successfully
[2017-06-01 19:23:21,581: INFO/job_scheduler] Job 1534: complete_job: Updating cache
[2017-06-01 19:23:21,581: INFO/job_scheduler] Job 1534 completing (errored=False, cancelled=False)
[2017-06-01 19:23:21,628: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:23:21,653: INFO/job_scheduler] Job 1533: complete_job: Updating cache
[2017-06-01 19:23:21,653: INFO/job_scheduler] Job 1533 completing (errored=False, cancelled=False)
[2017-06-01 19:23:21,754: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:23:23,457: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:23,458: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:23,481: INFO/command_plan] add_jobs: created Job 1535 (Test for host connectivity)
[2017-06-01 19:23:23,528: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:23,528: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:23,556: INFO/command_plan] add_jobs: created Job 1536 (Test for host connectivity)
[2017-06-01 19:23:23,686: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:23:23,694: INFO/job_scheduler] Job 1536: RunJobThread.run
[2017-06-01 19:23:23,695: INFO/job_scheduler] Job 1535: RunJobThread.run
[2017-06-01 19:23:23,717: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:23:23,760: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:23,760: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:23,806: INFO/command_plan] add_jobs: created Job 1537 (Test for host connectivity)
[2017-06-01 19:23:23,927: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:23:23,929: INFO/job_scheduler] Job 1537: RunJobThread.run
[2017-06-01 19:23:23,957: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:23,957: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:23,974: INFO/command_plan] add_jobs: created Job 1538 (Test for host connectivity)
[2017-06-01 19:23:24,054: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:23:24,071: INFO/job_scheduler] Job 1538: RunJobThread.run
[2017-06-01 19:23:25,892: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:23:26,169: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:23:26,254: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:23:26,527: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:23:27,317: INFO/job_scheduler] Job 1535 finished 1 steps successfully
[2017-06-01 19:23:27,347: INFO/job_scheduler] Job 1535: complete_job: Updating cache
[2017-06-01 19:23:27,347: INFO/job_scheduler] Job 1535 completing (errored=False, cancelled=False)
[2017-06-01 19:23:27,406: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:23:27,578: INFO/job_scheduler] Job 1537 finished 1 steps successfully
[2017-06-01 19:23:27,652: INFO/job_scheduler] Job 1537: complete_job: Updating cache
[2017-06-01 19:23:27,653: INFO/job_scheduler] Job 1537 completing (errored=False, cancelled=False)
[2017-06-01 19:23:27,730: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:23:27,851: INFO/job_scheduler] Job 1536 finished 1 steps successfully
[2017-06-01 19:23:27,913: INFO/job_scheduler] Job 1536: complete_job: Updating cache
[2017-06-01 19:23:27,913: INFO/job_scheduler] Job 1536 completing (errored=False, cancelled=False)
[2017-06-01 19:23:27,955: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:23:28,034: INFO/job_scheduler] Job 1538 finished 1 steps successfully
[2017-06-01 19:23:28,047: INFO/job_scheduler] Job 1538: complete_job: Updating cache
[2017-06-01 19:23:28,049: INFO/job_scheduler] Job 1538 completing (errored=False, cancelled=False)
[2017-06-01 19:23:28,107: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:23:29,710: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:29,712: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:29,721: INFO/command_plan] add_jobs: created Job 1539 (Test for host connectivity)
[2017-06-01 19:23:29,747: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:23:29,805: INFO/job_scheduler] Job 1539: RunJobThread.run
[2017-06-01 19:23:29,837: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:29,839: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:29,864: INFO/command_plan] add_jobs: created Job 1540 (Test for host connectivity)
[2017-06-01 19:23:29,880: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:23:29,930: INFO/job_scheduler] Job 1540: RunJobThread.run
[2017-06-01 19:23:30,016: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:30,019: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:30,055: INFO/command_plan] add_jobs: created Job 1541 (Test for host connectivity)
[2017-06-01 19:23:30,186: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:30,186: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:30,203: INFO/command_plan] add_jobs: created Job 1542 (Test for host connectivity)
[2017-06-01 19:23:30,257: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:23:30,264: INFO/job_scheduler] Job 1541: RunJobThread.run
[2017-06-01 19:23:30,316: INFO/job_scheduler] Job 1542: RunJobThread.run
[2017-06-01 19:23:30,336: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:23:32,136: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:23:32,335: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:23:32,639: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:23:32,698: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:23:33,460: INFO/job_scheduler] Job 1539 finished 1 steps successfully
[2017-06-01 19:23:33,504: INFO/job_scheduler] Job 1539: complete_job: Updating cache
[2017-06-01 19:23:33,504: INFO/job_scheduler] Job 1539 completing (errored=False, cancelled=False)
[2017-06-01 19:23:33,539: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:23:33,682: INFO/job_scheduler] Job 1540 finished 1 steps successfully
[2017-06-01 19:23:33,742: INFO/job_scheduler] Job 1540: complete_job: Updating cache
[2017-06-01 19:23:33,743: INFO/job_scheduler] Job 1540 completing (errored=False, cancelled=False)
[2017-06-01 19:23:33,820: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:23:34,180: INFO/job_scheduler] Job 1542 finished 1 steps successfully
[2017-06-01 19:23:34,198: INFO/job_scheduler] Job 1541 finished 1 steps successfully
[2017-06-01 19:23:34,203: INFO/job_scheduler] Job 1542: complete_job: Updating cache
[2017-06-01 19:23:34,204: INFO/job_scheduler] Job 1542 completing (errored=False, cancelled=False)
[2017-06-01 19:23:34,230: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:23:34,269: INFO/job_scheduler] Job 1541: complete_job: Updating cache
[2017-06-01 19:23:34,269: INFO/job_scheduler] Job 1541 completing (errored=False, cancelled=False)
[2017-06-01 19:23:34,346: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:23:35,973: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:35,975: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:35,987: INFO/command_plan] add_jobs: created Job 1543 (Test for host connectivity)
[2017-06-01 19:23:36,102: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:36,102: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:36,138: INFO/command_plan] add_jobs: created Job 1544 (Test for host connectivity)
[2017-06-01 19:23:36,147: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:23:36,149: INFO/job_scheduler] Job 1544: RunJobThread.run
[2017-06-01 19:23:36,214: INFO/job_scheduler] Job 1543: RunJobThread.run
[2017-06-01 19:23:36,278: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:36,279: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:36,286: INFO/command_plan] add_jobs: created Job 1545 (Test for host connectivity)
[2017-06-01 19:23:36,312: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:23:36,349: INFO/job_scheduler] Job 1545: RunJobThread.run
[2017-06-01 19:23:36,460: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:36,460: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:36,496: INFO/command_plan] add_jobs: created Job 1546 (Test for host connectivity)
[2017-06-01 19:23:36,582: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:23:36,588: INFO/job_scheduler] Job 1546: RunJobThread.run
[2017-06-01 19:23:36,641: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:23:38,519: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:23:38,656: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:23:38,756: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:23:38,876: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:23:39,836: INFO/job_scheduler] Job 1543 finished 1 steps successfully
[2017-06-01 19:23:39,902: INFO/job_scheduler] Job 1543: complete_job: Updating cache
[2017-06-01 19:23:39,902: INFO/job_scheduler] Job 1543 completing (errored=False, cancelled=False)
[2017-06-01 19:23:39,973: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:23:39,980: INFO/job_scheduler] Job 1544 finished 1 steps successfully
[2017-06-01 19:23:40,007: INFO/job_scheduler] Job 1544: complete_job: Updating cache
[2017-06-01 19:23:40,012: INFO/job_scheduler] Job 1544 completing (errored=False, cancelled=False)
[2017-06-01 19:23:40,144: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:23:40,174: INFO/job_scheduler] Job 1545 finished 1 steps successfully
[2017-06-01 19:23:40,231: INFO/job_scheduler] Job 1545: complete_job: Updating cache
[2017-06-01 19:23:40,231: INFO/job_scheduler] Job 1545 completing (errored=False, cancelled=False)
[2017-06-01 19:23:40,266: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:23:40,267: INFO/job_scheduler] Job 1546 finished 1 steps successfully
[2017-06-01 19:23:40,278: INFO/job_scheduler] Job 1546: complete_job: Updating cache
[2017-06-01 19:23:40,278: INFO/job_scheduler] Job 1546 completing (errored=False, cancelled=False)
[2017-06-01 19:23:40,353: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:23:42,388: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:42,388: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:42,426: INFO/command_plan] add_jobs: created Job 1547 (Test for host connectivity)
[2017-06-01 19:23:42,479: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:23:42,494: INFO/job_scheduler] Job 1547: RunJobThread.run
[2017-06-01 19:23:42,549: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:42,549: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:42,560: INFO/command_plan] add_jobs: created Job 1548 (Test for host connectivity)
[2017-06-01 19:23:42,600: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:23:42,630: INFO/job_scheduler] Job 1548: RunJobThread.run
[2017-06-01 19:23:42,690: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:42,691: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:42,712: INFO/command_plan] add_jobs: created Job 1549 (Test for host connectivity)
[2017-06-01 19:23:42,761: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:23:42,775: INFO/job_scheduler] Job 1549: RunJobThread.run
[2017-06-01 19:23:42,888: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:42,889: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:42,919: INFO/command_plan] add_jobs: created Job 1550 (Test for host connectivity)
[2017-06-01 19:23:42,940: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:23:42,977: INFO/job_scheduler] Job 1550: RunJobThread.run
[2017-06-01 19:23:44,812: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:23:44,945: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:23:45,129: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:23:45,264: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:23:46,141: INFO/job_scheduler] Job 1547 finished 1 steps successfully
[2017-06-01 19:23:46,184: INFO/job_scheduler] Job 1547: complete_job: Updating cache
[2017-06-01 19:23:46,184: INFO/job_scheduler] Job 1547 completing (errored=False, cancelled=False)
[2017-06-01 19:23:46,235: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:23:46,299: INFO/job_scheduler] Job 1548 finished 1 steps successfully
[2017-06-01 19:23:46,354: INFO/job_scheduler] Job 1548: complete_job: Updating cache
[2017-06-01 19:23:46,357: INFO/job_scheduler] Job 1548 completing (errored=False, cancelled=False)
[2017-06-01 19:23:46,419: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:23:46,451: INFO/job_scheduler] Job 1549 finished 1 steps successfully
[2017-06-01 19:23:46,496: INFO/job_scheduler] Job 1549: complete_job: Updating cache
[2017-06-01 19:23:46,496: INFO/job_scheduler] Job 1549 completing (errored=False, cancelled=False)
[2017-06-01 19:23:46,584: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:23:46,630: INFO/job_scheduler] Job 1550 finished 1 steps successfully
[2017-06-01 19:23:46,703: INFO/job_scheduler] Job 1550: complete_job: Updating cache
[2017-06-01 19:23:46,704: INFO/job_scheduler] Job 1550 completing (errored=False, cancelled=False)
[2017-06-01 19:23:46,770: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:23:48,644: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:48,653: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:48,668: INFO/command_plan] add_jobs: created Job 1551 (Test for host connectivity)
[2017-06-01 19:23:48,697: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:23:48,726: INFO/job_scheduler] Job 1551: RunJobThread.run
[2017-06-01 19:23:48,797: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:48,798: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:48,850: INFO/command_plan] add_jobs: created Job 1552 (Test for host connectivity)
[2017-06-01 19:23:48,907: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:23:48,927: INFO/job_scheduler] Job 1552: RunJobThread.run
[2017-06-01 19:23:48,990: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:48,992: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:49,020: INFO/command_plan] add_jobs: created Job 1553 (Test for host connectivity)
[2017-06-01 19:23:49,103: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:23:49,136: INFO/job_scheduler] Job 1553: RunJobThread.run
[2017-06-01 19:23:49,189: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:49,190: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:49,228: INFO/command_plan] add_jobs: created Job 1554 (Test for host connectivity)
[2017-06-01 19:23:49,293: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:23:49,325: INFO/job_scheduler] Job 1554: RunJobThread.run
[2017-06-01 19:23:51,027: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:23:51,279: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:23:51,421: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:23:51,607: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:23:52,344: INFO/job_scheduler] Job 1551 finished 1 steps successfully
[2017-06-01 19:23:52,383: INFO/job_scheduler] Job 1551: complete_job: Updating cache
[2017-06-01 19:23:52,383: INFO/job_scheduler] Job 1551 completing (errored=False, cancelled=False)
[2017-06-01 19:23:52,425: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:23:52,594: INFO/job_scheduler] Job 1552 finished 1 steps successfully
[2017-06-01 19:23:52,668: INFO/job_scheduler] Job 1552: complete_job: Updating cache
[2017-06-01 19:23:52,668: INFO/job_scheduler] Job 1552 completing (errored=False, cancelled=False)
[2017-06-01 19:23:52,760: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:23:52,953: INFO/job_scheduler] Job 1553 finished 1 steps successfully
[2017-06-01 19:23:53,008: INFO/job_scheduler] Job 1553: complete_job: Updating cache
[2017-06-01 19:23:53,008: INFO/job_scheduler] Job 1553 completing (errored=False, cancelled=False)
[2017-06-01 19:23:53,050: INFO/job_scheduler] Job 1554 finished 1 steps successfully
[2017-06-01 19:23:53,067: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:23:53,093: INFO/job_scheduler] Job 1554: complete_job: Updating cache
[2017-06-01 19:23:53,094: INFO/job_scheduler] Job 1554 completing (errored=False, cancelled=False)
[2017-06-01 19:23:53,178: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:23:55,025: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:55,027: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:55,050: INFO/command_plan] add_jobs: created Job 1555 (Test for host connectivity)
[2017-06-01 19:23:55,101: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:23:55,116: INFO/job_scheduler] Job 1555: RunJobThread.run
[2017-06-01 19:23:55,194: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:55,200: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:55,308: INFO/command_plan] add_jobs: created Job 1556 (Test for host connectivity)
[2017-06-01 19:23:55,349: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:23:55,357: INFO/job_scheduler] Job 1556: RunJobThread.run
[2017-06-01 19:23:55,431: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:55,433: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:55,467: INFO/command_plan] add_jobs: created Job 1557 (Test for host connectivity)
[2017-06-01 19:23:55,572: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:23:55,573: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:23:55,592: INFO/command_plan] add_jobs: created Job 1558 (Test for host connectivity)
[2017-06-01 19:23:55,645: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:23:55,683: INFO/job_scheduler] Job 1557: RunJobThread.run
[2017-06-01 19:23:55,723: INFO/job_scheduler] Job 1558: RunJobThread.run
[2017-06-01 19:23:55,733: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:23:57,378: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:23:57,644: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:23:58,007: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:23:58,064: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:23:58,715: INFO/job_scheduler] Job 1555 finished 1 steps successfully
[2017-06-01 19:23:58,742: INFO/job_scheduler] Job 1555: complete_job: Updating cache
[2017-06-01 19:23:58,742: INFO/job_scheduler] Job 1555 completing (errored=False, cancelled=False)
[2017-06-01 19:23:58,808: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:23:59,014: INFO/job_scheduler] Job 1556 finished 1 steps successfully
[2017-06-01 19:23:59,032: INFO/job_scheduler] Job 1556: complete_job: Updating cache
[2017-06-01 19:23:59,032: INFO/job_scheduler] Job 1556 completing (errored=False, cancelled=False)
[2017-06-01 19:23:59,209: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:23:59,383: INFO/job_scheduler] Job 1558 finished 1 steps successfully
[2017-06-01 19:23:59,432: INFO/job_scheduler] Job 1558: complete_job: Updating cache
[2017-06-01 19:23:59,432: INFO/job_scheduler] Job 1558 completing (errored=False, cancelled=False)
[2017-06-01 19:23:59,484: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:23:59,629: INFO/job_scheduler] Job 1557 finished 1 steps successfully
[2017-06-01 19:23:59,703: INFO/job_scheduler] Job 1557: complete_job: Updating cache
[2017-06-01 19:23:59,705: INFO/job_scheduler] Job 1557 completing (errored=False, cancelled=False)
[2017-06-01 19:23:59,750: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:24:01,343: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:01,345: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:01,360: INFO/command_plan] add_jobs: created Job 1559 (Test for host connectivity)
[2017-06-01 19:24:01,420: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:24:01,435: INFO/job_scheduler] Job 1559: RunJobThread.run
[2017-06-01 19:24:01,509: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:01,509: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:01,533: INFO/command_plan] add_jobs: created Job 1560 (Test for host connectivity)
[2017-06-01 19:24:01,590: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:24:01,627: INFO/job_scheduler] Job 1560: RunJobThread.run
[2017-06-01 19:24:01,659: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:01,660: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:01,673: INFO/command_plan] add_jobs: created Job 1561 (Test for host connectivity)
[2017-06-01 19:24:01,751: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:24:01,791: INFO/job_scheduler] Job 1561: RunJobThread.run
[2017-06-01 19:24:01,835: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:01,836: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:01,863: INFO/command_plan] add_jobs: created Job 1562 (Test for host connectivity)
[2017-06-01 19:24:01,966: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:24:01,971: INFO/job_scheduler] Job 1562: RunJobThread.run
[2017-06-01 19:24:03,771: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:24:03,971: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:24:04,086: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:24:04,200: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:24:05,098: INFO/job_scheduler] Job 1559 finished 1 steps successfully
[2017-06-01 19:24:05,155: INFO/job_scheduler] Job 1559: complete_job: Updating cache
[2017-06-01 19:24:05,156: INFO/job_scheduler] Job 1559 completing (errored=False, cancelled=False)
[2017-06-01 19:24:05,227: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:24:05,304: INFO/job_scheduler] Job 1560 finished 1 steps successfully
[2017-06-01 19:24:05,417: INFO/job_scheduler] Job 1560: complete_job: Updating cache
[2017-06-01 19:24:05,418: INFO/job_scheduler] Job 1560 completing (errored=False, cancelled=False)
[2017-06-01 19:24:05,465: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:24:05,520: INFO/job_scheduler] Job 1561 finished 1 steps successfully
[2017-06-01 19:24:05,581: INFO/job_scheduler] Job 1561: complete_job: Updating cache
[2017-06-01 19:24:05,581: INFO/job_scheduler] Job 1561 completing (errored=False, cancelled=False)
[2017-06-01 19:24:05,605: INFO/job_scheduler] Job 1562 finished 1 steps successfully
[2017-06-01 19:24:05,641: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:24:05,656: INFO/job_scheduler] Job 1562: complete_job: Updating cache
[2017-06-01 19:24:05,656: INFO/job_scheduler] Job 1562 completing (errored=False, cancelled=False)
[2017-06-01 19:24:05,735: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:24:07,607: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:07,608: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:07,648: INFO/command_plan] add_jobs: created Job 1563 (Test for host connectivity)
[2017-06-01 19:24:07,732: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:24:07,738: INFO/job_scheduler] Job 1563: RunJobThread.run
[2017-06-01 19:24:07,796: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:07,796: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:07,822: INFO/command_plan] add_jobs: created Job 1564 (Test for host connectivity)
[2017-06-01 19:24:07,944: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:24:07,950: INFO/job_scheduler] Job 1564: RunJobThread.run
[2017-06-01 19:24:08,004: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:08,005: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:08,155: INFO/command_plan] add_jobs: created Job 1565 (Test for host connectivity)
[2017-06-01 19:24:08,182: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:24:08,205: INFO/job_scheduler] Job 1565: RunJobThread.run
[2017-06-01 19:24:08,383: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:08,393: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:08,427: INFO/command_plan] add_jobs: created Job 1566 (Test for host connectivity)
[2017-06-01 19:24:08,516: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:24:08,539: INFO/job_scheduler] Job 1566: RunJobThread.run
[2017-06-01 19:24:10,002: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:24:10,233: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:24:10,445: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:24:10,706: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:24:11,327: INFO/job_scheduler] Job 1563 finished 1 steps successfully
[2017-06-01 19:24:11,346: INFO/job_scheduler] Job 1563: complete_job: Updating cache
[2017-06-01 19:24:11,346: INFO/job_scheduler] Job 1563 completing (errored=False, cancelled=False)
[2017-06-01 19:24:11,380: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:24:11,630: INFO/job_scheduler] Job 1564 finished 1 steps successfully
[2017-06-01 19:24:11,704: INFO/job_scheduler] Job 1564: complete_job: Updating cache
[2017-06-01 19:24:11,704: INFO/job_scheduler] Job 1564 completing (errored=False, cancelled=False)
[2017-06-01 19:24:11,753: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:24:11,944: INFO/job_scheduler] Job 1565 finished 1 steps successfully
[2017-06-01 19:24:11,996: INFO/job_scheduler] Job 1565: complete_job: Updating cache
[2017-06-01 19:24:11,996: INFO/job_scheduler] Job 1565 completing (errored=False, cancelled=False)
[2017-06-01 19:24:12,029: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:24:12,065: INFO/job_scheduler] Job 1566 finished 1 steps successfully
[2017-06-01 19:24:12,102: INFO/job_scheduler] Job 1566: complete_job: Updating cache
[2017-06-01 19:24:12,102: INFO/job_scheduler] Job 1566 completing (errored=False, cancelled=False)
[2017-06-01 19:24:12,155: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:24:12,940: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:12,940: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:13,021: INFO/command_plan] add_jobs: created Job 1567 (Test for host connectivity)
[2017-06-01 19:24:13,062: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:13,062: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:13,110: INFO/command_plan] add_jobs: created Job 1568 (Test for host connectivity)
[2017-06-01 19:24:13,129: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:24:13,132: INFO/job_scheduler] Job 1568: RunJobThread.run
[2017-06-01 19:24:13,133: INFO/job_scheduler] Job 1567: RunJobThread.run
[2017-06-01 19:24:13,156: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:24:13,267: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:13,267: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:13,321: INFO/command_plan] add_jobs: created Job 1569 (Test for host connectivity)
[2017-06-01 19:24:13,373: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:24:13,410: INFO/job_scheduler] Job 1569: RunJobThread.run
[2017-06-01 19:24:13,446: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:13,446: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:13,471: INFO/command_plan] add_jobs: created Job 1570 (Test for host connectivity)
[2017-06-01 19:24:13,549: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:24:13,552: INFO/job_scheduler] Job 1570: RunJobThread.run
[2017-06-01 19:24:15,338: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:24:15,445: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:24:15,661: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:24:15,856: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:24:16,679: INFO/job_scheduler] Job 1568 finished 1 steps successfully
[2017-06-01 19:24:16,725: INFO/job_scheduler] Job 1568: complete_job: Updating cache
[2017-06-01 19:24:16,726: INFO/job_scheduler] Job 1568 completing (errored=False, cancelled=False)
[2017-06-01 19:24:16,795: INFO/job_scheduler] Job 1567 finished 1 steps successfully
[2017-06-01 19:24:16,799: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:24:16,818: INFO/job_scheduler] Job 1567: complete_job: Updating cache
[2017-06-01 19:24:16,818: INFO/job_scheduler] Job 1567 completing (errored=False, cancelled=False)
[2017-06-01 19:24:16,955: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:24:17,115: INFO/job_scheduler] Job 1569 finished 1 steps successfully
[2017-06-01 19:24:17,166: INFO/job_scheduler] Job 1569: complete_job: Updating cache
[2017-06-01 19:24:17,166: INFO/job_scheduler] Job 1569 completing (errored=False, cancelled=False)
[2017-06-01 19:24:17,191: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:24:17,304: INFO/job_scheduler] Job 1570 finished 1 steps successfully
[2017-06-01 19:24:17,369: INFO/job_scheduler] Job 1570: complete_job: Updating cache
[2017-06-01 19:24:17,373: INFO/job_scheduler] Job 1570 completing (errored=False, cancelled=False)
[2017-06-01 19:24:17,434: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:24:19,226: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:19,228: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:19,241: INFO/command_plan] add_jobs: created Job 1571 (Test for host connectivity)
[2017-06-01 19:24:19,254: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:24:19,282: INFO/job_scheduler] Job 1571: RunJobThread.run
[2017-06-01 19:24:19,325: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:19,325: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:19,346: INFO/command_plan] add_jobs: created Job 1572 (Test for host connectivity)
[2017-06-01 19:24:19,457: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:24:19,465: INFO/job_scheduler] Job 1572: RunJobThread.run
[2017-06-01 19:24:19,584: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:19,585: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:19,608: INFO/command_plan] add_jobs: created Job 1573 (Test for host connectivity)
[2017-06-01 19:24:19,658: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:24:19,681: INFO/job_scheduler] Job 1573: RunJobThread.run
[2017-06-01 19:24:19,744: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:19,745: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:19,774: INFO/command_plan] add_jobs: created Job 1574 (Test for host connectivity)
[2017-06-01 19:24:19,857: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:24:19,891: INFO/job_scheduler] Job 1574: RunJobThread.run
[2017-06-01 19:24:21,754: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:24:21,770: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:24:22,021: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:24:22,139: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:24:23,054: INFO/job_scheduler] Job 1571 finished 1 steps successfully
[2017-06-01 19:24:23,107: INFO/job_scheduler] Job 1571: complete_job: Updating cache
[2017-06-01 19:24:23,107: INFO/job_scheduler] Job 1571 completing (errored=False, cancelled=False)
[2017-06-01 19:24:23,144: INFO/job_scheduler] Job 1572 finished 1 steps successfully
[2017-06-01 19:24:23,209: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:24:23,232: INFO/job_scheduler] Job 1572: complete_job: Updating cache
[2017-06-01 19:24:23,234: INFO/job_scheduler] Job 1572 completing (errored=False, cancelled=False)
[2017-06-01 19:24:23,319: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:24:23,414: INFO/job_scheduler] Job 1573 finished 1 steps successfully
[2017-06-01 19:24:23,474: INFO/job_scheduler] Job 1573: complete_job: Updating cache
[2017-06-01 19:24:23,474: INFO/job_scheduler] Job 1573 completing (errored=False, cancelled=False)
[2017-06-01 19:24:23,497: INFO/job_scheduler] Job 1574 finished 1 steps successfully
[2017-06-01 19:24:23,507: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:24:23,569: INFO/job_scheduler] Job 1574: complete_job: Updating cache
[2017-06-01 19:24:23,570: INFO/job_scheduler] Job 1574 completing (errored=False, cancelled=False)
[2017-06-01 19:24:23,623: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:24:25,511: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:25,513: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:25,524: INFO/command_plan] add_jobs: created Job 1575 (Test for host connectivity)
[2017-06-01 19:24:25,603: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:24:25,636: INFO/job_scheduler] Job 1575: RunJobThread.run
[2017-06-01 19:24:25,691: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:25,692: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:25,709: INFO/command_plan] add_jobs: created Job 1576 (Test for host connectivity)
[2017-06-01 19:24:25,766: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:24:25,823: INFO/job_scheduler] Job 1576: RunJobThread.run
[2017-06-01 19:24:25,900: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:25,900: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:25,932: INFO/command_plan] add_jobs: created Job 1577 (Test for host connectivity)
[2017-06-01 19:24:25,996: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:24:26,031: INFO/job_scheduler] Job 1577: RunJobThread.run
[2017-06-01 19:24:26,106: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:26,108: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:26,202: INFO/command_plan] add_jobs: created Job 1578 (Test for host connectivity)
[2017-06-01 19:24:26,269: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:24:26,280: INFO/job_scheduler] Job 1578: RunJobThread.run
[2017-06-01 19:24:27,938: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:24:28,184: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:24:28,323: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:24:28,440: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:24:29,291: INFO/job_scheduler] Job 1575 finished 1 steps successfully
[2017-06-01 19:24:29,338: INFO/job_scheduler] Job 1575: complete_job: Updating cache
[2017-06-01 19:24:29,338: INFO/job_scheduler] Job 1575 completing (errored=False, cancelled=False)
[2017-06-01 19:24:29,381: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:24:29,501: INFO/job_scheduler] Job 1576 finished 1 steps successfully
[2017-06-01 19:24:29,547: INFO/job_scheduler] Job 1576: complete_job: Updating cache
[2017-06-01 19:24:29,548: INFO/job_scheduler] Job 1576 completing (errored=False, cancelled=False)
[2017-06-01 19:24:29,635: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:24:29,768: INFO/job_scheduler] Job 1577 finished 1 steps successfully
[2017-06-01 19:24:29,781: INFO/job_scheduler] Job 1577: complete_job: Updating cache
[2017-06-01 19:24:29,783: INFO/job_scheduler] Job 1577 completing (errored=False, cancelled=False)
[2017-06-01 19:24:29,841: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:24:29,883: INFO/job_scheduler] Job 1578 finished 1 steps successfully
[2017-06-01 19:24:29,932: INFO/job_scheduler] Job 1578: complete_job: Updating cache
[2017-06-01 19:24:29,932: INFO/job_scheduler] Job 1578 completing (errored=False, cancelled=False)
[2017-06-01 19:24:29,997: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:24:31,935: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:31,936: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:31,946: INFO/command_plan] add_jobs: created Job 1579 (Test for host connectivity)
[2017-06-01 19:24:31,973: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:24:31,995: INFO/job_scheduler] Job 1579: RunJobThread.run
[2017-06-01 19:24:32,067: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:32,068: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:32,104: INFO/command_plan] add_jobs: created Job 1580 (Test for host connectivity)
[2017-06-01 19:24:32,176: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:24:32,204: INFO/job_scheduler] Job 1580: RunJobThread.run
[2017-06-01 19:24:32,265: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:32,265: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:32,280: INFO/command_plan] add_jobs: created Job 1581 (Test for host connectivity)
[2017-06-01 19:24:32,327: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:24:32,390: INFO/job_scheduler] Job 1581: RunJobThread.run
[2017-06-01 19:24:32,452: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:32,453: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:32,556: INFO/command_plan] add_jobs: created Job 1582 (Test for host connectivity)
[2017-06-01 19:24:32,725: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:24:32,736: INFO/job_scheduler] Job 1582: RunJobThread.run
[2017-06-01 19:24:34,384: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:24:34,464: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:24:34,651: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:24:34,984: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:24:35,718: INFO/job_scheduler] Job 1579 finished 1 steps successfully
[2017-06-01 19:24:35,737: INFO/job_scheduler] Job 1579: complete_job: Updating cache
[2017-06-01 19:24:35,737: INFO/job_scheduler] Job 1579 completing (errored=False, cancelled=False)
[2017-06-01 19:24:35,779: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:24:35,798: INFO/job_scheduler] Job 1580 finished 1 steps successfully
[2017-06-01 19:24:35,872: INFO/job_scheduler] Job 1580: complete_job: Updating cache
[2017-06-01 19:24:35,873: INFO/job_scheduler] Job 1580 completing (errored=False, cancelled=False)
[2017-06-01 19:24:35,922: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:24:36,055: INFO/job_scheduler] Job 1581 finished 1 steps successfully
[2017-06-01 19:24:36,087: INFO/job_scheduler] Job 1581: complete_job: Updating cache
[2017-06-01 19:24:36,088: INFO/job_scheduler] Job 1581 completing (errored=False, cancelled=False)
[2017-06-01 19:24:36,187: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:24:36,458: INFO/job_scheduler] Job 1582 finished 1 steps successfully
[2017-06-01 19:24:36,478: INFO/job_scheduler] Job 1582: complete_job: Updating cache
[2017-06-01 19:24:36,479: INFO/job_scheduler] Job 1582 completing (errored=False, cancelled=False)
[2017-06-01 19:24:36,534: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:24:38,905: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:38,907: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:38,920: INFO/command_plan] add_jobs: created Job 1583 (Test for host connectivity)
[2017-06-01 19:24:38,954: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:24:38,965: INFO/job_scheduler] Job 1583: RunJobThread.run
[2017-06-01 19:24:39,017: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:39,017: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:39,038: INFO/command_plan] add_jobs: created Job 1584 (Test for host connectivity)
[2017-06-01 19:24:39,080: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:24:39,146: INFO/job_scheduler] Job 1584: RunJobThread.run
[2017-06-01 19:24:39,269: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:39,272: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:39,305: INFO/command_plan] add_jobs: created Job 1585 (Test for host connectivity)
[2017-06-01 19:24:39,397: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:24:39,402: INFO/job_scheduler] Job 1585: RunJobThread.run
[2017-06-01 19:24:39,512: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:39,512: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:39,529: INFO/command_plan] add_jobs: created Job 1586 (Test for host connectivity)
[2017-06-01 19:24:39,609: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:24:39,612: INFO/job_scheduler] Job 1586: RunJobThread.run
[2017-06-01 19:24:41,387: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:24:41,474: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:24:41,634: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:24:41,852: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:24:42,645: INFO/job_scheduler] Job 1583 finished 1 steps successfully
[2017-06-01 19:24:42,685: INFO/job_scheduler] Job 1583: complete_job: Updating cache
[2017-06-01 19:24:42,685: INFO/job_scheduler] Job 1583 completing (errored=False, cancelled=False)
[2017-06-01 19:24:42,754: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:24:42,797: INFO/job_scheduler] Job 1584 finished 1 steps successfully
[2017-06-01 19:24:42,837: INFO/job_scheduler] Job 1584: complete_job: Updating cache
[2017-06-01 19:24:42,837: INFO/job_scheduler] Job 1584 completing (errored=False, cancelled=False)
[2017-06-01 19:24:42,870: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:24:43,035: INFO/job_scheduler] Job 1585 finished 1 steps successfully
[2017-06-01 19:24:43,102: INFO/job_scheduler] Job 1585: complete_job: Updating cache
[2017-06-01 19:24:43,105: INFO/job_scheduler] Job 1585 completing (errored=False, cancelled=False)
[2017-06-01 19:24:43,178: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:24:43,402: INFO/job_scheduler] Job 1586 finished 1 steps successfully
[2017-06-01 19:24:43,444: INFO/job_scheduler] Job 1586: complete_job: Updating cache
[2017-06-01 19:24:43,444: INFO/job_scheduler] Job 1586 completing (errored=False, cancelled=False)
[2017-06-01 19:24:43,477: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:24:45,301: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:45,302: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:45,318: INFO/command_plan] add_jobs: created Job 1587 (Test for host connectivity)
[2017-06-01 19:24:45,344: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:24:45,351: INFO/job_scheduler] Job 1587: RunJobThread.run
[2017-06-01 19:24:45,427: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:45,428: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:45,462: INFO/command_plan] add_jobs: created Job 1588 (Test for host connectivity)
[2017-06-01 19:24:45,497: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:24:45,527: INFO/job_scheduler] Job 1588: RunJobThread.run
[2017-06-01 19:24:45,625: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:45,627: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:45,643: INFO/command_plan] add_jobs: created Job 1589 (Test for host connectivity)
[2017-06-01 19:24:45,668: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:24:45,677: INFO/job_scheduler] Job 1589: RunJobThread.run
[2017-06-01 19:24:45,842: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:45,842: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:45,886: INFO/command_plan] add_jobs: created Job 1590 (Test for host connectivity)
[2017-06-01 19:24:45,951: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:24:45,980: INFO/job_scheduler] Job 1590: RunJobThread.run
[2017-06-01 19:24:47,666: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:24:47,880: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:24:48,041: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:24:48,242: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:24:48,999: INFO/job_scheduler] Job 1587 finished 1 steps successfully
[2017-06-01 19:24:49,026: INFO/job_scheduler] Job 1587: complete_job: Updating cache
[2017-06-01 19:24:49,027: INFO/job_scheduler] Job 1587 completing (errored=False, cancelled=False)
[2017-06-01 19:24:49,086: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:24:49,203: INFO/job_scheduler] Job 1588 finished 1 steps successfully
[2017-06-01 19:24:49,251: INFO/job_scheduler] Job 1588: complete_job: Updating cache
[2017-06-01 19:24:49,251: INFO/job_scheduler] Job 1588 completing (errored=False, cancelled=False)
[2017-06-01 19:24:49,376: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:24:49,406: INFO/job_scheduler] Job 1589 finished 1 steps successfully
[2017-06-01 19:24:49,435: INFO/job_scheduler] Job 1589: complete_job: Updating cache
[2017-06-01 19:24:49,435: INFO/job_scheduler] Job 1589 completing (errored=False, cancelled=False)
[2017-06-01 19:24:49,494: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:24:49,652: INFO/job_scheduler] Job 1590 finished 1 steps successfully
[2017-06-01 19:24:49,701: INFO/job_scheduler] Job 1590: complete_job: Updating cache
[2017-06-01 19:24:49,701: INFO/job_scheduler] Job 1590 completing (errored=False, cancelled=False)
[2017-06-01 19:24:49,768: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:24:51,497: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:51,500: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:51,518: INFO/command_plan] add_jobs: created Job 1591 (Test for host connectivity)
[2017-06-01 19:24:51,550: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:24:51,555: INFO/job_scheduler] Job 1591: RunJobThread.run
[2017-06-01 19:24:51,675: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:51,676: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:51,694: INFO/command_plan] add_jobs: created Job 1592 (Test for host connectivity)
[2017-06-01 19:24:51,808: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:24:51,838: INFO/job_scheduler] Job 1592: RunJobThread.run
[2017-06-01 19:24:51,873: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:51,874: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:51,893: INFO/command_plan] add_jobs: created Job 1593 (Test for host connectivity)
[2017-06-01 19:24:51,923: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:24:51,966: INFO/job_scheduler] Job 1593: RunJobThread.run
[2017-06-01 19:24:52,064: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:52,065: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:52,089: INFO/command_plan] add_jobs: created Job 1594 (Test for host connectivity)
[2017-06-01 19:24:52,134: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:24:52,136: INFO/job_scheduler] Job 1594: RunJobThread.run
[2017-06-01 19:24:54,027: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:24:54,183: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:24:54,224: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:24:54,400: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:24:55,361: INFO/job_scheduler] Job 1591 finished 1 steps successfully
[2017-06-01 19:24:55,400: INFO/job_scheduler] Job 1591: complete_job: Updating cache
[2017-06-01 19:24:55,400: INFO/job_scheduler] Job 1591 completing (errored=False, cancelled=False)
[2017-06-01 19:24:55,475: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:24:55,535: INFO/job_scheduler] Job 1592 finished 1 steps successfully
[2017-06-01 19:24:55,575: INFO/job_scheduler] Job 1592: complete_job: Updating cache
[2017-06-01 19:24:55,575: INFO/job_scheduler] Job 1592 completing (errored=False, cancelled=False)
[2017-06-01 19:24:55,625: INFO/job_scheduler] Job 1593 finished 1 steps successfully
[2017-06-01 19:24:55,668: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:24:55,710: INFO/job_scheduler] Job 1593: complete_job: Updating cache
[2017-06-01 19:24:55,710: INFO/job_scheduler] Job 1593 completing (errored=False, cancelled=False)
[2017-06-01 19:24:55,735: INFO/job_scheduler] Job 1594 finished 1 steps successfully
[2017-06-01 19:24:55,800: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:24:55,843: INFO/job_scheduler] Job 1594: complete_job: Updating cache
[2017-06-01 19:24:55,843: INFO/job_scheduler] Job 1594 completing (errored=False, cancelled=False)
[2017-06-01 19:24:55,902: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:24:57,671: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:57,671: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:57,703: INFO/command_plan] add_jobs: created Job 1595 (Test for host connectivity)
[2017-06-01 19:24:57,738: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:24:57,763: INFO/job_scheduler] Job 1595: RunJobThread.run
[2017-06-01 19:24:57,828: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:57,830: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:57,844: INFO/command_plan] add_jobs: created Job 1596 (Test for host connectivity)
[2017-06-01 19:24:57,942: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:24:57,959: INFO/job_scheduler] Job 1596: RunJobThread.run
[2017-06-01 19:24:58,019: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:58,020: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:58,027: INFO/command_plan] add_jobs: created Job 1597 (Test for host connectivity)
[2017-06-01 19:24:58,233: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:24:58,264: INFO/job_scheduler] Job 1597: RunJobThread.run
[2017-06-01 19:24:58,310: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:24:58,310: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:24:58,357: INFO/command_plan] add_jobs: created Job 1598 (Test for host connectivity)
[2017-06-01 19:24:58,498: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:24:58,506: INFO/job_scheduler] Job 1598: RunJobThread.run
[2017-06-01 19:25:00,179: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:25:00,395: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:25:00,626: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:25:00,797: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:25:01,518: INFO/job_scheduler] Job 1595 finished 1 steps successfully
[2017-06-01 19:25:01,557: INFO/job_scheduler] Job 1595: complete_job: Updating cache
[2017-06-01 19:25:01,558: INFO/job_scheduler] Job 1595 completing (errored=False, cancelled=False)
[2017-06-01 19:25:01,621: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:25:01,857: INFO/job_scheduler] Job 1596 finished 1 steps successfully
[2017-06-01 19:25:01,910: INFO/job_scheduler] Job 1596: complete_job: Updating cache
[2017-06-01 19:25:01,910: INFO/job_scheduler] Job 1596 completing (errored=False, cancelled=False)
[2017-06-01 19:25:01,945: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:25:01,993: INFO/job_scheduler] Job 1597 finished 1 steps successfully
[2017-06-01 19:25:02,067: INFO/job_scheduler] Job 1597: complete_job: Updating cache
[2017-06-01 19:25:02,067: INFO/job_scheduler] Job 1597 completing (errored=False, cancelled=False)
[2017-06-01 19:25:02,112: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:25:02,393: INFO/job_scheduler] Job 1598 finished 1 steps successfully
[2017-06-01 19:25:02,418: INFO/job_scheduler] Job 1598: complete_job: Updating cache
[2017-06-01 19:25:02,418: INFO/job_scheduler] Job 1598 completing (errored=False, cancelled=False)
[2017-06-01 19:25:02,481: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:25:04,166: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:04,167: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:04,193: INFO/command_plan] add_jobs: created Job 1599 (Test for host connectivity)
[2017-06-01 19:25:04,223: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:25:04,270: INFO/job_scheduler] Job 1599: RunJobThread.run
[2017-06-01 19:25:04,346: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:04,346: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:04,360: INFO/command_plan] add_jobs: created Job 1600 (Test for host connectivity)
[2017-06-01 19:25:04,427: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:25:04,458: INFO/job_scheduler] Job 1600: RunJobThread.run
[2017-06-01 19:25:04,524: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:04,524: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:04,553: INFO/command_plan] add_jobs: created Job 1601 (Test for host connectivity)
[2017-06-01 19:25:04,610: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:25:04,635: INFO/job_scheduler] Job 1601: RunJobThread.run
[2017-06-01 19:25:04,715: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:04,717: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:04,731: INFO/command_plan] add_jobs: created Job 1602 (Test for host connectivity)
[2017-06-01 19:25:04,815: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:25:04,847: INFO/job_scheduler] Job 1602: RunJobThread.run
[2017-06-01 19:25:06,608: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:25:06,767: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:25:06,982: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:25:07,156: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:25:07,904: INFO/job_scheduler] Job 1599 finished 1 steps successfully
[2017-06-01 19:25:07,953: INFO/job_scheduler] Job 1599: complete_job: Updating cache
[2017-06-01 19:25:07,953: INFO/job_scheduler] Job 1599 completing (errored=False, cancelled=False)
[2017-06-01 19:25:08,003: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:25:08,185: INFO/job_scheduler] Job 1600 finished 1 steps successfully
[2017-06-01 19:25:08,245: INFO/job_scheduler] Job 1600: complete_job: Updating cache
[2017-06-01 19:25:08,245: INFO/job_scheduler] Job 1600 completing (errored=False, cancelled=False)
[2017-06-01 19:25:08,317: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:25:08,318: INFO/job_scheduler] Job 1601 finished 1 steps successfully
[2017-06-01 19:25:08,349: INFO/job_scheduler] Job 1601: complete_job: Updating cache
[2017-06-01 19:25:08,350: INFO/job_scheduler] Job 1601 completing (errored=False, cancelled=False)
[2017-06-01 19:25:08,454: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:25:08,675: INFO/job_scheduler] Job 1602 finished 1 steps successfully
[2017-06-01 19:25:08,700: INFO/job_scheduler] Job 1602: complete_job: Updating cache
[2017-06-01 19:25:08,700: INFO/job_scheduler] Job 1602 completing (errored=False, cancelled=False)
[2017-06-01 19:25:08,735: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:25:10,355: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:10,356: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:10,384: INFO/command_plan] add_jobs: created Job 1603 (Test for host connectivity)
[2017-06-01 19:25:10,411: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:25:10,428: INFO/job_scheduler] Job 1603: RunJobThread.run
[2017-06-01 19:25:10,491: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:10,491: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:10,502: INFO/command_plan] add_jobs: created Job 1604 (Test for host connectivity)
[2017-06-01 19:25:10,555: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:25:10,621: INFO/job_scheduler] Job 1604: RunJobThread.run
[2017-06-01 19:25:10,680: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:10,682: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:10,701: INFO/command_plan] add_jobs: created Job 1605 (Test for host connectivity)
[2017-06-01 19:25:10,719: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:25:10,760: INFO/job_scheduler] Job 1605: RunJobThread.run
[2017-06-01 19:25:10,842: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:10,843: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:10,867: INFO/command_plan] add_jobs: created Job 1606 (Test for host connectivity)
[2017-06-01 19:25:10,916: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:25:11,020: INFO/job_scheduler] Job 1606: RunJobThread.run
[2017-06-01 19:25:12,875: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:25:13,030: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:25:13,143: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:25:13,214: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:25:14,197: INFO/job_scheduler] Job 1603 finished 1 steps successfully
[2017-06-01 19:25:14,243: INFO/job_scheduler] Job 1603: complete_job: Updating cache
[2017-06-01 19:25:14,244: INFO/job_scheduler] Job 1603 completing (errored=False, cancelled=False)
[2017-06-01 19:25:14,349: INFO/job_scheduler] Job 1604 finished 1 steps successfully
[2017-06-01 19:25:14,398: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:25:14,438: INFO/job_scheduler] Job 1604: complete_job: Updating cache
[2017-06-01 19:25:14,438: INFO/job_scheduler] Job 1604 completing (errored=False, cancelled=False)
[2017-06-01 19:25:14,499: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:25:14,599: INFO/job_scheduler] Job 1606 finished 1 steps successfully
[2017-06-01 19:25:14,646: INFO/job_scheduler] Job 1606: complete_job: Updating cache
[2017-06-01 19:25:14,651: INFO/job_scheduler] Job 1606 completing (errored=False, cancelled=False)
[2017-06-01 19:25:14,663: INFO/job_scheduler] Job 1605 finished 1 steps successfully
[2017-06-01 19:25:14,725: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:25:14,741: INFO/job_scheduler] Job 1605: complete_job: Updating cache
[2017-06-01 19:25:14,757: INFO/job_scheduler] Job 1605 completing (errored=False, cancelled=False)
[2017-06-01 19:25:14,834: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:25:16,525: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:16,526: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:16,541: INFO/command_plan] add_jobs: created Job 1607 (Test for host connectivity)
[2017-06-01 19:25:16,619: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:25:16,642: INFO/job_scheduler] Job 1607: RunJobThread.run
[2017-06-01 19:25:16,679: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:16,681: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:16,707: INFO/command_plan] add_jobs: created Job 1608 (Test for host connectivity)
[2017-06-01 19:25:16,789: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:25:16,792: INFO/job_scheduler] Job 1608: RunJobThread.run
[2017-06-01 19:25:16,869: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:16,878: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:16,915: INFO/command_plan] add_jobs: created Job 1609 (Test for host connectivity)
[2017-06-01 19:25:16,994: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:25:17,014: INFO/job_scheduler] Job 1609: RunJobThread.run
[2017-06-01 19:25:17,078: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:17,080: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:17,088: INFO/command_plan] add_jobs: created Job 1610 (Test for host connectivity)
[2017-06-01 19:25:17,129: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:25:17,183: INFO/job_scheduler] Job 1610: RunJobThread.run
[2017-06-01 19:25:19,032: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:25:19,094: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:25:19,299: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:25:19,500: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:25:20,348: INFO/job_scheduler] Job 1607 finished 1 steps successfully
[2017-06-01 19:25:20,430: INFO/job_scheduler] Job 1607: complete_job: Updating cache
[2017-06-01 19:25:20,431: INFO/job_scheduler] Job 1607 completing (errored=False, cancelled=False)
[2017-06-01 19:25:20,487: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:25:20,501: INFO/job_scheduler] Job 1608 finished 1 steps successfully
[2017-06-01 19:25:20,586: INFO/job_scheduler] Job 1608: complete_job: Updating cache
[2017-06-01 19:25:20,587: INFO/job_scheduler] Job 1608 completing (errored=False, cancelled=False)
[2017-06-01 19:25:20,704: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:25:20,777: INFO/job_scheduler] Job 1609 finished 1 steps successfully
[2017-06-01 19:25:20,811: INFO/job_scheduler] Job 1609: complete_job: Updating cache
[2017-06-01 19:25:20,812: INFO/job_scheduler] Job 1609 completing (errored=False, cancelled=False)
[2017-06-01 19:25:20,936: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:25:20,987: INFO/job_scheduler] Job 1610 finished 1 steps successfully
[2017-06-01 19:25:21,047: INFO/job_scheduler] Job 1610: complete_job: Updating cache
[2017-06-01 19:25:21,047: INFO/job_scheduler] Job 1610 completing (errored=False, cancelled=False)
[2017-06-01 19:25:21,079: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:25:22,761: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:22,761: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:22,788: INFO/command_plan] add_jobs: created Job 1611 (Test for host connectivity)
[2017-06-01 19:25:22,859: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:25:22,892: INFO/job_scheduler] Job 1611: RunJobThread.run
[2017-06-01 19:25:22,941: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:22,944: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:22,956: INFO/command_plan] add_jobs: created Job 1612 (Test for host connectivity)
[2017-06-01 19:25:22,998: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:25:23,022: INFO/job_scheduler] Job 1612: RunJobThread.run
[2017-06-01 19:25:23,150: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:23,151: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:23,204: INFO/command_plan] add_jobs: created Job 1613 (Test for host connectivity)
[2017-06-01 19:25:23,286: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:25:23,301: INFO/job_scheduler] Job 1613: RunJobThread.run
[2017-06-01 19:25:23,381: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:23,382: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:23,394: INFO/command_plan] add_jobs: created Job 1614 (Test for host connectivity)
[2017-06-01 19:25:23,491: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:25:23,501: INFO/job_scheduler] Job 1614: RunJobThread.run
[2017-06-01 19:25:25,386: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:25:25,445: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:25:25,640: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:25:25,803: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:25:26,780: INFO/job_scheduler] Job 1612 finished 1 steps successfully
[2017-06-01 19:25:26,844: INFO/job_scheduler] Job 1612: complete_job: Updating cache
[2017-06-01 19:25:26,844: INFO/job_scheduler] Job 1612 completing (errored=False, cancelled=False)
[2017-06-01 19:25:26,846: INFO/job_scheduler] Job 1611 finished 1 steps successfully
[2017-06-01 19:25:26,937: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:25:27,039: INFO/job_scheduler] Job 1613 finished 1 steps successfully
[2017-06-01 19:25:27,077: INFO/job_scheduler] Job 1611: complete_job: Updating cache
[2017-06-01 19:25:27,077: INFO/job_scheduler] Job 1611 completing (errored=False, cancelled=False)
[2017-06-01 19:25:27,152: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:25:27,161: INFO/job_scheduler] Job 1613: complete_job: Updating cache
[2017-06-01 19:25:27,162: INFO/job_scheduler] Job 1613 completing (errored=False, cancelled=False)
[2017-06-01 19:25:27,206: INFO/job_scheduler] Job 1614 finished 1 steps successfully
[2017-06-01 19:25:27,262: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:25:27,270: INFO/job_scheduler] Job 1614: complete_job: Updating cache
[2017-06-01 19:25:27,270: INFO/job_scheduler] Job 1614 completing (errored=False, cancelled=False)
[2017-06-01 19:25:27,357: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:25:29,068: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:29,068: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:29,101: INFO/command_plan] add_jobs: created Job 1615 (Test for host connectivity)
[2017-06-01 19:25:29,186: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:25:29,188: INFO/job_scheduler] Job 1615: RunJobThread.run
[2017-06-01 19:25:29,240: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:29,240: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:29,269: INFO/command_plan] add_jobs: created Job 1616 (Test for host connectivity)
[2017-06-01 19:25:29,359: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:25:29,375: INFO/job_scheduler] Job 1616: RunJobThread.run
[2017-06-01 19:25:29,443: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:29,446: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:29,473: INFO/command_plan] add_jobs: created Job 1617 (Test for host connectivity)
[2017-06-01 19:25:29,606: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:25:29,613: INFO/job_scheduler] Job 1617: RunJobThread.run
[2017-06-01 19:25:29,718: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:29,719: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:29,742: INFO/command_plan] add_jobs: created Job 1618 (Test for host connectivity)
[2017-06-01 19:25:29,778: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:25:29,824: INFO/job_scheduler] Job 1618: RunJobThread.run
[2017-06-01 19:25:31,518: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:25:31,772: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:25:31,902: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:25:32,129: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:25:32,855: INFO/job_scheduler] Job 1615 finished 1 steps successfully
[2017-06-01 19:25:32,893: INFO/job_scheduler] Job 1615: complete_job: Updating cache
[2017-06-01 19:25:32,894: INFO/job_scheduler] Job 1615 completing (errored=False, cancelled=False)
[2017-06-01 19:25:32,952: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:25:33,088: INFO/job_scheduler] Job 1616 finished 1 steps successfully
[2017-06-01 19:25:33,135: INFO/job_scheduler] Job 1616: complete_job: Updating cache
[2017-06-01 19:25:33,137: INFO/job_scheduler] Job 1616 completing (errored=False, cancelled=False)
[2017-06-01 19:25:33,235: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:25:33,356: INFO/job_scheduler] Job 1617 finished 1 steps successfully
[2017-06-01 19:25:33,418: INFO/job_scheduler] Job 1617: complete_job: Updating cache
[2017-06-01 19:25:33,419: INFO/job_scheduler] Job 1617 completing (errored=False, cancelled=False)
[2017-06-01 19:25:33,468: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:25:33,645: INFO/job_scheduler] Job 1618 finished 1 steps successfully
[2017-06-01 19:25:33,668: INFO/job_scheduler] Job 1618: complete_job: Updating cache
[2017-06-01 19:25:33,668: INFO/job_scheduler] Job 1618 completing (errored=False, cancelled=False)
[2017-06-01 19:25:33,719: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:25:35,395: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:35,396: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:35,412: INFO/command_plan] add_jobs: created Job 1619 (Test for host connectivity)
[2017-06-01 19:25:35,469: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:25:35,479: INFO/job_scheduler] Job 1619: RunJobThread.run
[2017-06-01 19:25:35,510: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:35,510: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:35,528: INFO/command_plan] add_jobs: created Job 1620 (Test for host connectivity)
[2017-06-01 19:25:35,570: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:25:35,606: INFO/job_scheduler] Job 1620: RunJobThread.run
[2017-06-01 19:25:35,785: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:35,787: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:35,801: INFO/command_plan] add_jobs: created Job 1621 (Test for host connectivity)
[2017-06-01 19:25:35,860: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:25:35,864: INFO/job_scheduler] Job 1621: RunJobThread.run
[2017-06-01 19:25:35,976: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:35,977: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:36,001: INFO/command_plan] add_jobs: created Job 1622 (Test for host connectivity)
[2017-06-01 19:25:36,081: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:25:36,131: INFO/job_scheduler] Job 1622: RunJobThread.run
[2017-06-01 19:25:37,897: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:25:37,949: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:25:38,220: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:25:38,323: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:25:39,213: INFO/job_scheduler] Job 1619 finished 1 steps successfully
[2017-06-01 19:25:39,259: INFO/job_scheduler] Job 1619: complete_job: Updating cache
[2017-06-01 19:25:39,259: INFO/job_scheduler] Job 1619 completing (errored=False, cancelled=False)
[2017-06-01 19:25:39,354: INFO/job_scheduler] Job 1620 finished 1 steps successfully
[2017-06-01 19:25:39,378: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:25:39,421: INFO/job_scheduler] Job 1620: complete_job: Updating cache
[2017-06-01 19:25:39,421: INFO/job_scheduler] Job 1620 completing (errored=False, cancelled=False)
[2017-06-01 19:25:39,469: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:25:39,628: INFO/job_scheduler] Job 1621 finished 1 steps successfully
[2017-06-01 19:25:39,659: INFO/job_scheduler] Job 1621: complete_job: Updating cache
[2017-06-01 19:25:39,660: INFO/job_scheduler] Job 1621 completing (errored=False, cancelled=False)
[2017-06-01 19:25:39,720: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:25:39,754: INFO/job_scheduler] Job 1622 finished 1 steps successfully
[2017-06-01 19:25:39,820: INFO/job_scheduler] Job 1622: complete_job: Updating cache
[2017-06-01 19:25:39,821: INFO/job_scheduler] Job 1622 completing (errored=False, cancelled=False)
[2017-06-01 19:25:39,876: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:25:41,594: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:41,594: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:41,678: INFO/command_plan] add_jobs: created Job 1623 (Test for host connectivity)
[2017-06-01 19:25:41,701: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:25:41,709: INFO/job_scheduler] Job 1623: RunJobThread.run
[2017-06-01 19:25:41,779: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:41,779: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:41,804: INFO/command_plan] add_jobs: created Job 1624 (Test for host connectivity)
[2017-06-01 19:25:41,953: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:41,954: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:41,970: INFO/command_plan] add_jobs: created Job 1625 (Test for host connectivity)
[2017-06-01 19:25:41,985: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:25:42,068: INFO/job_scheduler] Job 1624: RunJobThread.run
[2017-06-01 19:25:42,074: INFO/job_scheduler] Job 1625: RunJobThread.run
[2017-06-01 19:25:42,132: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:42,132: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:42,156: INFO/command_plan] add_jobs: created Job 1626 (Test for host connectivity)
[2017-06-01 19:25:42,215: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:25:42,236: INFO/job_scheduler] Job 1626: RunJobThread.run
[2017-06-01 19:25:42,321: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:25:44,181: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:25:44,325: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:25:44,478: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:25:44,559: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:25:45,465: INFO/job_scheduler] Job 1623 finished 1 steps successfully
[2017-06-01 19:25:45,494: INFO/job_scheduler] Job 1623: complete_job: Updating cache
[2017-06-01 19:25:45,494: INFO/job_scheduler] Job 1623 completing (errored=False, cancelled=False)
[2017-06-01 19:25:45,550: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:25:45,675: INFO/job_scheduler] Job 1624 finished 1 steps successfully
[2017-06-01 19:25:45,729: INFO/job_scheduler] Job 1624: complete_job: Updating cache
[2017-06-01 19:25:45,731: INFO/job_scheduler] Job 1624 completing (errored=False, cancelled=False)
[2017-06-01 19:25:45,785: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:25:45,950: INFO/job_scheduler] Job 1626 finished 1 steps successfully
[2017-06-01 19:25:45,969: INFO/job_scheduler] Job 1625 finished 1 steps successfully
[2017-06-01 19:25:45,983: INFO/job_scheduler] Job 1626: complete_job: Updating cache
[2017-06-01 19:25:45,983: INFO/job_scheduler] Job 1626 completing (errored=False, cancelled=False)
[2017-06-01 19:25:46,027: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:25:46,055: INFO/job_scheduler] Job 1625: complete_job: Updating cache
[2017-06-01 19:25:46,060: INFO/job_scheduler] Job 1625 completing (errored=False, cancelled=False)
[2017-06-01 19:25:46,170: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:25:47,870: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:47,870: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:47,892: INFO/command_plan] add_jobs: created Job 1627 (Test for host connectivity)
[2017-06-01 19:25:47,952: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:25:47,968: INFO/job_scheduler] Job 1627: RunJobThread.run
[2017-06-01 19:25:48,020: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:48,021: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:48,045: INFO/command_plan] add_jobs: created Job 1628 (Test for host connectivity)
[2017-06-01 19:25:48,083: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:25:48,100: INFO/job_scheduler] Job 1628: RunJobThread.run
[2017-06-01 19:25:48,251: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:48,254: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:48,276: INFO/command_plan] add_jobs: created Job 1629 (Test for host connectivity)
[2017-06-01 19:25:48,373: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:25:48,397: INFO/job_scheduler] Job 1629: RunJobThread.run
[2017-06-01 19:25:48,446: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:48,447: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:48,469: INFO/command_plan] add_jobs: created Job 1630 (Test for host connectivity)
[2017-06-01 19:25:48,566: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:25:48,607: INFO/job_scheduler] Job 1630: RunJobThread.run
[2017-06-01 19:25:50,501: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:25:50,600: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:25:50,730: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:25:51,004: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:25:51,845: INFO/job_scheduler] Job 1627 finished 1 steps successfully
[2017-06-01 19:25:51,874: INFO/job_scheduler] Job 1627: complete_job: Updating cache
[2017-06-01 19:25:51,874: INFO/job_scheduler] Job 1627 completing (errored=False, cancelled=False)
[2017-06-01 19:25:51,908: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:25:51,925: INFO/job_scheduler] Job 1628 finished 1 steps successfully
[2017-06-01 19:25:51,979: INFO/job_scheduler] Job 1628: complete_job: Updating cache
[2017-06-01 19:25:51,979: INFO/job_scheduler] Job 1628 completing (errored=False, cancelled=False)
[2017-06-01 19:25:52,047: INFO/job_scheduler] Job 1629 finished 1 steps successfully
[2017-06-01 19:25:52,087: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:25:52,111: INFO/job_scheduler] Job 1629: complete_job: Updating cache
[2017-06-01 19:25:52,112: INFO/job_scheduler] Job 1629 completing (errored=False, cancelled=False)
[2017-06-01 19:25:52,223: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:25:52,471: INFO/job_scheduler] Job 1630 finished 1 steps successfully
[2017-06-01 19:25:52,499: INFO/job_scheduler] Job 1630: complete_job: Updating cache
[2017-06-01 19:25:52,500: INFO/job_scheduler] Job 1630 completing (errored=False, cancelled=False)
[2017-06-01 19:25:52,568: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:25:54,103: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:54,105: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:54,116: INFO/command_plan] add_jobs: created Job 1631 (Test for host connectivity)
[2017-06-01 19:25:54,164: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:25:54,180: INFO/job_scheduler] Job 1631: RunJobThread.run
[2017-06-01 19:25:54,245: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:54,245: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:54,272: INFO/command_plan] add_jobs: created Job 1632 (Test for host connectivity)
[2017-06-01 19:25:54,393: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:25:54,406: INFO/job_scheduler] Job 1632: RunJobThread.run
[2017-06-01 19:25:54,462: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:54,464: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:54,474: INFO/command_plan] add_jobs: created Job 1633 (Test for host connectivity)
[2017-06-01 19:25:54,512: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:25:54,542: INFO/job_scheduler] Job 1633: RunJobThread.run
[2017-06-01 19:25:54,633: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:25:54,635: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:25:54,665: INFO/command_plan] add_jobs: created Job 1634 (Test for host connectivity)
[2017-06-01 19:25:54,752: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:25:54,801: INFO/job_scheduler] Job 1634: RunJobThread.run
[2017-06-01 19:25:56,679: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:25:56,710: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:25:56,870: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:25:57,143: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:25:58,014: INFO/job_scheduler] Job 1632 finished 1 steps successfully
[2017-06-01 19:25:58,065: INFO/job_scheduler] Job 1632: complete_job: Updating cache
[2017-06-01 19:25:58,066: INFO/job_scheduler] Job 1632 completing (errored=False, cancelled=False)
[2017-06-01 19:25:58,184: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:25:58,186: INFO/job_scheduler] Job 1631 finished 1 steps successfully
[2017-06-01 19:25:58,187: INFO/job_scheduler] Job 1633 finished 1 steps successfully
[2017-06-01 19:25:58,266: INFO/job_scheduler] Job 1631: complete_job: Updating cache
[2017-06-01 19:25:58,266: INFO/job_scheduler] Job 1631 completing (errored=False, cancelled=False)
[2017-06-01 19:25:58,475: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:25:58,499: INFO/job_scheduler] Job 1633: complete_job: Updating cache
[2017-06-01 19:25:58,499: INFO/job_scheduler] Job 1633 completing (errored=False, cancelled=False)
[2017-06-01 19:25:58,583: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:25:58,829: INFO/job_scheduler] Job 1634 finished 1 steps successfully
[2017-06-01 19:25:58,878: INFO/job_scheduler] Job 1634: complete_job: Updating cache
[2017-06-01 19:25:58,879: INFO/job_scheduler] Job 1634 completing (errored=False, cancelled=False)
[2017-06-01 19:25:58,996: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:26:00,361: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:00,364: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:00,372: INFO/command_plan] add_jobs: created Job 1635 (Test for host connectivity)
[2017-06-01 19:26:00,418: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:00,418: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:00,457: INFO/command_plan] add_jobs: created Job 1636 (Test for host connectivity)
[2017-06-01 19:26:00,536: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:26:00,556: INFO/job_scheduler] Job 1635: RunJobThread.run
[2017-06-01 19:26:00,576: INFO/job_scheduler] Job 1636: RunJobThread.run
[2017-06-01 19:26:00,598: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:26:00,736: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:00,737: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:00,752: INFO/command_plan] add_jobs: created Job 1637 (Test for host connectivity)
[2017-06-01 19:26:00,813: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:26:00,858: INFO/job_scheduler] Job 1637: RunJobThread.run
[2017-06-01 19:26:00,934: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:00,934: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:00,965: INFO/command_plan] add_jobs: created Job 1638 (Test for host connectivity)
[2017-06-01 19:26:01,104: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:26:01,119: INFO/job_scheduler] Job 1638: RunJobThread.run
[2017-06-01 19:26:02,886: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:26:03,094: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:26:03,118: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:26:03,455: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:26:04,235: INFO/job_scheduler] Job 1635 finished 1 steps successfully
[2017-06-01 19:26:04,271: INFO/job_scheduler] Job 1635: complete_job: Updating cache
[2017-06-01 19:26:04,272: INFO/job_scheduler] Job 1635 completing (errored=False, cancelled=False)
[2017-06-01 19:26:04,331: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:26:04,444: INFO/job_scheduler] Job 1636 finished 1 steps successfully
[2017-06-01 19:26:04,453: INFO/job_scheduler] Job 1637 finished 1 steps successfully
[2017-06-01 19:26:04,539: INFO/job_scheduler] Job 1636: complete_job: Updating cache
[2017-06-01 19:26:04,540: INFO/job_scheduler] Job 1636 completing (errored=False, cancelled=False)
[2017-06-01 19:26:04,605: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:26:04,625: INFO/job_scheduler] Job 1637: complete_job: Updating cache
[2017-06-01 19:26:04,625: INFO/job_scheduler] Job 1637 completing (errored=False, cancelled=False)
[2017-06-01 19:26:04,736: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:26:04,822: INFO/job_scheduler] Job 1638 finished 1 steps successfully
[2017-06-01 19:26:04,855: INFO/job_scheduler] Job 1638: complete_job: Updating cache
[2017-06-01 19:26:04,855: INFO/job_scheduler] Job 1638 completing (errored=False, cancelled=False)
[2017-06-01 19:26:04,889: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:26:06,682: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:06,682: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:06,698: INFO/command_plan] add_jobs: created Job 1639 (Test for host connectivity)
[2017-06-01 19:26:06,786: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:06,786: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:06,796: INFO/command_plan] add_jobs: created Job 1640 (Test for host connectivity)
[2017-06-01 19:26:06,806: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:26:06,831: INFO/job_scheduler] Job 1640: RunJobThread.run
[2017-06-01 19:26:06,870: INFO/job_scheduler] Job 1639: RunJobThread.run
[2017-06-01 19:26:06,957: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:06,958: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:06,980: INFO/command_plan] add_jobs: created Job 1641 (Test for host connectivity)
[2017-06-01 19:26:07,025: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:26:07,080: INFO/job_scheduler] Job 1641: RunJobThread.run
[2017-06-01 19:26:07,172: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:07,173: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:07,202: INFO/command_plan] add_jobs: created Job 1642 (Test for host connectivity)
[2017-06-01 19:26:07,228: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:26:07,239: INFO/job_scheduler] Job 1642: RunJobThread.run
[2017-06-01 19:26:07,296: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:26:09,271: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:26:09,279: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:26:09,503: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:26:09,528: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:26:10,611: INFO/job_scheduler] Job 1639 finished 1 steps successfully
[2017-06-01 19:26:10,615: INFO/job_scheduler] Job 1640 finished 1 steps successfully
[2017-06-01 19:26:10,662: INFO/job_scheduler] Job 1639: complete_job: Updating cache
[2017-06-01 19:26:10,662: INFO/job_scheduler] Job 1639 completing (errored=False, cancelled=False)
[2017-06-01 19:26:10,698: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:26:10,712: INFO/job_scheduler] Job 1640: complete_job: Updating cache
[2017-06-01 19:26:10,713: INFO/job_scheduler] Job 1640 completing (errored=False, cancelled=False)
[2017-06-01 19:26:10,829: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:26:10,883: INFO/job_scheduler] Job 1641 finished 1 steps successfully
[2017-06-01 19:26:10,889: INFO/job_scheduler] Job 1642 finished 1 steps successfully
[2017-06-01 19:26:10,979: INFO/job_scheduler] Job 1641: complete_job: Updating cache
[2017-06-01 19:26:10,982: INFO/job_scheduler] Job 1641 completing (errored=False, cancelled=False)
[2017-06-01 19:26:11,028: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:26:11,045: INFO/job_scheduler] Job 1642: complete_job: Updating cache
[2017-06-01 19:26:11,045: INFO/job_scheduler] Job 1642 completing (errored=False, cancelled=False)
[2017-06-01 19:26:11,138: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:26:12,471: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:12,471: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:12,487: INFO/command_plan] add_jobs: created Job 1643 (Test for host connectivity)
[2017-06-01 19:26:12,507: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:26:12,553: INFO/job_scheduler] Job 1643: RunJobThread.run
[2017-06-01 19:26:12,642: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:12,642: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:12,703: INFO/command_plan] add_jobs: created Job 1644 (Test for host connectivity)
[2017-06-01 19:26:12,748: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:26:12,784: INFO/job_scheduler] Job 1644: RunJobThread.run
[2017-06-01 19:26:12,859: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:12,859: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:12,880: INFO/command_plan] add_jobs: created Job 1645 (Test for host connectivity)
[2017-06-01 19:26:12,938: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:26:12,968: INFO/job_scheduler] Job 1645: RunJobThread.run
[2017-06-01 19:26:13,058: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:13,059: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:13,084: INFO/command_plan] add_jobs: created Job 1646 (Test for host connectivity)
[2017-06-01 19:26:13,129: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:26:13,190: INFO/job_scheduler] Job 1646: RunJobThread.run
[2017-06-01 19:26:15,091: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:26:15,144: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:26:15,309: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:26:15,446: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:26:16,424: INFO/job_scheduler] Job 1643 finished 1 steps successfully
[2017-06-01 19:26:16,529: INFO/job_scheduler] Job 1643: complete_job: Updating cache
[2017-06-01 19:26:16,535: INFO/job_scheduler] Job 1643 completing (errored=False, cancelled=False)
[2017-06-01 19:26:16,536: INFO/job_scheduler] Job 1644 finished 1 steps successfully
[2017-06-01 19:26:16,584: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:26:16,639: INFO/job_scheduler] Job 1644: complete_job: Updating cache
[2017-06-01 19:26:16,639: INFO/job_scheduler] Job 1644 completing (errored=False, cancelled=False)
[2017-06-01 19:26:16,712: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:26:16,931: INFO/job_scheduler] Job 1645 finished 1 steps successfully
[2017-06-01 19:26:16,965: INFO/job_scheduler] Job 1645: complete_job: Updating cache
[2017-06-01 19:26:16,966: INFO/job_scheduler] Job 1645 completing (errored=False, cancelled=False)
[2017-06-01 19:26:16,968: INFO/job_scheduler] Job 1646 finished 1 steps successfully
[2017-06-01 19:26:17,049: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:26:17,067: INFO/job_scheduler] Job 1646: complete_job: Updating cache
[2017-06-01 19:26:17,067: INFO/job_scheduler] Job 1646 completing (errored=False, cancelled=False)
[2017-06-01 19:26:17,174: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:26:18,956: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:18,958: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:18,975: INFO/command_plan] add_jobs: created Job 1647 (Test for host connectivity)
[2017-06-01 19:26:19,120: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:19,120: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:19,143: INFO/command_plan] add_jobs: created Job 1648 (Test for host connectivity)
[2017-06-01 19:26:19,219: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:26:19,247: INFO/job_scheduler] Job 1648: RunJobThread.run
[2017-06-01 19:26:19,248: INFO/job_scheduler] Job 1647: RunJobThread.run
[2017-06-01 19:26:19,286: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:19,289: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:19,314: INFO/command_plan] add_jobs: created Job 1649 (Test for host connectivity)
[2017-06-01 19:26:19,330: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:26:19,348: INFO/job_scheduler] Job 1649: RunJobThread.run
[2017-06-01 19:26:19,375: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:26:19,424: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:19,424: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:19,499: INFO/command_plan] add_jobs: created Job 1650 (Test for host connectivity)
[2017-06-01 19:26:19,563: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:26:19,565: INFO/job_scheduler] Job 1650: RunJobThread.run
[2017-06-01 19:26:21,544: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:26:21,603: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:26:21,697: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:26:21,788: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:26:22,883: INFO/job_scheduler] Job 1648 finished 1 steps successfully
[2017-06-01 19:26:23,006: INFO/job_scheduler] Job 1648: complete_job: Updating cache
[2017-06-01 19:26:23,006: INFO/job_scheduler] Job 1648 completing (errored=False, cancelled=False)
[2017-06-01 19:26:23,034: INFO/job_scheduler] Job 1647 finished 1 steps successfully
[2017-06-01 19:26:23,048: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:26:23,135: INFO/job_scheduler] Job 1650 finished 1 steps successfully
[2017-06-01 19:26:23,139: INFO/job_scheduler] Job 1649 finished 1 steps successfully
[2017-06-01 19:26:23,254: INFO/job_scheduler] Job 1647: complete_job: Updating cache
[2017-06-01 19:26:23,254: INFO/job_scheduler] Job 1647 completing (errored=False, cancelled=False)
[2017-06-01 19:26:23,334: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:26:23,363: INFO/job_scheduler] Job 1650: complete_job: Updating cache
[2017-06-01 19:26:23,363: INFO/job_scheduler] Job 1650 completing (errored=False, cancelled=False)
[2017-06-01 19:26:23,442: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:26:23,557: INFO/job_scheduler] Job 1649: complete_job: Updating cache
[2017-06-01 19:26:23,558: INFO/job_scheduler] Job 1649 completing (errored=False, cancelled=False)
[2017-06-01 19:26:23,632: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:26:25,351: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:25,353: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:25,373: INFO/command_plan] add_jobs: created Job 1651 (Test for host connectivity)
[2017-06-01 19:26:25,421: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:26:25,432: INFO/job_scheduler] Job 1651: RunJobThread.run
[2017-06-01 19:26:25,527: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:25,528: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:25,542: INFO/command_plan] add_jobs: created Job 1652 (Test for host connectivity)
[2017-06-01 19:26:25,626: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:26:25,657: INFO/job_scheduler] Job 1652: RunJobThread.run
[2017-06-01 19:26:25,720: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:25,724: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:25,760: INFO/command_plan] add_jobs: created Job 1653 (Test for host connectivity)
[2017-06-01 19:26:25,791: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:26:25,833: INFO/job_scheduler] Job 1653: RunJobThread.run
[2017-06-01 19:26:25,962: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:25,963: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:25,990: INFO/command_plan] add_jobs: created Job 1654 (Test for host connectivity)
[2017-06-01 19:26:26,051: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:26:26,072: INFO/job_scheduler] Job 1654: RunJobThread.run
[2017-06-01 19:26:27,782: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:26:27,926: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:26:28,082: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:26:28,296: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:26:29,123: INFO/job_scheduler] Job 1651 finished 1 steps successfully
[2017-06-01 19:26:29,155: INFO/job_scheduler] Job 1651: complete_job: Updating cache
[2017-06-01 19:26:29,155: INFO/job_scheduler] Job 1651 completing (errored=False, cancelled=False)
[2017-06-01 19:26:29,205: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:26:29,399: INFO/job_scheduler] Job 1652 finished 1 steps successfully
[2017-06-01 19:26:29,469: INFO/job_scheduler] Job 1652: complete_job: Updating cache
[2017-06-01 19:26:29,472: INFO/job_scheduler] Job 1652 completing (errored=False, cancelled=False)
[2017-06-01 19:26:29,573: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:26:29,717: INFO/job_scheduler] Job 1653 finished 1 steps successfully
[2017-06-01 19:26:29,738: INFO/job_scheduler] Job 1653: complete_job: Updating cache
[2017-06-01 19:26:29,738: INFO/job_scheduler] Job 1653 completing (errored=False, cancelled=False)
[2017-06-01 19:26:29,780: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:26:29,900: INFO/job_scheduler] Job 1654 finished 1 steps successfully
[2017-06-01 19:26:29,971: INFO/job_scheduler] Job 1654: complete_job: Updating cache
[2017-06-01 19:26:29,971: INFO/job_scheduler] Job 1654 completing (errored=False, cancelled=False)
[2017-06-01 19:26:30,041: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:26:31,815: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:31,817: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:31,838: INFO/command_plan] add_jobs: created Job 1655 (Test for host connectivity)
[2017-06-01 19:26:31,872: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:26:31,941: INFO/job_scheduler] Job 1655: RunJobThread.run
[2017-06-01 19:26:32,032: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:32,032: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:32,048: INFO/command_plan] add_jobs: created Job 1656 (Test for host connectivity)
[2017-06-01 19:26:32,153: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:26:32,223: INFO/job_scheduler] Job 1656: RunJobThread.run
[2017-06-01 19:26:32,426: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:32,427: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:32,442: INFO/command_plan] add_jobs: created Job 1657 (Test for host connectivity)
[2017-06-01 19:26:32,569: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:26:32,583: INFO/job_scheduler] Job 1657: RunJobThread.run
[2017-06-01 19:26:32,682: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:32,683: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:32,696: INFO/command_plan] add_jobs: created Job 1658 (Test for host connectivity)
[2017-06-01 19:26:32,748: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:26:32,834: INFO/job_scheduler] Job 1658: RunJobThread.run
[2017-06-01 19:26:34,341: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:26:34,653: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:26:34,856: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:26:34,995: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:26:35,719: INFO/job_scheduler] Job 1655 finished 1 steps successfully
[2017-06-01 19:26:35,778: INFO/job_scheduler] Job 1655: complete_job: Updating cache
[2017-06-01 19:26:35,779: INFO/job_scheduler] Job 1655 completing (errored=False, cancelled=False)
[2017-06-01 19:26:35,812: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:26:36,037: INFO/job_scheduler] Job 1656 finished 1 steps successfully
[2017-06-01 19:26:36,103: INFO/job_scheduler] Job 1656: complete_job: Updating cache
[2017-06-01 19:26:36,104: INFO/job_scheduler] Job 1656 completing (errored=False, cancelled=False)
[2017-06-01 19:26:36,138: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:26:36,300: INFO/job_scheduler] Job 1657 finished 1 steps successfully
[2017-06-01 19:26:36,354: INFO/job_scheduler] Job 1657: complete_job: Updating cache
[2017-06-01 19:26:36,354: INFO/job_scheduler] Job 1657 completing (errored=False, cancelled=False)
[2017-06-01 19:26:36,390: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:26:36,463: INFO/job_scheduler] Job 1658 finished 1 steps successfully
[2017-06-01 19:26:36,499: INFO/job_scheduler] Job 1658: complete_job: Updating cache
[2017-06-01 19:26:36,500: INFO/job_scheduler] Job 1658 completing (errored=False, cancelled=False)
[2017-06-01 19:26:36,556: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:26:38,548: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:38,549: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:38,563: INFO/command_plan] add_jobs: created Job 1659 (Test for host connectivity)
[2017-06-01 19:26:38,581: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:26:38,652: INFO/job_scheduler] Job 1659: RunJobThread.run
[2017-06-01 19:26:38,705: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:38,706: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:38,744: INFO/command_plan] add_jobs: created Job 1660 (Test for host connectivity)
[2017-06-01 19:26:38,760: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:26:38,773: INFO/job_scheduler] Job 1660: RunJobThread.run
[2017-06-01 19:26:38,947: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:38,948: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:38,965: INFO/command_plan] add_jobs: created Job 1661 (Test for host connectivity)
[2017-06-01 19:26:39,049: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:26:39,084: INFO/job_scheduler] Job 1661: RunJobThread.run
[2017-06-01 19:26:39,117: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:39,119: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:39,179: INFO/command_plan] add_jobs: created Job 1662 (Test for host connectivity)
[2017-06-01 19:26:39,255: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:26:39,346: INFO/job_scheduler] Job 1662: RunJobThread.run
[2017-06-01 19:26:41,271: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:26:41,401: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:26:41,670: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:26:41,818: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:26:42,610: INFO/job_scheduler] Job 1659 finished 1 steps successfully
[2017-06-01 19:26:42,694: INFO/job_scheduler] Job 1659: complete_job: Updating cache
[2017-06-01 19:26:42,694: INFO/job_scheduler] Job 1659 completing (errored=False, cancelled=False)
[2017-06-01 19:26:42,736: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:26:42,780: INFO/job_scheduler] Job 1660 finished 1 steps successfully
[2017-06-01 19:26:42,935: INFO/job_scheduler] Job 1660: complete_job: Updating cache
[2017-06-01 19:26:42,936: INFO/job_scheduler] Job 1660 completing (errored=False, cancelled=False)
[2017-06-01 19:26:43,009: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:26:43,247: INFO/job_scheduler] Job 1661 finished 1 steps successfully
[2017-06-01 19:26:43,285: INFO/job_scheduler] Job 1661: complete_job: Updating cache
[2017-06-01 19:26:43,285: INFO/job_scheduler] Job 1661 completing (errored=False, cancelled=False)
[2017-06-01 19:26:43,336: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:26:43,375: INFO/job_scheduler] Job 1662 finished 1 steps successfully
[2017-06-01 19:26:43,427: INFO/job_scheduler] Job 1662: complete_job: Updating cache
[2017-06-01 19:26:43,427: INFO/job_scheduler] Job 1662 completing (errored=False, cancelled=False)
[2017-06-01 19:26:43,485: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:26:45,078: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:45,080: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:45,098: INFO/command_plan] add_jobs: created Job 1663 (Test for host connectivity)
[2017-06-01 19:26:45,164: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:26:45,201: INFO/job_scheduler] Job 1663: RunJobThread.run
[2017-06-01 19:26:45,240: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:45,253: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:45,277: INFO/command_plan] add_jobs: created Job 1664 (Test for host connectivity)
[2017-06-01 19:26:45,350: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:26:45,378: INFO/job_scheduler] Job 1664: RunJobThread.run
[2017-06-01 19:26:45,445: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:45,446: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:45,466: INFO/command_plan] add_jobs: created Job 1665 (Test for host connectivity)
[2017-06-01 19:26:45,520: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:26:45,558: INFO/job_scheduler] Job 1665: RunJobThread.run
[2017-06-01 19:26:45,659: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:45,661: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:45,679: INFO/command_plan] add_jobs: created Job 1666 (Test for host connectivity)
[2017-06-01 19:26:45,758: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:26:45,788: INFO/job_scheduler] Job 1666: RunJobThread.run
[2017-06-01 19:26:47,717: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:26:47,744: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:26:47,887: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:26:48,176: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:26:49,070: INFO/job_scheduler] Job 1663 finished 1 steps successfully
[2017-06-01 19:26:49,078: INFO/job_scheduler] Job 1664 finished 1 steps successfully
[2017-06-01 19:26:49,110: INFO/job_scheduler] Job 1663: complete_job: Updating cache
[2017-06-01 19:26:49,110: INFO/job_scheduler] Job 1663 completing (errored=False, cancelled=False)
[2017-06-01 19:26:49,189: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:26:49,246: INFO/job_scheduler] Job 1664: complete_job: Updating cache
[2017-06-01 19:26:49,247: INFO/job_scheduler] Job 1664 completing (errored=False, cancelled=False)
[2017-06-01 19:26:49,293: INFO/job_scheduler] Job 1665 finished 1 steps successfully
[2017-06-01 19:26:49,334: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:26:49,352: INFO/job_scheduler] Job 1665: complete_job: Updating cache
[2017-06-01 19:26:49,352: INFO/job_scheduler] Job 1665 completing (errored=False, cancelled=False)
[2017-06-01 19:26:49,463: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:26:49,577: INFO/job_scheduler] Job 1666 finished 1 steps successfully
[2017-06-01 19:26:49,601: INFO/job_scheduler] Job 1666: complete_job: Updating cache
[2017-06-01 19:26:49,601: INFO/job_scheduler] Job 1666 completing (errored=False, cancelled=False)
[2017-06-01 19:26:49,668: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:26:51,618: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:51,618: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:51,636: INFO/command_plan] add_jobs: created Job 1667 (Test for host connectivity)
[2017-06-01 19:26:51,692: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:26:51,703: INFO/job_scheduler] Job 1667: RunJobThread.run
[2017-06-01 19:26:51,757: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:51,760: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:51,776: INFO/command_plan] add_jobs: created Job 1668 (Test for host connectivity)
[2017-06-01 19:26:51,862: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:26:51,869: INFO/job_scheduler] Job 1668: RunJobThread.run
[2017-06-01 19:26:52,049: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:52,049: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:52,077: INFO/command_plan] add_jobs: created Job 1669 (Test for host connectivity)
[2017-06-01 19:26:52,129: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:26:52,164: INFO/job_scheduler] Job 1669: RunJobThread.run
[2017-06-01 19:26:52,369: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:52,381: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:52,418: INFO/command_plan] add_jobs: created Job 1670 (Test for host connectivity)
[2017-06-01 19:26:52,500: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:26:52,547: INFO/job_scheduler] Job 1670: RunJobThread.run
[2017-06-01 19:26:54,090: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:26:54,364: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:26:54,518: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:26:54,803: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:26:55,406: INFO/job_scheduler] Job 1667 finished 1 steps successfully
[2017-06-01 19:26:55,449: INFO/job_scheduler] Job 1667: complete_job: Updating cache
[2017-06-01 19:26:55,450: INFO/job_scheduler] Job 1667 completing (errored=False, cancelled=False)
[2017-06-01 19:26:55,514: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:26:55,748: INFO/job_scheduler] Job 1668 finished 1 steps successfully
[2017-06-01 19:26:55,783: INFO/job_scheduler] Job 1668: complete_job: Updating cache
[2017-06-01 19:26:55,783: INFO/job_scheduler] Job 1668 completing (errored=False, cancelled=False)
[2017-06-01 19:26:55,853: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:26:55,975: INFO/job_scheduler] Job 1669 finished 1 steps successfully
[2017-06-01 19:26:56,000: INFO/job_scheduler] Job 1669: complete_job: Updating cache
[2017-06-01 19:26:56,000: INFO/job_scheduler] Job 1669 completing (errored=False, cancelled=False)
[2017-06-01 19:26:56,093: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:26:56,155: INFO/job_scheduler] Job 1670 finished 1 steps successfully
[2017-06-01 19:26:56,341: INFO/job_scheduler] Job 1670: complete_job: Updating cache
[2017-06-01 19:26:56,341: INFO/job_scheduler] Job 1670 completing (errored=False, cancelled=False)
[2017-06-01 19:26:56,375: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:26:58,246: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:58,246: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:58,258: INFO/command_plan] add_jobs: created Job 1671 (Test for host connectivity)
[2017-06-01 19:26:58,309: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:26:58,313: INFO/job_scheduler] Job 1671: RunJobThread.run
[2017-06-01 19:26:58,376: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:58,377: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:58,416: INFO/command_plan] add_jobs: created Job 1672 (Test for host connectivity)
[2017-06-01 19:26:58,437: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:26:58,471: INFO/job_scheduler] Job 1672: RunJobThread.run
[2017-06-01 19:26:58,550: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:58,552: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:58,575: INFO/command_plan] add_jobs: created Job 1673 (Test for host connectivity)
[2017-06-01 19:26:58,606: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:26:58,638: INFO/job_scheduler] Job 1673: RunJobThread.run
[2017-06-01 19:26:58,783: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:26:58,786: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:26:58,831: INFO/command_plan] add_jobs: created Job 1674 (Test for host connectivity)
[2017-06-01 19:26:58,879: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:26:58,905: INFO/job_scheduler] Job 1674: RunJobThread.run
[2017-06-01 19:27:00,719: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:27:00,842: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:27:00,991: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:27:01,243: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:27:02,144: INFO/job_scheduler] Job 1671 finished 1 steps successfully
[2017-06-01 19:27:02,206: INFO/job_scheduler] Job 1671: complete_job: Updating cache
[2017-06-01 19:27:02,207: INFO/job_scheduler] Job 1671 completing (errored=False, cancelled=False)
[2017-06-01 19:27:02,240: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:27:02,295: INFO/job_scheduler] Job 1672 finished 1 steps successfully
[2017-06-01 19:27:02,341: INFO/job_scheduler] Job 1672: complete_job: Updating cache
[2017-06-01 19:27:02,341: INFO/job_scheduler] Job 1672 completing (errored=False, cancelled=False)
[2017-06-01 19:27:02,425: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:27:02,536: INFO/job_scheduler] Job 1673 finished 1 steps successfully
[2017-06-01 19:27:02,578: INFO/job_scheduler] Job 1673: complete_job: Updating cache
[2017-06-01 19:27:02,582: INFO/job_scheduler] Job 1673 completing (errored=False, cancelled=False)
[2017-06-01 19:27:02,632: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:27:02,814: INFO/job_scheduler] Job 1674 finished 1 steps successfully
[2017-06-01 19:27:02,876: INFO/job_scheduler] Job 1674: complete_job: Updating cache
[2017-06-01 19:27:02,876: INFO/job_scheduler] Job 1674 completing (errored=False, cancelled=False)
[2017-06-01 19:27:02,999: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:27:04,644: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:04,645: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:04,665: INFO/command_plan] add_jobs: created Job 1675 (Test for host connectivity)
[2017-06-01 19:27:04,717: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:04,718: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:04,731: INFO/command_plan] add_jobs: created Job 1676 (Test for host connectivity)
[2017-06-01 19:27:04,741: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:27:04,788: INFO/job_scheduler] Job 1675: RunJobThread.run
[2017-06-01 19:27:04,789: INFO/job_scheduler] Job 1676: RunJobThread.run
[2017-06-01 19:27:04,802: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:27:04,884: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:04,885: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:04,898: INFO/command_plan] add_jobs: created Job 1677 (Test for host connectivity)
[2017-06-01 19:27:04,995: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:27:05,005: INFO/job_scheduler] Job 1677: RunJobThread.run
[2017-06-01 19:27:05,092: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:05,093: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:05,111: INFO/command_plan] add_jobs: created Job 1678 (Test for host connectivity)
[2017-06-01 19:27:05,150: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:27:05,181: INFO/job_scheduler] Job 1678: RunJobThread.run
[2017-06-01 19:27:07,118: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:27:07,214: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:27:07,420: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:27:07,583: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:27:08,484: INFO/job_scheduler] Job 1676 finished 1 steps successfully
[2017-06-01 19:27:08,497: INFO/job_scheduler] Job 1676: complete_job: Updating cache
[2017-06-01 19:27:08,497: INFO/job_scheduler] Job 1676 completing (errored=False, cancelled=False)
[2017-06-01 19:27:08,545: INFO/job_scheduler] Job 1675 finished 1 steps successfully
[2017-06-01 19:27:08,548: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:27:08,583: INFO/job_scheduler] Job 1675: complete_job: Updating cache
[2017-06-01 19:27:08,584: INFO/job_scheduler] Job 1675 completing (errored=False, cancelled=False)
[2017-06-01 19:27:08,732: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:27:08,819: INFO/job_scheduler] Job 1677 finished 1 steps successfully
[2017-06-01 19:27:08,856: INFO/job_scheduler] Job 1677: complete_job: Updating cache
[2017-06-01 19:27:08,857: INFO/job_scheduler] Job 1677 completing (errored=False, cancelled=False)
[2017-06-01 19:27:08,930: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:27:09,024: INFO/job_scheduler] Job 1678 finished 1 steps successfully
[2017-06-01 19:27:09,081: INFO/job_scheduler] Job 1678: complete_job: Updating cache
[2017-06-01 19:27:09,082: INFO/job_scheduler] Job 1678 completing (errored=False, cancelled=False)
[2017-06-01 19:27:09,148: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:27:10,898: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:10,899: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:10,913: INFO/command_plan] add_jobs: created Job 1679 (Test for host connectivity)
[2017-06-01 19:27:10,983: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:10,983: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:11,030: INFO/command_plan] add_jobs: created Job 1680 (Test for host connectivity)
[2017-06-01 19:27:11,056: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:27:11,159: INFO/job_scheduler] Job 1680: RunJobThread.run
[2017-06-01 19:27:11,163: INFO/job_scheduler] Job 1679: RunJobThread.run
[2017-06-01 19:27:11,188: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:27:11,239: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:11,240: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:11,261: INFO/command_plan] add_jobs: created Job 1681 (Test for host connectivity)
[2017-06-01 19:27:11,376: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:27:11,403: INFO/job_scheduler] Job 1681: RunJobThread.run
[2017-06-01 19:27:11,451: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:11,452: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:11,483: INFO/command_plan] add_jobs: created Job 1682 (Test for host connectivity)
[2017-06-01 19:27:11,518: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:27:11,546: INFO/job_scheduler] Job 1682: RunJobThread.run
[2017-06-01 19:27:13,482: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:27:13,485: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:27:13,677: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:27:13,893: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:27:14,802: INFO/job_scheduler] Job 1679 finished 1 steps successfully
[2017-06-01 19:27:14,836: INFO/job_scheduler] Job 1680 finished 1 steps successfully
[2017-06-01 19:27:14,842: INFO/job_scheduler] Job 1679: complete_job: Updating cache
[2017-06-01 19:27:14,842: INFO/job_scheduler] Job 1679 completing (errored=False, cancelled=False)
[2017-06-01 19:27:14,899: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:27:14,906: INFO/job_scheduler] Job 1680: complete_job: Updating cache
[2017-06-01 19:27:14,906: INFO/job_scheduler] Job 1680 completing (errored=False, cancelled=False)
[2017-06-01 19:27:14,987: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:27:15,033: INFO/job_scheduler] Job 1681 finished 1 steps successfully
[2017-06-01 19:27:15,089: INFO/job_scheduler] Job 1681: complete_job: Updating cache
[2017-06-01 19:27:15,090: INFO/job_scheduler] Job 1681 completing (errored=False, cancelled=False)
[2017-06-01 19:27:15,164: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:27:15,277: INFO/job_scheduler] Job 1682 finished 1 steps successfully
[2017-06-01 19:27:15,323: INFO/job_scheduler] Job 1682: complete_job: Updating cache
[2017-06-01 19:27:15,325: INFO/job_scheduler] Job 1682 completing (errored=False, cancelled=False)
[2017-06-01 19:27:15,373: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:27:17,284: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:17,284: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:17,306: INFO/command_plan] add_jobs: created Job 1683 (Test for host connectivity)
[2017-06-01 19:27:17,359: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:27:17,391: INFO/job_scheduler] Job 1683: RunJobThread.run
[2017-06-01 19:27:17,445: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:17,447: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:17,463: INFO/command_plan] add_jobs: created Job 1684 (Test for host connectivity)
[2017-06-01 19:27:17,492: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:27:17,508: INFO/job_scheduler] Job 1684: RunJobThread.run
[2017-06-01 19:27:17,613: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:17,614: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:17,650: INFO/command_plan] add_jobs: created Job 1685 (Test for host connectivity)
[2017-06-01 19:27:17,709: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:27:17,727: INFO/job_scheduler] Job 1685: RunJobThread.run
[2017-06-01 19:27:17,852: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:17,852: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:17,868: INFO/command_plan] add_jobs: created Job 1686 (Test for host connectivity)
[2017-06-01 19:27:17,958: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:27:18,013: INFO/job_scheduler] Job 1686: RunJobThread.run
[2017-06-01 19:27:19,648: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:27:19,838: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:27:20,070: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:27:20,217: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:27:20,976: INFO/job_scheduler] Job 1683 finished 1 steps successfully
[2017-06-01 19:27:21,158: INFO/job_scheduler] Job 1684 finished 1 steps successfully
[2017-06-01 19:27:21,196: INFO/job_scheduler] Job 1683: complete_job: Updating cache
[2017-06-01 19:27:21,197: INFO/job_scheduler] Job 1683 completing (errored=False, cancelled=False)
[2017-06-01 19:27:21,299: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:27:21,355: INFO/job_scheduler] Job 1684: complete_job: Updating cache
[2017-06-01 19:27:21,357: INFO/job_scheduler] Job 1684 completing (errored=False, cancelled=False)
[2017-06-01 19:27:21,395: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:27:21,446: INFO/job_scheduler] Job 1685 finished 1 steps successfully
[2017-06-01 19:27:21,505: INFO/job_scheduler] Job 1685: complete_job: Updating cache
[2017-06-01 19:27:21,505: INFO/job_scheduler] Job 1685 completing (errored=False, cancelled=False)
[2017-06-01 19:27:21,557: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:27:21,691: INFO/job_scheduler] Job 1686 finished 1 steps successfully
[2017-06-01 19:27:21,713: INFO/job_scheduler] Job 1686: complete_job: Updating cache
[2017-06-01 19:27:21,714: INFO/job_scheduler] Job 1686 completing (errored=False, cancelled=False)
[2017-06-01 19:27:21,747: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:27:23,689: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:23,689: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:23,695: INFO/command_plan] add_jobs: created Job 1687 (Test for host connectivity)
[2017-06-01 19:27:23,724: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:27:23,753: INFO/job_scheduler] Job 1687: RunJobThread.run
[2017-06-01 19:27:23,897: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:23,898: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:23,913: INFO/command_plan] add_jobs: created Job 1688 (Test for host connectivity)
[2017-06-01 19:27:23,999: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:27:24,065: INFO/job_scheduler] Job 1688: RunJobThread.run
[2017-06-01 19:27:24,115: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:24,116: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:24,132: INFO/command_plan] add_jobs: created Job 1689 (Test for host connectivity)
[2017-06-01 19:27:24,159: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:27:24,253: INFO/job_scheduler] Job 1689: RunJobThread.run
[2017-06-01 19:27:24,309: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:24,314: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:24,331: INFO/command_plan] add_jobs: created Job 1690 (Test for host connectivity)
[2017-06-01 19:27:24,376: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:27:24,457: INFO/job_scheduler] Job 1690: RunJobThread.run
[2017-06-01 19:27:26,170: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:27:26,372: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:27:26,567: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:27:26,686: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:27:27,506: INFO/job_scheduler] Job 1687 finished 1 steps successfully
[2017-06-01 19:27:27,612: INFO/job_scheduler] Job 1687: complete_job: Updating cache
[2017-06-01 19:27:27,612: INFO/job_scheduler] Job 1687 completing (errored=False, cancelled=False)
[2017-06-01 19:27:27,680: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:27:27,711: INFO/job_scheduler] Job 1688 finished 1 steps successfully
[2017-06-01 19:27:27,761: INFO/job_scheduler] Job 1688: complete_job: Updating cache
[2017-06-01 19:27:27,761: INFO/job_scheduler] Job 1688 completing (errored=False, cancelled=False)
[2017-06-01 19:27:27,864: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:27:27,959: INFO/job_scheduler] Job 1689 finished 1 steps successfully
[2017-06-01 19:27:28,011: INFO/job_scheduler] Job 1689: complete_job: Updating cache
[2017-06-01 19:27:28,012: INFO/job_scheduler] Job 1689 completing (errored=False, cancelled=False)
[2017-06-01 19:27:28,071: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:27:28,106: INFO/job_scheduler] Job 1690 finished 1 steps successfully
[2017-06-01 19:27:28,175: INFO/job_scheduler] Job 1690: complete_job: Updating cache
[2017-06-01 19:27:28,176: INFO/job_scheduler] Job 1690 completing (errored=False, cancelled=False)
[2017-06-01 19:27:28,362: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:27:30,235: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:30,235: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:30,255: INFO/command_plan] add_jobs: created Job 1691 (Test for host connectivity)
[2017-06-01 19:27:30,293: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:27:30,331: INFO/job_scheduler] Job 1691: RunJobThread.run
[2017-06-01 19:27:30,427: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:30,428: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:30,451: INFO/command_plan] add_jobs: created Job 1692 (Test for host connectivity)
[2017-06-01 19:27:30,487: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:27:30,577: INFO/job_scheduler] Job 1692: RunJobThread.run
[2017-06-01 19:27:30,630: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:30,631: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:30,645: INFO/command_plan] add_jobs: created Job 1693 (Test for host connectivity)
[2017-06-01 19:27:30,695: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:27:30,697: INFO/job_scheduler] Job 1693: RunJobThread.run
[2017-06-01 19:27:30,791: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:30,791: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:30,803: INFO/command_plan] add_jobs: created Job 1694 (Test for host connectivity)
[2017-06-01 19:27:30,834: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:27:30,858: INFO/job_scheduler] Job 1694: RunJobThread.run
[2017-06-01 19:27:32,651: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:27:32,891: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:27:33,073: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:27:33,226: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:27:33,988: INFO/job_scheduler] Job 1691 finished 1 steps successfully
[2017-06-01 19:27:34,035: INFO/job_scheduler] Job 1691: complete_job: Updating cache
[2017-06-01 19:27:34,036: INFO/job_scheduler] Job 1691 completing (errored=False, cancelled=False)
[2017-06-01 19:27:34,089: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:27:34,249: INFO/job_scheduler] Job 1692 finished 1 steps successfully
[2017-06-01 19:27:34,299: INFO/job_scheduler] Job 1692: complete_job: Updating cache
[2017-06-01 19:27:34,299: INFO/job_scheduler] Job 1692 completing (errored=False, cancelled=False)
[2017-06-01 19:27:34,385: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:27:34,515: INFO/job_scheduler] Job 1693 finished 1 steps successfully
[2017-06-01 19:27:34,536: INFO/job_scheduler] Job 1693: complete_job: Updating cache
[2017-06-01 19:27:34,538: INFO/job_scheduler] Job 1693 completing (errored=False, cancelled=False)
[2017-06-01 19:27:34,627: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:27:34,630: INFO/job_scheduler] Job 1694 finished 1 steps successfully
[2017-06-01 19:27:34,645: INFO/job_scheduler] Job 1694: complete_job: Updating cache
[2017-06-01 19:27:34,645: INFO/job_scheduler] Job 1694 completing (errored=False, cancelled=False)
[2017-06-01 19:27:34,761: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:27:36,630: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:36,632: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:36,644: INFO/command_plan] add_jobs: created Job 1695 (Test for host connectivity)
[2017-06-01 19:27:36,686: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:27:36,744: INFO/job_scheduler] Job 1695: RunJobThread.run
[2017-06-01 19:27:36,802: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:36,802: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:36,818: INFO/command_plan] add_jobs: created Job 1696 (Test for host connectivity)
[2017-06-01 19:27:36,922: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:27:36,929: INFO/job_scheduler] Job 1696: RunJobThread.run
[2017-06-01 19:27:36,994: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:36,994: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:37,033: INFO/command_plan] add_jobs: created Job 1697 (Test for host connectivity)
[2017-06-01 19:27:37,093: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:27:37,119: INFO/job_scheduler] Job 1697: RunJobThread.run
[2017-06-01 19:27:37,188: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:37,189: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:37,201: INFO/command_plan] add_jobs: created Job 1698 (Test for host connectivity)
[2017-06-01 19:27:37,301: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:27:37,346: INFO/job_scheduler] Job 1698: RunJobThread.run
[2017-06-01 19:27:39,051: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:27:39,199: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:27:39,413: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:27:39,546: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:27:40,394: INFO/job_scheduler] Job 1695 finished 1 steps successfully
[2017-06-01 19:27:40,418: INFO/job_scheduler] Job 1695: complete_job: Updating cache
[2017-06-01 19:27:40,419: INFO/job_scheduler] Job 1695 completing (errored=False, cancelled=False)
[2017-06-01 19:27:40,520: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:27:40,537: INFO/job_scheduler] Job 1696 finished 1 steps successfully
[2017-06-01 19:27:40,556: INFO/job_scheduler] Job 1696: complete_job: Updating cache
[2017-06-01 19:27:40,575: INFO/job_scheduler] Job 1696 completing (errored=False, cancelled=False)
[2017-06-01 19:27:40,677: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:27:40,753: INFO/job_scheduler] Job 1697 finished 1 steps successfully
[2017-06-01 19:27:40,817: INFO/job_scheduler] Job 1697: complete_job: Updating cache
[2017-06-01 19:27:40,817: INFO/job_scheduler] Job 1697 completing (errored=False, cancelled=False)
[2017-06-01 19:27:40,851: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:27:40,958: INFO/job_scheduler] Job 1698 finished 1 steps successfully
[2017-06-01 19:27:41,009: INFO/job_scheduler] Job 1698: complete_job: Updating cache
[2017-06-01 19:27:41,013: INFO/job_scheduler] Job 1698 completing (errored=False, cancelled=False)
[2017-06-01 19:27:41,088: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:27:42,479: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:42,479: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:42,509: INFO/command_plan] add_jobs: created Job 1699 (Test for host connectivity)
[2017-06-01 19:27:42,558: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:27:42,615: INFO/job_scheduler] Job 1699: RunJobThread.run
[2017-06-01 19:27:42,670: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:42,670: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:42,713: INFO/command_plan] add_jobs: created Job 1700 (Test for host connectivity)
[2017-06-01 19:27:42,768: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:27:42,806: INFO/job_scheduler] Job 1700: RunJobThread.run
[2017-06-01 19:27:42,863: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:42,864: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:42,894: INFO/command_plan] add_jobs: created Job 1701 (Test for host connectivity)
[2017-06-01 19:27:43,076: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:43,078: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:43,100: INFO/command_plan] add_jobs: created Job 1702 (Test for host connectivity)
[2017-06-01 19:27:43,118: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:27:43,151: INFO/job_scheduler] Job 1701: RunJobThread.run
[2017-06-01 19:27:43,203: INFO/job_scheduler] Job 1702: RunJobThread.run
[2017-06-01 19:27:43,210: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:27:44,932: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:27:45,221: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:27:45,508: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:27:45,560: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:27:46,255: INFO/job_scheduler] Job 1699 finished 1 steps successfully
[2017-06-01 19:27:46,325: INFO/job_scheduler] Job 1699: complete_job: Updating cache
[2017-06-01 19:27:46,325: INFO/job_scheduler] Job 1699 completing (errored=False, cancelled=False)
[2017-06-01 19:27:46,370: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:27:46,569: INFO/job_scheduler] Job 1700 finished 1 steps successfully
[2017-06-01 19:27:46,612: INFO/job_scheduler] Job 1700: complete_job: Updating cache
[2017-06-01 19:27:46,616: INFO/job_scheduler] Job 1700 completing (errored=False, cancelled=False)
[2017-06-01 19:27:46,716: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:27:46,931: INFO/job_scheduler] Job 1702 finished 1 steps successfully
[2017-06-01 19:27:46,966: INFO/job_scheduler] Job 1702: complete_job: Updating cache
[2017-06-01 19:27:46,966: INFO/job_scheduler] Job 1702 completing (errored=False, cancelled=False)
[2017-06-01 19:27:46,995: INFO/job_scheduler] Job 1701 finished 1 steps successfully
[2017-06-01 19:27:47,000: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:27:47,042: INFO/job_scheduler] Job 1701: complete_job: Updating cache
[2017-06-01 19:27:47,042: INFO/job_scheduler] Job 1701 completing (errored=False, cancelled=False)
[2017-06-01 19:27:47,124: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:27:48,928: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:48,930: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:48,940: INFO/command_plan] add_jobs: created Job 1703 (Test for host connectivity)
[2017-06-01 19:27:49,013: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:27:49,060: INFO/job_scheduler] Job 1703: RunJobThread.run
[2017-06-01 19:27:49,090: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:49,090: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:49,108: INFO/command_plan] add_jobs: created Job 1704 (Test for host connectivity)
[2017-06-01 19:27:49,128: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:27:49,214: INFO/job_scheduler] Job 1704: RunJobThread.run
[2017-06-01 19:27:49,270: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:49,271: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:49,292: INFO/command_plan] add_jobs: created Job 1705 (Test for host connectivity)
[2017-06-01 19:27:49,324: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:27:49,337: INFO/job_scheduler] Job 1705: RunJobThread.run
[2017-06-01 19:27:49,441: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:49,443: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:49,494: INFO/command_plan] add_jobs: created Job 1706 (Test for host connectivity)
[2017-06-01 19:27:49,576: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:27:49,602: INFO/job_scheduler] Job 1706: RunJobThread.run
[2017-06-01 19:27:51,586: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:27:51,670: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:27:51,701: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:27:51,872: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:27:52,912: INFO/job_scheduler] Job 1703 finished 1 steps successfully
[2017-06-01 19:27:52,981: INFO/job_scheduler] Job 1703: complete_job: Updating cache
[2017-06-01 19:27:52,982: INFO/job_scheduler] Job 1703 completing (errored=False, cancelled=False)
[2017-06-01 19:27:53,015: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:27:53,086: INFO/job_scheduler] Job 1705 finished 1 steps successfully
[2017-06-01 19:27:53,141: INFO/job_scheduler] Job 1704 finished 1 steps successfully
[2017-06-01 19:27:53,192: INFO/job_scheduler] Job 1705: complete_job: Updating cache
[2017-06-01 19:27:53,192: INFO/job_scheduler] Job 1705 completing (errored=False, cancelled=False)
[2017-06-01 19:27:53,248: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:27:53,271: INFO/job_scheduler] Job 1704: complete_job: Updating cache
[2017-06-01 19:27:53,273: INFO/job_scheduler] Job 1704 completing (errored=False, cancelled=False)
[2017-06-01 19:27:53,303: INFO/job_scheduler] Job 1706 finished 1 steps successfully
[2017-06-01 19:27:53,357: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:27:53,369: INFO/job_scheduler] Job 1706: complete_job: Updating cache
[2017-06-01 19:27:53,369: INFO/job_scheduler] Job 1706 completing (errored=False, cancelled=False)
[2017-06-01 19:27:53,521: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:27:55,204: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:55,204: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:55,215: INFO/command_plan] add_jobs: created Job 1707 (Test for host connectivity)
[2017-06-01 19:27:55,244: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:27:55,249: INFO/job_scheduler] Job 1707: RunJobThread.run
[2017-06-01 19:27:55,393: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:55,395: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:55,414: INFO/command_plan] add_jobs: created Job 1708 (Test for host connectivity)
[2017-06-01 19:27:55,493: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:27:55,530: INFO/job_scheduler] Job 1708: RunJobThread.run
[2017-06-01 19:27:55,581: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:55,582: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:55,600: INFO/command_plan] add_jobs: created Job 1709 (Test for host connectivity)
[2017-06-01 19:27:55,651: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:27:55,677: INFO/job_scheduler] Job 1709: RunJobThread.run
[2017-06-01 19:27:55,784: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:27:55,785: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:27:55,819: INFO/command_plan] add_jobs: created Job 1710 (Test for host connectivity)
[2017-06-01 19:27:55,959: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:27:55,967: INFO/job_scheduler] Job 1710: RunJobThread.run
[2017-06-01 19:27:57,700: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:27:58,002: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:27:58,062: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:27:58,288: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:27:59,050: INFO/job_scheduler] Job 1707 finished 1 steps successfully
[2017-06-01 19:27:59,105: INFO/job_scheduler] Job 1707: complete_job: Updating cache
[2017-06-01 19:27:59,105: INFO/job_scheduler] Job 1707 completing (errored=False, cancelled=False)
[2017-06-01 19:27:59,151: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:27:59,452: INFO/job_scheduler] Job 1708 finished 1 steps successfully
[2017-06-01 19:27:59,489: INFO/job_scheduler] Job 1708: complete_job: Updating cache
[2017-06-01 19:27:59,489: INFO/job_scheduler] Job 1708 completing (errored=False, cancelled=False)
[2017-06-01 19:27:59,531: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:27:59,580: INFO/job_scheduler] Job 1709 finished 1 steps successfully
[2017-06-01 19:27:59,611: INFO/job_scheduler] Job 1709: complete_job: Updating cache
[2017-06-01 19:27:59,611: INFO/job_scheduler] Job 1709 completing (errored=False, cancelled=False)
[2017-06-01 19:27:59,641: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:27:59,697: INFO/job_scheduler] Job 1710 finished 1 steps successfully
[2017-06-01 19:27:59,714: INFO/job_scheduler] Job 1710: complete_job: Updating cache
[2017-06-01 19:27:59,714: INFO/job_scheduler] Job 1710 completing (errored=False, cancelled=False)
[2017-06-01 19:27:59,773: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:28:01,552: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:01,552: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:01,567: INFO/command_plan] add_jobs: created Job 1711 (Test for host connectivity)
[2017-06-01 19:28:01,614: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:28:01,639: INFO/job_scheduler] Job 1711: RunJobThread.run
[2017-06-01 19:28:01,729: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:01,730: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:01,746: INFO/command_plan] add_jobs: created Job 1712 (Test for host connectivity)
[2017-06-01 19:28:01,799: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:28:01,845: INFO/job_scheduler] Job 1712: RunJobThread.run
[2017-06-01 19:28:01,906: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:01,906: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:01,934: INFO/command_plan] add_jobs: created Job 1713 (Test for host connectivity)
[2017-06-01 19:28:02,024: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:28:02,042: INFO/job_scheduler] Job 1713: RunJobThread.run
[2017-06-01 19:28:02,120: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:02,121: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:02,149: INFO/command_plan] add_jobs: created Job 1714 (Test for host connectivity)
[2017-06-01 19:28:02,188: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:28:02,223: INFO/job_scheduler] Job 1714: RunJobThread.run
[2017-06-01 19:28:04,095: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:28:04,240: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:28:04,333: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:28:04,542: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:28:05,408: INFO/job_scheduler] Job 1711 finished 1 steps successfully
[2017-06-01 19:28:05,463: INFO/job_scheduler] Job 1711: complete_job: Updating cache
[2017-06-01 19:28:05,464: INFO/job_scheduler] Job 1711 completing (errored=False, cancelled=False)
[2017-06-01 19:28:05,504: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:28:05,582: INFO/job_scheduler] Job 1712 finished 1 steps successfully
[2017-06-01 19:28:05,657: INFO/job_scheduler] Job 1712: complete_job: Updating cache
[2017-06-01 19:28:05,667: INFO/job_scheduler] Job 1712 completing (errored=False, cancelled=False)
[2017-06-01 19:28:05,673: INFO/job_scheduler] Job 1713 finished 1 steps successfully
[2017-06-01 19:28:05,704: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:28:05,731: INFO/job_scheduler] Job 1713: complete_job: Updating cache
[2017-06-01 19:28:05,731: INFO/job_scheduler] Job 1713 completing (errored=False, cancelled=False)
[2017-06-01 19:28:05,788: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:28:05,973: INFO/job_scheduler] Job 1714 finished 1 steps successfully
[2017-06-01 19:28:06,029: INFO/job_scheduler] Job 1714: complete_job: Updating cache
[2017-06-01 19:28:06,029: INFO/job_scheduler] Job 1714 completing (errored=False, cancelled=False)
[2017-06-01 19:28:06,063: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:28:07,901: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:07,901: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:07,912: INFO/command_plan] add_jobs: created Job 1715 (Test for host connectivity)
[2017-06-01 19:28:08,064: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:08,064: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:08,082: INFO/command_plan] add_jobs: created Job 1716 (Test for host connectivity)
[2017-06-01 19:28:08,120: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:28:08,135: INFO/job_scheduler] Job 1715: RunJobThread.run
[2017-06-01 19:28:08,183: INFO/job_scheduler] Job 1716: RunJobThread.run
[2017-06-01 19:28:08,240: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:08,246: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:08,272: INFO/command_plan] add_jobs: created Job 1717 (Test for host connectivity)
[2017-06-01 19:28:08,300: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:28:08,370: INFO/job_scheduler] Job 1717: RunJobThread.run
[2017-06-01 19:28:08,505: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:08,506: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:08,542: INFO/command_plan] add_jobs: created Job 1718 (Test for host connectivity)
[2017-06-01 19:28:08,610: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:28:08,643: INFO/job_scheduler] Job 1718: RunJobThread.run
[2017-06-01 19:28:08,872: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:28:10,503: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:28:10,600: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:28:10,697: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:28:10,944: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:28:11,854: INFO/job_scheduler] Job 1716 finished 1 steps successfully
[2017-06-01 19:28:11,928: INFO/job_scheduler] Job 1716: complete_job: Updating cache
[2017-06-01 19:28:11,928: INFO/job_scheduler] Job 1716 completing (errored=False, cancelled=False)
[2017-06-01 19:28:11,949: INFO/job_scheduler] Job 1715 finished 1 steps successfully
[2017-06-01 19:28:11,970: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:28:11,986: INFO/job_scheduler] Job 1715: complete_job: Updating cache
[2017-06-01 19:28:11,986: INFO/job_scheduler] Job 1715 completing (errored=False, cancelled=False)
[2017-06-01 19:28:12,065: INFO/job_scheduler] Job 1717 finished 1 steps successfully
[2017-06-01 19:28:12,087: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:28:12,162: INFO/job_scheduler] Job 1717: complete_job: Updating cache
[2017-06-01 19:28:12,179: INFO/job_scheduler] Job 1717 completing (errored=False, cancelled=False)
[2017-06-01 19:28:12,252: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:28:12,296: INFO/job_scheduler] Job 1718 finished 1 steps successfully
[2017-06-01 19:28:12,352: INFO/job_scheduler] Job 1718: complete_job: Updating cache
[2017-06-01 19:28:12,361: INFO/job_scheduler] Job 1718 completing (errored=False, cancelled=False)
[2017-06-01 19:28:12,403: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:28:14,304: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:14,305: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:14,324: INFO/command_plan] add_jobs: created Job 1719 (Test for host connectivity)
[2017-06-01 19:28:14,373: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:28:14,389: INFO/job_scheduler] Job 1719: RunJobThread.run
[2017-06-01 19:28:14,505: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:14,506: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:14,527: INFO/command_plan] add_jobs: created Job 1720 (Test for host connectivity)
[2017-06-01 19:28:14,597: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:28:14,683: INFO/job_scheduler] Job 1720: RunJobThread.run
[2017-06-01 19:28:14,709: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:14,710: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:14,760: INFO/command_plan] add_jobs: created Job 1721 (Test for host connectivity)
[2017-06-01 19:28:14,783: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:28:14,812: INFO/job_scheduler] Job 1721: RunJobThread.run
[2017-06-01 19:28:14,929: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:14,929: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:14,982: INFO/command_plan] add_jobs: created Job 1722 (Test for host connectivity)
[2017-06-01 19:28:15,039: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:28:15,075: INFO/job_scheduler] Job 1722: RunJobThread.run
[2017-06-01 19:28:16,761: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:28:17,066: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:28:17,231: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:28:17,320: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:28:18,099: INFO/job_scheduler] Job 1719 finished 1 steps successfully
[2017-06-01 19:28:18,147: INFO/job_scheduler] Job 1719: complete_job: Updating cache
[2017-06-01 19:28:18,147: INFO/job_scheduler] Job 1719 completing (errored=False, cancelled=False)
[2017-06-01 19:28:18,223: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:28:18,432: INFO/job_scheduler] Job 1720 finished 1 steps successfully
[2017-06-01 19:28:18,489: INFO/job_scheduler] Job 1720: complete_job: Updating cache
[2017-06-01 19:28:18,489: INFO/job_scheduler] Job 1720 completing (errored=False, cancelled=False)
[2017-06-01 19:28:18,542: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:28:18,654: INFO/job_scheduler] Job 1721 finished 1 steps successfully
[2017-06-01 19:28:18,722: INFO/job_scheduler] Job 1721: complete_job: Updating cache
[2017-06-01 19:28:18,723: INFO/job_scheduler] Job 1721 completing (errored=False, cancelled=False)
[2017-06-01 19:28:18,789: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:28:18,833: INFO/job_scheduler] Job 1722 finished 1 steps successfully
[2017-06-01 19:28:18,881: INFO/job_scheduler] Job 1722: complete_job: Updating cache
[2017-06-01 19:28:18,887: INFO/job_scheduler] Job 1722 completing (errored=False, cancelled=False)
[2017-06-01 19:28:18,940: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:28:20,781: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:20,782: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:20,788: INFO/command_plan] add_jobs: created Job 1723 (Test for host connectivity)
[2017-06-01 19:28:20,876: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:28:20,885: INFO/job_scheduler] Job 1723: RunJobThread.run
[2017-06-01 19:28:20,958: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:20,962: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:20,993: INFO/command_plan] add_jobs: created Job 1724 (Test for host connectivity)
[2017-06-01 19:28:21,058: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:28:21,104: INFO/job_scheduler] Job 1724: RunJobThread.run
[2017-06-01 19:28:21,146: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:21,147: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:21,191: INFO/command_plan] add_jobs: created Job 1725 (Test for host connectivity)
[2017-06-01 19:28:21,213: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:28:21,227: INFO/job_scheduler] Job 1725: RunJobThread.run
[2017-06-01 19:28:21,347: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:21,348: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:21,385: INFO/command_plan] add_jobs: created Job 1726 (Test for host connectivity)
[2017-06-01 19:28:21,423: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:28:21,430: INFO/job_scheduler] Job 1726: RunJobThread.run
[2017-06-01 19:28:23,382: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:28:23,383: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:28:23,558: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:28:23,825: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:28:24,735: INFO/job_scheduler] Job 1723 finished 1 steps successfully
[2017-06-01 19:28:24,753: INFO/job_scheduler] Job 1724 finished 1 steps successfully
[2017-06-01 19:28:24,755: INFO/job_scheduler] Job 1723: complete_job: Updating cache
[2017-06-01 19:28:24,756: INFO/job_scheduler] Job 1723 completing (errored=False, cancelled=False)
[2017-06-01 19:28:24,805: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:28:24,864: INFO/job_scheduler] Job 1724: complete_job: Updating cache
[2017-06-01 19:28:24,865: INFO/job_scheduler] Job 1724 completing (errored=False, cancelled=False)
[2017-06-01 19:28:24,908: INFO/job_scheduler] Job 1725 finished 1 steps successfully
[2017-06-01 19:28:25,090: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:28:25,107: INFO/job_scheduler] Job 1725: complete_job: Updating cache
[2017-06-01 19:28:25,108: INFO/job_scheduler] Job 1725 completing (errored=False, cancelled=False)
[2017-06-01 19:28:25,205: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:28:25,322: INFO/job_scheduler] Job 1726 finished 1 steps successfully
[2017-06-01 19:28:25,371: INFO/job_scheduler] Job 1726: complete_job: Updating cache
[2017-06-01 19:28:25,372: INFO/job_scheduler] Job 1726 completing (errored=False, cancelled=False)
[2017-06-01 19:28:25,406: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:28:27,117: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:27,117: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:27,137: INFO/command_plan] add_jobs: created Job 1727 (Test for host connectivity)
[2017-06-01 19:28:27,182: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:28:27,205: INFO/job_scheduler] Job 1727: RunJobThread.run
[2017-06-01 19:28:27,278: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:27,279: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:27,324: INFO/command_plan] add_jobs: created Job 1728 (Test for host connectivity)
[2017-06-01 19:28:27,388: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:28:27,406: INFO/job_scheduler] Job 1728: RunJobThread.run
[2017-06-01 19:28:27,447: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:27,447: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:27,489: INFO/command_plan] add_jobs: created Job 1729 (Test for host connectivity)
[2017-06-01 19:28:27,556: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:28:27,591: INFO/job_scheduler] Job 1729: RunJobThread.run
[2017-06-01 19:28:27,639: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:27,641: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:27,698: INFO/command_plan] add_jobs: created Job 1730 (Test for host connectivity)
[2017-06-01 19:28:27,773: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:28:27,783: INFO/job_scheduler] Job 1730: RunJobThread.run
[2017-06-01 19:28:29,635: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:28:29,848: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:28:29,939: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:28:30,134: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:28:30,942: INFO/job_scheduler] Job 1727 finished 1 steps successfully
[2017-06-01 19:28:30,987: INFO/job_scheduler] Job 1727: complete_job: Updating cache
[2017-06-01 19:28:30,987: INFO/job_scheduler] Job 1727 completing (errored=False, cancelled=False)
[2017-06-01 19:28:31,095: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:28:31,248: INFO/job_scheduler] Job 1728 finished 1 steps successfully
[2017-06-01 19:28:31,310: INFO/job_scheduler] Job 1728: complete_job: Updating cache
[2017-06-01 19:28:31,310: INFO/job_scheduler] Job 1728 completing (errored=False, cancelled=False)
[2017-06-01 19:28:31,361: INFO/job_scheduler] Job 1729 finished 1 steps successfully
[2017-06-01 19:28:31,365: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:28:31,382: INFO/job_scheduler] Job 1729: complete_job: Updating cache
[2017-06-01 19:28:31,384: INFO/job_scheduler] Job 1729 completing (errored=False, cancelled=False)
[2017-06-01 19:28:31,453: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:28:31,517: INFO/job_scheduler] Job 1730 finished 1 steps successfully
[2017-06-01 19:28:31,571: INFO/job_scheduler] Job 1730: complete_job: Updating cache
[2017-06-01 19:28:31,572: INFO/job_scheduler] Job 1730 completing (errored=False, cancelled=False)
[2017-06-01 19:28:31,623: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:28:33,435: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:33,435: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:33,446: INFO/command_plan] add_jobs: created Job 1731 (Test for host connectivity)
[2017-06-01 19:28:33,492: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:28:33,494: INFO/job_scheduler] Job 1731: RunJobThread.run
[2017-06-01 19:28:33,532: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:33,532: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:33,547: INFO/command_plan] add_jobs: created Job 1732 (Test for host connectivity)
[2017-06-01 19:28:33,597: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:28:33,628: INFO/job_scheduler] Job 1732: RunJobThread.run
[2017-06-01 19:28:33,694: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:33,699: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:33,811: INFO/command_plan] add_jobs: created Job 1733 (Test for host connectivity)
[2017-06-01 19:28:33,845: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:28:33,858: INFO/job_scheduler] Job 1733: RunJobThread.run
[2017-06-01 19:28:33,906: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:33,907: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:33,930: INFO/command_plan] add_jobs: created Job 1734 (Test for host connectivity)
[2017-06-01 19:28:34,005: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:28:34,029: INFO/job_scheduler] Job 1734: RunJobThread.run
[2017-06-01 19:28:35,797: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:28:35,939: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:28:36,249: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:28:36,371: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:28:37,128: INFO/job_scheduler] Job 1731 finished 1 steps successfully
[2017-06-01 19:28:37,145: INFO/job_scheduler] Job 1731: complete_job: Updating cache
[2017-06-01 19:28:37,145: INFO/job_scheduler] Job 1731 completing (errored=False, cancelled=False)
[2017-06-01 19:28:37,196: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:28:37,312: INFO/job_scheduler] Job 1732 finished 1 steps successfully
[2017-06-01 19:28:37,366: INFO/job_scheduler] Job 1732: complete_job: Updating cache
[2017-06-01 19:28:37,366: INFO/job_scheduler] Job 1732 completing (errored=False, cancelled=False)
[2017-06-01 19:28:37,514: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:28:37,769: INFO/job_scheduler] Job 1733 finished 1 steps successfully
[2017-06-01 19:28:37,786: INFO/job_scheduler] Job 1733: complete_job: Updating cache
[2017-06-01 19:28:37,787: INFO/job_scheduler] Job 1733 completing (errored=False, cancelled=False)
[2017-06-01 19:28:37,814: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:28:37,823: INFO/job_scheduler] Job 1734 finished 1 steps successfully
[2017-06-01 19:28:37,858: INFO/job_scheduler] Job 1734: complete_job: Updating cache
[2017-06-01 19:28:37,858: INFO/job_scheduler] Job 1734 completing (errored=False, cancelled=False)
[2017-06-01 19:28:37,922: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:28:39,708: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:39,708: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:39,746: INFO/command_plan] add_jobs: created Job 1735 (Test for host connectivity)
[2017-06-01 19:28:39,800: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:28:39,804: INFO/job_scheduler] Job 1735: RunJobThread.run
[2017-06-01 19:28:39,894: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:39,895: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:39,903: INFO/command_plan] add_jobs: created Job 1736 (Test for host connectivity)
[2017-06-01 19:28:39,976: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:28:40,004: INFO/job_scheduler] Job 1736: RunJobThread.run
[2017-06-01 19:28:40,059: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:40,064: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:40,088: INFO/command_plan] add_jobs: created Job 1737 (Test for host connectivity)
[2017-06-01 19:28:40,168: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:28:40,185: INFO/job_scheduler] Job 1737: RunJobThread.run
[2017-06-01 19:28:40,226: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:40,226: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:40,263: INFO/command_plan] add_jobs: created Job 1738 (Test for host connectivity)
[2017-06-01 19:28:40,458: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:28:40,471: INFO/job_scheduler] Job 1738: RunJobThread.run
[2017-06-01 19:28:42,249: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:28:42,298: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:28:42,477: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:28:42,871: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:28:43,734: INFO/job_scheduler] Job 1736 finished 1 steps successfully
[2017-06-01 19:28:43,735: INFO/job_scheduler] Job 1735 finished 1 steps successfully
[2017-06-01 19:28:43,794: INFO/job_scheduler] Job 1736: complete_job: Updating cache
[2017-06-01 19:28:43,794: INFO/job_scheduler] Job 1736 completing (errored=False, cancelled=False)
[2017-06-01 19:28:43,828: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:28:43,845: INFO/job_scheduler] Job 1735: complete_job: Updating cache
[2017-06-01 19:28:43,846: INFO/job_scheduler] Job 1735 completing (errored=False, cancelled=False)
[2017-06-01 19:28:43,858: INFO/job_scheduler] Job 1737 finished 1 steps successfully
[2017-06-01 19:28:44,000: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:28:44,030: INFO/job_scheduler] Job 1737: complete_job: Updating cache
[2017-06-01 19:28:44,030: INFO/job_scheduler] Job 1737 completing (errored=False, cancelled=False)
[2017-06-01 19:28:44,112: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:28:44,298: INFO/job_scheduler] Job 1738 finished 1 steps successfully
[2017-06-01 19:28:44,336: INFO/job_scheduler] Job 1738: complete_job: Updating cache
[2017-06-01 19:28:44,336: INFO/job_scheduler] Job 1738 completing (errored=False, cancelled=False)
[2017-06-01 19:28:44,361: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:28:46,099: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:46,102: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:46,111: INFO/command_plan] add_jobs: created Job 1739 (Test for host connectivity)
[2017-06-01 19:28:46,135: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:28:46,153: INFO/job_scheduler] Job 1739: RunJobThread.run
[2017-06-01 19:28:46,236: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:46,236: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:46,244: INFO/command_plan] add_jobs: created Job 1740 (Test for host connectivity)
[2017-06-01 19:28:46,329: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:28:46,346: INFO/job_scheduler] Job 1740: RunJobThread.run
[2017-06-01 19:28:46,419: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:46,420: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:46,435: INFO/command_plan] add_jobs: created Job 1741 (Test for host connectivity)
[2017-06-01 19:28:46,598: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:46,600: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:46,610: INFO/command_plan] add_jobs: created Job 1742 (Test for host connectivity)
[2017-06-01 19:28:46,637: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:28:46,705: INFO/job_scheduler] Job 1741: RunJobThread.run
[2017-06-01 19:28:46,737: INFO/job_scheduler] Job 1742: RunJobThread.run
[2017-06-01 19:28:46,759: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:28:48,670: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:28:48,686: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:28:49,039: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:28:49,135: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:28:49,998: INFO/job_scheduler] Job 1739 finished 1 steps successfully
[2017-06-01 19:28:50,056: INFO/job_scheduler] Job 1740 finished 1 steps successfully
[2017-06-01 19:28:50,105: INFO/job_scheduler] Job 1739: complete_job: Updating cache
[2017-06-01 19:28:50,106: INFO/job_scheduler] Job 1739 completing (errored=False, cancelled=False)
[2017-06-01 19:28:50,189: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:28:50,302: INFO/job_scheduler] Job 1740: complete_job: Updating cache
[2017-06-01 19:28:50,303: INFO/job_scheduler] Job 1740 completing (errored=False, cancelled=False)
[2017-06-01 19:28:50,354: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:28:50,405: INFO/job_scheduler] Job 1741 finished 1 steps successfully
[2017-06-01 19:28:50,448: INFO/job_scheduler] Job 1741: complete_job: Updating cache
[2017-06-01 19:28:50,457: INFO/job_scheduler] Job 1741 completing (errored=False, cancelled=False)
[2017-06-01 19:28:50,510: INFO/job_scheduler] Job 1742 finished 1 steps successfully
[2017-06-01 19:28:50,553: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:28:50,578: INFO/job_scheduler] Job 1742: complete_job: Updating cache
[2017-06-01 19:28:50,579: INFO/job_scheduler] Job 1742 completing (errored=False, cancelled=False)
[2017-06-01 19:28:50,611: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:28:52,363: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:52,364: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:52,386: INFO/command_plan] add_jobs: created Job 1743 (Test for host connectivity)
[2017-06-01 19:28:52,436: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:28:52,452: INFO/job_scheduler] Job 1743: RunJobThread.run
[2017-06-01 19:28:52,543: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:52,544: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:52,560: INFO/command_plan] add_jobs: created Job 1744 (Test for host connectivity)
[2017-06-01 19:28:52,595: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:28:52,658: INFO/job_scheduler] Job 1744: RunJobThread.run
[2017-06-01 19:28:52,747: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:52,759: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:52,777: INFO/command_plan] add_jobs: created Job 1745 (Test for host connectivity)
[2017-06-01 19:28:52,839: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:52,840: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:52,862: INFO/command_plan] add_jobs: created Job 1746 (Test for host connectivity)
[2017-06-01 19:28:52,910: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:28:52,921: INFO/job_scheduler] Job 1745: RunJobThread.run
[2017-06-01 19:28:52,935: INFO/job_scheduler] Job 1746: RunJobThread.run
[2017-06-01 19:28:52,955: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:28:54,947: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:28:55,038: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:28:55,304: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:28:55,362: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:28:56,290: INFO/job_scheduler] Job 1743 finished 1 steps successfully
[2017-06-01 19:28:56,334: INFO/job_scheduler] Job 1743: complete_job: Updating cache
[2017-06-01 19:28:56,334: INFO/job_scheduler] Job 1743 completing (errored=False, cancelled=False)
[2017-06-01 19:28:56,463: INFO/job_scheduler] Job 1744 finished 1 steps successfully
[2017-06-01 19:28:56,471: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:28:56,493: INFO/job_scheduler] Job 1744: complete_job: Updating cache
[2017-06-01 19:28:56,496: INFO/job_scheduler] Job 1744 completing (errored=False, cancelled=False)
[2017-06-01 19:28:56,627: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:28:56,698: INFO/job_scheduler] Job 1746 finished 1 steps successfully
[2017-06-01 19:28:56,744: INFO/job_scheduler] Job 1746: complete_job: Updating cache
[2017-06-01 19:28:56,745: INFO/job_scheduler] Job 1746 completing (errored=False, cancelled=False)
[2017-06-01 19:28:56,801: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:28:56,857: INFO/job_scheduler] Job 1745 finished 1 steps successfully
[2017-06-01 19:28:56,882: INFO/job_scheduler] Job 1745: complete_job: Updating cache
[2017-06-01 19:28:56,882: INFO/job_scheduler] Job 1745 completing (errored=False, cancelled=False)
[2017-06-01 19:28:56,956: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:28:58,598: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:58,598: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:58,644: INFO/command_plan] add_jobs: created Job 1747 (Test for host connectivity)
[2017-06-01 19:28:58,741: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:28:58,778: INFO/job_scheduler] Job 1747: RunJobThread.run
[2017-06-01 19:28:58,828: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:58,828: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:58,944: INFO/command_plan] add_jobs: created Job 1748 (Test for host connectivity)
[2017-06-01 19:28:58,989: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:28:59,016: INFO/job_scheduler] Job 1748: RunJobThread.run
[2017-06-01 19:28:59,039: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:59,039: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:59,189: INFO/command_plan] add_jobs: created Job 1749 (Test for host connectivity)
[2017-06-01 19:28:59,220: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:28:59,235: INFO/job_scheduler] Job 1749: RunJobThread.run
[2017-06-01 19:28:59,293: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:28:59,293: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:28:59,329: INFO/command_plan] add_jobs: created Job 1750 (Test for host connectivity)
[2017-06-01 19:28:59,445: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:28:59,478: INFO/job_scheduler] Job 1750: RunJobThread.run
[2017-06-01 19:29:01,119: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:29:01,327: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:29:01,624: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:29:01,664: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:29:02,447: INFO/job_scheduler] Job 1747 finished 1 steps successfully
[2017-06-01 19:29:02,493: INFO/job_scheduler] Job 1747: complete_job: Updating cache
[2017-06-01 19:29:02,493: INFO/job_scheduler] Job 1747 completing (errored=False, cancelled=False)
[2017-06-01 19:29:02,529: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:29:02,636: INFO/job_scheduler] Job 1748 finished 1 steps successfully
[2017-06-01 19:29:02,701: INFO/job_scheduler] Job 1748: complete_job: Updating cache
[2017-06-01 19:29:02,703: INFO/job_scheduler] Job 1748 completing (errored=False, cancelled=False)
[2017-06-01 19:29:02,791: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:29:03,088: INFO/job_scheduler] Job 1749 finished 1 steps successfully
[2017-06-01 19:29:03,125: INFO/job_scheduler] Job 1749: complete_job: Updating cache
[2017-06-01 19:29:03,125: INFO/job_scheduler] Job 1749 completing (errored=False, cancelled=False)
[2017-06-01 19:29:03,144: INFO/job_scheduler] Job 1750 finished 1 steps successfully
[2017-06-01 19:29:03,184: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:29:03,202: INFO/job_scheduler] Job 1750: complete_job: Updating cache
[2017-06-01 19:29:03,203: INFO/job_scheduler] Job 1750 completing (errored=False, cancelled=False)
[2017-06-01 19:29:03,316: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:29:05,071: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:05,072: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:05,084: INFO/command_plan] add_jobs: created Job 1751 (Test for host connectivity)
[2017-06-01 19:29:05,197: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:05,197: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:05,212: INFO/command_plan] add_jobs: created Job 1752 (Test for host connectivity)
[2017-06-01 19:29:05,231: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:29:05,264: INFO/job_scheduler] Job 1752: RunJobThread.run
[2017-06-01 19:29:05,284: INFO/job_scheduler] Job 1751: RunJobThread.run
[2017-06-01 19:29:05,292: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:29:05,348: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:05,348: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:05,423: INFO/command_plan] add_jobs: created Job 1753 (Test for host connectivity)
[2017-06-01 19:29:05,552: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:29:05,572: INFO/job_scheduler] Job 1753: RunJobThread.run
[2017-06-01 19:29:05,616: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:05,617: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:05,683: INFO/command_plan] add_jobs: created Job 1754 (Test for host connectivity)
[2017-06-01 19:29:05,755: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:29:05,800: INFO/job_scheduler] Job 1754: RunJobThread.run
[2017-06-01 19:29:07,546: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:29:07,655: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:29:07,914: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:29:08,168: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:29:08,903: INFO/job_scheduler] Job 1751 finished 1 steps successfully
[2017-06-01 19:29:08,933: INFO/job_scheduler] Job 1751: complete_job: Updating cache
[2017-06-01 19:29:08,933: INFO/job_scheduler] Job 1751 completing (errored=False, cancelled=False)
[2017-06-01 19:29:08,969: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:29:09,061: INFO/job_scheduler] Job 1752 finished 1 steps successfully
[2017-06-01 19:29:09,170: INFO/job_scheduler] Job 1752: complete_job: Updating cache
[2017-06-01 19:29:09,171: INFO/job_scheduler] Job 1752 completing (errored=False, cancelled=False)
[2017-06-01 19:29:09,250: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:29:09,419: INFO/job_scheduler] Job 1753 finished 1 steps successfully
[2017-06-01 19:29:09,446: INFO/job_scheduler] Job 1753: complete_job: Updating cache
[2017-06-01 19:29:09,446: INFO/job_scheduler] Job 1753 completing (errored=False, cancelled=False)
[2017-06-01 19:29:09,474: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:29:12,006: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:12,007: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:12,031: INFO/command_plan] add_jobs: created Job 1755 (Test for host connectivity)
[2017-06-01 19:29:12,049: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:29:12,076: INFO/job_scheduler] Job 1755: RunJobThread.run
[2017-06-01 19:29:12,155: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:12,157: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:12,272: INFO/command_plan] add_jobs: created Job 1756 (Test for host connectivity)
[2017-06-01 19:29:12,324: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:29:12,352: INFO/job_scheduler] Job 1756: RunJobThread.run
[2017-06-01 19:29:12,413: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:12,425: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:12,455: INFO/command_plan] add_jobs: created Job 1757 (Test for host connectivity)
[2017-06-01 19:29:12,605: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:12,606: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:12,640: INFO/command_plan] add_jobs: created Job 1758 (Test for host connectivity)
[2017-06-01 19:29:12,670: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 3 tasked)
[2017-06-01 19:29:12,713: INFO/job_scheduler] Job 1757: RunJobThread.run
[2017-06-01 19:29:12,753: INFO/job_scheduler] Job 1758: RunJobThread.run
[2017-06-01 19:29:12,767: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 5 tasked)
[2017-06-01 19:29:13,682: INFO/job_scheduler] Job 1754 finished 1 steps successfully
[2017-06-01 19:29:13,759: INFO/job_scheduler] Job 1754: complete_job: Updating cache
[2017-06-01 19:29:13,763: INFO/job_scheduler] Job 1754 completing (errored=False, cancelled=False)
[2017-06-01 19:29:13,922: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:29:15,459: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:29:15,553: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:29:15,604: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:29:15,702: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:29:16,807: INFO/job_scheduler] Job 1755 finished 1 steps successfully
[2017-06-01 19:29:16,878: INFO/job_scheduler] Job 1755: complete_job: Updating cache
[2017-06-01 19:29:16,878: INFO/job_scheduler] Job 1755 completing (errored=False, cancelled=False)
[2017-06-01 19:29:16,913: INFO/job_scheduler] Job 1756 finished 1 steps successfully
[2017-06-01 19:29:16,921: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:29:16,933: INFO/job_scheduler] Job 1758 finished 1 steps successfully
[2017-06-01 19:29:16,939: INFO/job_scheduler] Job 1756: complete_job: Updating cache
[2017-06-01 19:29:16,940: INFO/job_scheduler] Job 1756 completing (errored=False, cancelled=False)
[2017-06-01 19:29:17,046: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:29:17,078: INFO/job_scheduler] Job 1758: complete_job: Updating cache
[2017-06-01 19:29:17,078: INFO/job_scheduler] Job 1758 completing (errored=False, cancelled=False)
[2017-06-01 19:29:17,156: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:29:17,157: INFO/job_scheduler] Job 1757 finished 1 steps successfully
[2017-06-01 19:29:17,175: INFO/job_scheduler] Job 1757: complete_job: Updating cache
[2017-06-01 19:29:17,175: INFO/job_scheduler] Job 1757 completing (errored=False, cancelled=False)
[2017-06-01 19:29:17,237: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:29:18,736: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:18,737: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:18,744: INFO/command_plan] add_jobs: created Job 1759 (Test for host connectivity)
[2017-06-01 19:29:18,757: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:29:18,775: INFO/job_scheduler] Job 1759: RunJobThread.run
[2017-06-01 19:29:18,830: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:18,831: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:18,845: INFO/command_plan] add_jobs: created Job 1760 (Test for host connectivity)
[2017-06-01 19:29:18,873: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:29:18,892: INFO/job_scheduler] Job 1760: RunJobThread.run
[2017-06-01 19:29:19,026: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:19,042: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:19,075: INFO/command_plan] add_jobs: created Job 1761 (Test for host connectivity)
[2017-06-01 19:29:19,128: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:29:19,158: INFO/job_scheduler] Job 1761: RunJobThread.run
[2017-06-01 19:29:19,217: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:19,226: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:19,245: INFO/command_plan] add_jobs: created Job 1762 (Test for host connectivity)
[2017-06-01 19:29:19,340: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:29:19,377: INFO/job_scheduler] Job 1762: RunJobThread.run
[2017-06-01 19:29:21,206: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:29:21,253: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:29:21,484: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:29:21,580: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:29:22,509: INFO/job_scheduler] Job 1759 finished 1 steps successfully
[2017-06-01 19:29:22,587: INFO/job_scheduler] Job 1759: complete_job: Updating cache
[2017-06-01 19:29:22,588: INFO/job_scheduler] Job 1759 completing (errored=False, cancelled=False)
[2017-06-01 19:29:22,644: INFO/job_scheduler] Job 1760 finished 1 steps successfully
[2017-06-01 19:29:22,645: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:29:22,715: INFO/job_scheduler] Job 1760: complete_job: Updating cache
[2017-06-01 19:29:22,715: INFO/job_scheduler] Job 1760 completing (errored=False, cancelled=False)
[2017-06-01 19:29:22,823: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:29:22,857: INFO/job_scheduler] Job 1761 finished 1 steps successfully
[2017-06-01 19:29:22,908: INFO/job_scheduler] Job 1761: complete_job: Updating cache
[2017-06-01 19:29:22,908: INFO/job_scheduler] Job 1761 completing (errored=False, cancelled=False)
[2017-06-01 19:29:22,964: INFO/job_scheduler] Job 1762 finished 1 steps successfully
[2017-06-01 19:29:22,987: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:29:23,004: INFO/job_scheduler] Job 1762: complete_job: Updating cache
[2017-06-01 19:29:23,012: INFO/job_scheduler] Job 1762 completing (errored=False, cancelled=False)
[2017-06-01 19:29:23,093: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:29:24,928: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:24,930: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:24,952: INFO/command_plan] add_jobs: created Job 1763 (Test for host connectivity)
[2017-06-01 19:29:24,969: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:29:24,978: INFO/job_scheduler] Job 1763: RunJobThread.run
[2017-06-01 19:29:25,112: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:25,114: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:25,127: INFO/command_plan] add_jobs: created Job 1764 (Test for host connectivity)
[2017-06-01 19:29:25,175: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:29:25,215: INFO/job_scheduler] Job 1764: RunJobThread.run
[2017-06-01 19:29:25,327: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:25,328: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:25,343: INFO/command_plan] add_jobs: created Job 1765 (Test for host connectivity)
[2017-06-01 19:29:25,381: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:29:25,404: INFO/job_scheduler] Job 1765: RunJobThread.run
[2017-06-01 19:29:25,545: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:25,549: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:25,568: INFO/command_plan] add_jobs: created Job 1766 (Test for host connectivity)
[2017-06-01 19:29:25,654: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:29:25,681: INFO/job_scheduler] Job 1766: RunJobThread.run
[2017-06-01 19:29:27,439: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:29:27,661: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:29:27,854: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:29:27,861: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:29:28,768: INFO/job_scheduler] Job 1763 finished 1 steps successfully
[2017-06-01 19:29:28,825: INFO/job_scheduler] Job 1763: complete_job: Updating cache
[2017-06-01 19:29:28,826: INFO/job_scheduler] Job 1763 completing (errored=False, cancelled=False)
[2017-06-01 19:29:28,893: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:29:28,997: INFO/job_scheduler] Job 1764 finished 1 steps successfully
[2017-06-01 19:29:29,064: INFO/job_scheduler] Job 1764: complete_job: Updating cache
[2017-06-01 19:29:29,065: INFO/job_scheduler] Job 1764 completing (errored=False, cancelled=False)
[2017-06-01 19:29:29,140: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:29:29,172: INFO/job_scheduler] Job 1765 finished 1 steps successfully
[2017-06-01 19:29:29,224: INFO/job_scheduler] Job 1765: complete_job: Updating cache
[2017-06-01 19:29:29,224: INFO/job_scheduler] Job 1765 completing (errored=False, cancelled=False)
[2017-06-01 19:29:29,282: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:29:29,333: INFO/job_scheduler] Job 1766 finished 1 steps successfully
[2017-06-01 19:29:29,402: INFO/job_scheduler] Job 1766: complete_job: Updating cache
[2017-06-01 19:29:29,403: INFO/job_scheduler] Job 1766 completing (errored=False, cancelled=False)
[2017-06-01 19:29:29,443: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:29:31,189: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:31,192: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:31,210: INFO/command_plan] add_jobs: created Job 1767 (Test for host connectivity)
[2017-06-01 19:29:31,299: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:29:31,301: INFO/job_scheduler] Job 1767: RunJobThread.run
[2017-06-01 19:29:31,364: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:31,365: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:31,401: INFO/command_plan] add_jobs: created Job 1768 (Test for host connectivity)
[2017-06-01 19:29:31,454: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:29:31,503: INFO/job_scheduler] Job 1768: RunJobThread.run
[2017-06-01 19:29:31,562: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:31,562: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:31,659: INFO/command_plan] add_jobs: created Job 1769 (Test for host connectivity)
[2017-06-01 19:29:31,678: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:29:31,714: INFO/job_scheduler] Job 1769: RunJobThread.run
[2017-06-01 19:29:31,799: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:31,799: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:31,817: INFO/command_plan] add_jobs: created Job 1770 (Test for host connectivity)
[2017-06-01 19:29:31,959: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:29:31,992: INFO/job_scheduler] Job 1770: RunJobThread.run
[2017-06-01 19:29:33,800: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:29:33,946: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:29:34,094: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:29:34,182: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:29:35,109: INFO/job_scheduler] Job 1767 finished 1 steps successfully
[2017-06-01 19:29:35,274: INFO/job_scheduler] Job 1767: complete_job: Updating cache
[2017-06-01 19:29:35,275: INFO/job_scheduler] Job 1767 completing (errored=False, cancelled=False)
[2017-06-01 19:29:35,323: INFO/job_scheduler] Job 1768 finished 1 steps successfully
[2017-06-01 19:29:35,341: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:29:35,391: INFO/job_scheduler] Job 1768: complete_job: Updating cache
[2017-06-01 19:29:35,391: INFO/job_scheduler] Job 1768 completing (errored=False, cancelled=False)
[2017-06-01 19:29:35,441: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:29:35,548: INFO/job_scheduler] Job 1770 finished 1 steps successfully
[2017-06-01 19:29:35,552: INFO/job_scheduler] Job 1769 finished 1 steps successfully
[2017-06-01 19:29:35,566: INFO/job_scheduler] Job 1770: complete_job: Updating cache
[2017-06-01 19:29:35,566: INFO/job_scheduler] Job 1770 completing (errored=False, cancelled=False)
[2017-06-01 19:29:35,632: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:29:35,674: INFO/job_scheduler] Job 1769: complete_job: Updating cache
[2017-06-01 19:29:35,675: INFO/job_scheduler] Job 1769 completing (errored=False, cancelled=False)
[2017-06-01 19:29:35,733: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:29:37,567: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:37,567: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:37,593: INFO/command_plan] add_jobs: created Job 1771 (Test for host connectivity)
[2017-06-01 19:29:37,652: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:29:37,678: INFO/job_scheduler] Job 1771: RunJobThread.run
[2017-06-01 19:29:37,770: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:37,772: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:37,851: INFO/command_plan] add_jobs: created Job 1772 (Test for host connectivity)
[2017-06-01 19:29:37,911: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:29:37,933: INFO/job_scheduler] Job 1772: RunJobThread.run
[2017-06-01 19:29:37,988: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:37,997: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:38,012: INFO/command_plan] add_jobs: created Job 1773 (Test for host connectivity)
[2017-06-01 19:29:38,100: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:29:38,136: INFO/job_scheduler] Job 1773: RunJobThread.run
[2017-06-01 19:29:38,199: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:38,200: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:38,209: INFO/command_plan] add_jobs: created Job 1774 (Test for host connectivity)
[2017-06-01 19:29:38,257: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:29:38,307: INFO/job_scheduler] Job 1774: RunJobThread.run
[2017-06-01 19:29:40,206: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:29:40,295: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:29:40,555: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:29:40,610: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:29:41,525: INFO/job_scheduler] Job 1771 finished 1 steps successfully
[2017-06-01 19:29:41,564: INFO/job_scheduler] Job 1771: complete_job: Updating cache
[2017-06-01 19:29:41,564: INFO/job_scheduler] Job 1771 completing (errored=False, cancelled=False)
[2017-06-01 19:29:41,618: INFO/job_scheduler] Job 1772 finished 1 steps successfully
[2017-06-01 19:29:41,663: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:29:41,682: INFO/job_scheduler] Job 1772: complete_job: Updating cache
[2017-06-01 19:29:41,704: INFO/job_scheduler] Job 1772 completing (errored=False, cancelled=False)
[2017-06-01 19:29:41,798: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:29:42,027: INFO/job_scheduler] Job 1774 finished 1 steps successfully
[2017-06-01 19:29:42,053: INFO/job_scheduler] Job 1773 finished 1 steps successfully
[2017-06-01 19:29:42,056: INFO/job_scheduler] Job 1774: complete_job: Updating cache
[2017-06-01 19:29:42,056: INFO/job_scheduler] Job 1774 completing (errored=False, cancelled=False)
[2017-06-01 19:29:42,114: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:29:42,170: INFO/job_scheduler] Job 1773: complete_job: Updating cache
[2017-06-01 19:29:42,171: INFO/job_scheduler] Job 1773 completing (errored=False, cancelled=False)
[2017-06-01 19:29:42,223: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:29:43,915: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:43,915: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:43,922: INFO/command_plan] add_jobs: created Job 1775 (Test for host connectivity)
[2017-06-01 19:29:43,951: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:29:43,991: INFO/job_scheduler] Job 1775: RunJobThread.run
[2017-06-01 19:29:44,050: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:44,051: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:44,079: INFO/command_plan] add_jobs: created Job 1776 (Test for host connectivity)
[2017-06-01 19:29:44,105: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:29:44,130: INFO/job_scheduler] Job 1776: RunJobThread.run
[2017-06-01 19:29:44,231: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:44,232: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:44,249: INFO/command_plan] add_jobs: created Job 1777 (Test for host connectivity)
[2017-06-01 19:29:44,324: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:29:44,355: INFO/job_scheduler] Job 1777: RunJobThread.run
[2017-06-01 19:29:44,396: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:44,396: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:44,423: INFO/command_plan] add_jobs: created Job 1778 (Test for host connectivity)
[2017-06-01 19:29:44,460: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:29:44,500: INFO/job_scheduler] Job 1778: RunJobThread.run
[2017-06-01 19:29:46,367: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:29:46,464: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:29:46,664: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:29:46,716: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:29:47,746: INFO/job_scheduler] Job 1775 finished 1 steps successfully
[2017-06-01 19:29:47,822: INFO/job_scheduler] Job 1775: complete_job: Updating cache
[2017-06-01 19:29:47,822: INFO/job_scheduler] Job 1775 completing (errored=False, cancelled=False)
[2017-06-01 19:29:47,856: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:29:47,880: INFO/job_scheduler] Job 1776 finished 1 steps successfully
[2017-06-01 19:29:47,921: INFO/job_scheduler] Job 1776: complete_job: Updating cache
[2017-06-01 19:29:47,921: INFO/job_scheduler] Job 1776 completing (errored=False, cancelled=False)
[2017-06-01 19:29:47,991: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:29:48,084: INFO/job_scheduler] Job 1777 finished 1 steps successfully
[2017-06-01 19:29:48,084: INFO/job_scheduler] Job 1778 finished 1 steps successfully
[2017-06-01 19:29:48,155: INFO/job_scheduler] Job 1777: complete_job: Updating cache
[2017-06-01 19:29:48,155: INFO/job_scheduler] Job 1777 completing (errored=False, cancelled=False)
[2017-06-01 19:29:48,181: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:29:48,184: INFO/job_scheduler] Job 1778: complete_job: Updating cache
[2017-06-01 19:29:48,185: INFO/job_scheduler] Job 1778 completing (errored=False, cancelled=False)
[2017-06-01 19:29:48,324: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:29:50,153: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:50,154: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:50,164: INFO/command_plan] add_jobs: created Job 1779 (Test for host connectivity)
[2017-06-01 19:29:50,241: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:29:50,247: INFO/job_scheduler] Job 1779: RunJobThread.run
[2017-06-01 19:29:50,313: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:50,317: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:50,339: INFO/command_plan] add_jobs: created Job 1780 (Test for host connectivity)
[2017-06-01 19:29:50,411: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:29:50,417: INFO/job_scheduler] Job 1780: RunJobThread.run
[2017-06-01 19:29:50,483: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:50,484: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:50,497: INFO/command_plan] add_jobs: created Job 1781 (Test for host connectivity)
[2017-06-01 19:29:50,545: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:29:50,575: INFO/job_scheduler] Job 1781: RunJobThread.run
[2017-06-01 19:29:50,727: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:50,728: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:50,749: INFO/command_plan] add_jobs: created Job 1782 (Test for host connectivity)
[2017-06-01 19:29:50,885: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:29:50,921: INFO/job_scheduler] Job 1782: RunJobThread.run
[2017-06-01 19:29:52,633: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:29:52,775: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:29:52,988: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:29:53,177: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:29:54,014: INFO/job_scheduler] Job 1779 finished 1 steps successfully
[2017-06-01 19:29:54,054: INFO/job_scheduler] Job 1779: complete_job: Updating cache
[2017-06-01 19:29:54,054: INFO/job_scheduler] Job 1779 completing (errored=False, cancelled=False)
[2017-06-01 19:29:54,093: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:29:54,223: INFO/job_scheduler] Job 1780 finished 1 steps successfully
[2017-06-01 19:29:54,262: INFO/job_scheduler] Job 1780: complete_job: Updating cache
[2017-06-01 19:29:54,264: INFO/job_scheduler] Job 1780 completing (errored=False, cancelled=False)
[2017-06-01 19:29:54,329: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:29:54,387: INFO/job_scheduler] Job 1781 finished 1 steps successfully
[2017-06-01 19:29:54,438: INFO/job_scheduler] Job 1781: complete_job: Updating cache
[2017-06-01 19:29:54,439: INFO/job_scheduler] Job 1781 completing (errored=False, cancelled=False)
[2017-06-01 19:29:54,488: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:29:54,536: INFO/job_scheduler] Job 1782 finished 1 steps successfully
[2017-06-01 19:29:54,549: INFO/job_scheduler] Job 1782: complete_job: Updating cache
[2017-06-01 19:29:54,549: INFO/job_scheduler] Job 1782 completing (errored=False, cancelled=False)
[2017-06-01 19:29:54,615: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:29:56,506: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:56,506: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:56,536: INFO/command_plan] add_jobs: created Job 1783 (Test for host connectivity)
[2017-06-01 19:29:56,589: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:29:56,608: INFO/job_scheduler] Job 1783: RunJobThread.run
[2017-06-01 19:29:56,676: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:56,677: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:56,694: INFO/command_plan] add_jobs: created Job 1784 (Test for host connectivity)
[2017-06-01 19:29:56,733: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:29:56,754: INFO/job_scheduler] Job 1784: RunJobThread.run
[2017-06-01 19:29:56,863: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:56,864: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:56,876: INFO/command_plan] add_jobs: created Job 1785 (Test for host connectivity)
[2017-06-01 19:29:56,988: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:29:57,080: INFO/job_scheduler] Job 1785: RunJobThread.run
[2017-06-01 19:29:57,124: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:29:57,124: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:29:57,169: INFO/command_plan] add_jobs: created Job 1786 (Test for host connectivity)
[2017-06-01 19:29:57,204: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:29:57,269: INFO/job_scheduler] Job 1786: RunJobThread.run
[2017-06-01 19:29:58,989: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:29:59,301: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:29:59,516: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:29:59,605: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:30:00,439: INFO/job_scheduler] Job 1783 finished 1 steps successfully
[2017-06-01 19:30:00,493: INFO/job_scheduler] Job 1783: complete_job: Updating cache
[2017-06-01 19:30:00,494: INFO/job_scheduler] Job 1783 completing (errored=False, cancelled=False)
[2017-06-01 19:30:00,554: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:30:00,680: INFO/job_scheduler] Job 1784 finished 1 steps successfully
[2017-06-01 19:30:00,737: INFO/job_scheduler] Job 1784: complete_job: Updating cache
[2017-06-01 19:30:00,737: INFO/job_scheduler] Job 1784 completing (errored=False, cancelled=False)
[2017-06-01 19:30:00,805: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:30:00,949: INFO/job_scheduler] Job 1785 finished 1 steps successfully
[2017-06-01 19:30:01,011: INFO/job_scheduler] Job 1785: complete_job: Updating cache
[2017-06-01 19:30:01,012: INFO/job_scheduler] Job 1785 completing (errored=False, cancelled=False)
[2017-06-01 19:30:01,077: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:30:01,134: INFO/job_scheduler] Job 1786 finished 1 steps successfully
[2017-06-01 19:30:01,198: INFO/job_scheduler] Job 1786: complete_job: Updating cache
[2017-06-01 19:30:01,198: INFO/job_scheduler] Job 1786 completing (errored=False, cancelled=False)
[2017-06-01 19:30:01,296: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:30:02,847: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:02,847: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:02,868: INFO/command_plan] add_jobs: created Job 1787 (Test for host connectivity)
[2017-06-01 19:30:02,935: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:02,935: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:02,947: INFO/command_plan] add_jobs: created Job 1788 (Test for host connectivity)
[2017-06-01 19:30:03,003: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:30:03,007: INFO/job_scheduler] Job 1787: RunJobThread.run
[2017-06-01 19:30:03,009: INFO/job_scheduler] Job 1788: RunJobThread.run
[2017-06-01 19:30:03,021: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:30:03,130: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:03,131: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:03,143: INFO/command_plan] add_jobs: created Job 1789 (Test for host connectivity)
[2017-06-01 19:30:03,218: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:30:03,266: INFO/job_scheduler] Job 1789: RunJobThread.run
[2017-06-01 19:30:03,334: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:03,339: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:03,375: INFO/command_plan] add_jobs: created Job 1790 (Test for host connectivity)
[2017-06-01 19:30:03,443: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:30:03,476: INFO/job_scheduler] Job 1790: RunJobThread.run
[2017-06-01 19:30:05,344: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:30:05,443: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:30:05,582: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:30:05,673: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:30:06,651: INFO/job_scheduler] Job 1787 finished 1 steps successfully
[2017-06-01 19:30:06,717: INFO/job_scheduler] Job 1787: complete_job: Updating cache
[2017-06-01 19:30:06,719: INFO/job_scheduler] Job 1787 completing (errored=False, cancelled=False)
[2017-06-01 19:30:06,790: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:30:06,852: INFO/job_scheduler] Job 1788 finished 1 steps successfully
[2017-06-01 19:30:06,921: INFO/job_scheduler] Job 1788: complete_job: Updating cache
[2017-06-01 19:30:06,921: INFO/job_scheduler] Job 1788 completing (errored=False, cancelled=False)
[2017-06-01 19:30:07,001: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:30:07,011: INFO/job_scheduler] Job 1789 finished 1 steps successfully
[2017-06-01 19:30:07,063: INFO/job_scheduler] Job 1789: complete_job: Updating cache
[2017-06-01 19:30:07,064: INFO/job_scheduler] Job 1789 completing (errored=False, cancelled=False)
[2017-06-01 19:30:07,126: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:30:07,216: INFO/job_scheduler] Job 1790 finished 1 steps successfully
[2017-06-01 19:30:07,284: INFO/job_scheduler] Job 1790: complete_job: Updating cache
[2017-06-01 19:30:07,285: INFO/job_scheduler] Job 1790 completing (errored=False, cancelled=False)
[2017-06-01 19:30:07,350: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:30:09,202: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:09,202: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:09,226: INFO/command_plan] add_jobs: created Job 1791 (Test for host connectivity)
[2017-06-01 19:30:09,291: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:30:09,318: INFO/job_scheduler] Job 1791: RunJobThread.run
[2017-06-01 19:30:09,367: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:09,368: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:09,384: INFO/command_plan] add_jobs: created Job 1792 (Test for host connectivity)
[2017-06-01 19:30:09,427: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:30:09,473: INFO/job_scheduler] Job 1792: RunJobThread.run
[2017-06-01 19:30:09,583: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:09,585: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:09,604: INFO/command_plan] add_jobs: created Job 1793 (Test for host connectivity)
[2017-06-01 19:30:09,664: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:30:09,709: INFO/job_scheduler] Job 1793: RunJobThread.run
[2017-06-01 19:30:09,835: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:09,844: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:09,873: INFO/command_plan] add_jobs: created Job 1794 (Test for host connectivity)
[2017-06-01 19:30:09,959: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:30:09,962: INFO/job_scheduler] Job 1794: RunJobThread.run
[2017-06-01 19:30:11,782: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:30:11,878: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:30:12,233: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:30:12,258: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:30:13,140: INFO/job_scheduler] Job 1791 finished 1 steps successfully
[2017-06-01 19:30:13,188: INFO/job_scheduler] Job 1792 finished 1 steps successfully
[2017-06-01 19:30:13,216: INFO/job_scheduler] Job 1791: complete_job: Updating cache
[2017-06-01 19:30:13,216: INFO/job_scheduler] Job 1791 completing (errored=False, cancelled=False)
[2017-06-01 19:30:13,324: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:30:13,340: INFO/job_scheduler] Job 1792: complete_job: Updating cache
[2017-06-01 19:30:13,344: INFO/job_scheduler] Job 1792 completing (errored=False, cancelled=False)
[2017-06-01 19:30:13,433: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:30:13,608: INFO/job_scheduler] Job 1794 finished 1 steps successfully
[2017-06-01 19:30:13,665: INFO/job_scheduler] Job 1794: complete_job: Updating cache
[2017-06-01 19:30:13,665: INFO/job_scheduler] Job 1794 completing (errored=False, cancelled=False)
[2017-06-01 19:30:13,690: INFO/job_scheduler] Job 1793 finished 1 steps successfully
[2017-06-01 19:30:13,690: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:30:13,710: INFO/job_scheduler] Job 1793: complete_job: Updating cache
[2017-06-01 19:30:13,711: INFO/job_scheduler] Job 1793 completing (errored=False, cancelled=False)
[2017-06-01 19:30:13,820: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:30:15,569: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:15,570: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:15,590: INFO/command_plan] add_jobs: created Job 1795 (Test for host connectivity)
[2017-06-01 19:30:15,641: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:30:15,707: INFO/job_scheduler] Job 1795: RunJobThread.run
[2017-06-01 19:30:15,739: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:15,739: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:15,765: INFO/command_plan] add_jobs: created Job 1796 (Test for host connectivity)
[2017-06-01 19:30:15,797: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:30:15,898: INFO/job_scheduler] Job 1796: RunJobThread.run
[2017-06-01 19:30:15,949: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:15,949: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:16,007: INFO/command_plan] add_jobs: created Job 1797 (Test for host connectivity)
[2017-06-01 19:30:16,065: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:30:16,103: INFO/job_scheduler] Job 1797: RunJobThread.run
[2017-06-01 19:30:16,148: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:16,148: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:16,172: INFO/command_plan] add_jobs: created Job 1798 (Test for host connectivity)
[2017-06-01 19:30:16,264: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:30:16,290: INFO/job_scheduler] Job 1798: RunJobThread.run
[2017-06-01 19:30:18,302: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:30:18,384: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:30:18,681: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:30:18,794: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:30:19,663: INFO/job_scheduler] Job 1795 finished 1 steps successfully
[2017-06-01 19:30:19,697: INFO/job_scheduler] Job 1795: complete_job: Updating cache
[2017-06-01 19:30:19,697: INFO/job_scheduler] Job 1795 completing (errored=False, cancelled=False)
[2017-06-01 19:30:19,753: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:30:19,756: INFO/job_scheduler] Job 1796 finished 1 steps successfully
[2017-06-01 19:30:19,839: INFO/job_scheduler] Job 1796: complete_job: Updating cache
[2017-06-01 19:30:19,840: INFO/job_scheduler] Job 1796 completing (errored=False, cancelled=False)
[2017-06-01 19:30:19,915: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:30:20,105: INFO/job_scheduler] Job 1797 finished 1 steps successfully
[2017-06-01 19:30:20,139: INFO/job_scheduler] Job 1797: complete_job: Updating cache
[2017-06-01 19:30:20,141: INFO/job_scheduler] Job 1797 completing (errored=False, cancelled=False)
[2017-06-01 19:30:20,205: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:30:20,351: INFO/job_scheduler] Job 1798 finished 1 steps successfully
[2017-06-01 19:30:20,389: INFO/job_scheduler] Job 1798: complete_job: Updating cache
[2017-06-01 19:30:20,389: INFO/job_scheduler] Job 1798 completing (errored=False, cancelled=False)
[2017-06-01 19:30:20,414: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:30:21,961: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:21,963: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:21,981: INFO/command_plan] add_jobs: created Job 1799 (Test for host connectivity)
[2017-06-01 19:30:22,039: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:30:22,054: INFO/job_scheduler] Job 1799: RunJobThread.run
[2017-06-01 19:30:22,129: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:22,131: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:22,142: INFO/command_plan] add_jobs: created Job 1800 (Test for host connectivity)
[2017-06-01 19:30:22,210: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:30:22,280: INFO/job_scheduler] Job 1800: RunJobThread.run
[2017-06-01 19:30:22,339: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:22,340: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:22,361: INFO/command_plan] add_jobs: created Job 1801 (Test for host connectivity)
[2017-06-01 19:30:22,400: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:30:22,447: INFO/job_scheduler] Job 1801: RunJobThread.run
[2017-06-01 19:30:22,537: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:22,538: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:22,552: INFO/command_plan] add_jobs: created Job 1802 (Test for host connectivity)
[2017-06-01 19:30:22,612: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:30:22,653: INFO/job_scheduler] Job 1802: RunJobThread.run
[2017-06-01 19:30:24,425: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:30:24,624: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:30:24,863: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:30:24,926: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:30:25,750: INFO/job_scheduler] Job 1799 finished 1 steps successfully
[2017-06-01 19:30:25,757: INFO/job_scheduler] Job 1799: complete_job: Updating cache
[2017-06-01 19:30:25,757: INFO/job_scheduler] Job 1799 completing (errored=False, cancelled=False)
[2017-06-01 19:30:25,816: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:30:25,952: INFO/job_scheduler] Job 1800 finished 1 steps successfully
[2017-06-01 19:30:26,015: INFO/job_scheduler] Job 1800: complete_job: Updating cache
[2017-06-01 19:30:26,016: INFO/job_scheduler] Job 1800 completing (errored=False, cancelled=False)
[2017-06-01 19:30:26,090: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:30:26,272: INFO/job_scheduler] Job 1802 finished 1 steps successfully
[2017-06-01 19:30:26,331: INFO/job_scheduler] Job 1802: complete_job: Updating cache
[2017-06-01 19:30:26,332: INFO/job_scheduler] Job 1802 completing (errored=False, cancelled=False)
[2017-06-01 19:30:26,357: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:30:26,466: INFO/job_scheduler] Job 1801 finished 1 steps successfully
[2017-06-01 19:30:26,523: INFO/job_scheduler] Job 1801: complete_job: Updating cache
[2017-06-01 19:30:26,524: INFO/job_scheduler] Job 1801 completing (errored=False, cancelled=False)
[2017-06-01 19:30:26,573: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:30:28,245: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:28,246: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:28,256: INFO/command_plan] add_jobs: created Job 1803 (Test for host connectivity)
[2017-06-01 19:30:28,318: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:30:28,351: INFO/job_scheduler] Job 1803: RunJobThread.run
[2017-06-01 19:30:28,410: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:28,414: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:28,435: INFO/command_plan] add_jobs: created Job 1804 (Test for host connectivity)
[2017-06-01 19:30:28,499: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:30:28,518: INFO/job_scheduler] Job 1804: RunJobThread.run
[2017-06-01 19:30:28,625: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:28,626: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:28,642: INFO/command_plan] add_jobs: created Job 1805 (Test for host connectivity)
[2017-06-01 19:30:28,674: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:30:28,738: INFO/job_scheduler] Job 1805: RunJobThread.run
[2017-06-01 19:30:28,809: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:28,809: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:28,823: INFO/command_plan] add_jobs: created Job 1806 (Test for host connectivity)
[2017-06-01 19:30:28,874: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:30:28,908: INFO/job_scheduler] Job 1806: RunJobThread.run
[2017-06-01 19:30:30,836: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:30:31,005: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:30:31,087: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:30:31,170: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:30:32,195: INFO/job_scheduler] Job 1803 finished 1 steps successfully
[2017-06-01 19:30:32,248: INFO/job_scheduler] Job 1803: complete_job: Updating cache
[2017-06-01 19:30:32,249: INFO/job_scheduler] Job 1803 completing (errored=False, cancelled=False)
[2017-06-01 19:30:32,309: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:30:32,349: INFO/job_scheduler] Job 1804 finished 1 steps successfully
[2017-06-01 19:30:32,422: INFO/job_scheduler] Job 1804: complete_job: Updating cache
[2017-06-01 19:30:32,423: INFO/job_scheduler] Job 1804 completing (errored=False, cancelled=False)
[2017-06-01 19:30:32,519: INFO/job_scheduler] Job 1805 finished 1 steps successfully
[2017-06-01 19:30:32,530: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:30:32,568: INFO/job_scheduler] Job 1805: complete_job: Updating cache
[2017-06-01 19:30:32,589: INFO/job_scheduler] Job 1805 completing (errored=False, cancelled=False)
[2017-06-01 19:30:32,654: INFO/job_scheduler] Job 1806 finished 1 steps successfully
[2017-06-01 19:30:32,681: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:30:32,702: INFO/job_scheduler] Job 1806: complete_job: Updating cache
[2017-06-01 19:30:32,714: INFO/job_scheduler] Job 1806 completing (errored=False, cancelled=False)
[2017-06-01 19:30:32,806: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:30:34,504: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:34,504: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:34,526: INFO/command_plan] add_jobs: created Job 1807 (Test for host connectivity)
[2017-06-01 19:30:34,582: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:30:34,627: INFO/job_scheduler] Job 1807: RunJobThread.run
[2017-06-01 19:30:34,674: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:34,678: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:34,693: INFO/command_plan] add_jobs: created Job 1808 (Test for host connectivity)
[2017-06-01 19:30:34,785: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:30:34,795: INFO/job_scheduler] Job 1808: RunJobThread.run
[2017-06-01 19:30:34,869: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:34,870: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:34,892: INFO/command_plan] add_jobs: created Job 1809 (Test for host connectivity)
[2017-06-01 19:30:34,966: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:30:35,038: INFO/job_scheduler] Job 1809: RunJobThread.run
[2017-06-01 19:30:35,090: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:35,090: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:35,114: INFO/command_plan] add_jobs: created Job 1810 (Test for host connectivity)
[2017-06-01 19:30:35,173: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:30:35,240: INFO/job_scheduler] Job 1810: RunJobThread.run
[2017-06-01 19:30:36,934: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:30:37,180: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:30:37,328: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:30:37,410: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:30:38,245: INFO/job_scheduler] Job 1807 finished 1 steps successfully
[2017-06-01 19:30:38,288: INFO/job_scheduler] Job 1807: complete_job: Updating cache
[2017-06-01 19:30:38,288: INFO/job_scheduler] Job 1807 completing (errored=False, cancelled=False)
[2017-06-01 19:30:38,382: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:30:38,593: INFO/job_scheduler] Job 1808 finished 1 steps successfully
[2017-06-01 19:30:38,654: INFO/job_scheduler] Job 1808: complete_job: Updating cache
[2017-06-01 19:30:38,654: INFO/job_scheduler] Job 1808 completing (errored=False, cancelled=False)
[2017-06-01 19:30:38,705: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:30:38,854: INFO/job_scheduler] Job 1810 finished 1 steps successfully
[2017-06-01 19:30:38,867: INFO/job_scheduler] Job 1809 finished 1 steps successfully
[2017-06-01 19:30:38,887: INFO/job_scheduler] Job 1810: complete_job: Updating cache
[2017-06-01 19:30:38,888: INFO/job_scheduler] Job 1810 completing (errored=False, cancelled=False)
[2017-06-01 19:30:38,912: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:30:38,929: INFO/job_scheduler] Job 1809: complete_job: Updating cache
[2017-06-01 19:30:38,930: INFO/job_scheduler] Job 1809 completing (errored=False, cancelled=False)
[2017-06-01 19:30:39,024: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:30:40,832: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:40,833: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:40,847: INFO/command_plan] add_jobs: created Job 1811 (Test for host connectivity)
[2017-06-01 19:30:40,895: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:30:40,912: INFO/job_scheduler] Job 1811: RunJobThread.run
[2017-06-01 19:30:41,007: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:41,007: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:41,095: INFO/command_plan] add_jobs: created Job 1812 (Test for host connectivity)
[2017-06-01 19:30:41,149: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:30:41,194: INFO/job_scheduler] Job 1812: RunJobThread.run
[2017-06-01 19:30:41,234: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:41,236: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:41,246: INFO/command_plan] add_jobs: created Job 1813 (Test for host connectivity)
[2017-06-01 19:30:41,306: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:30:41,319: INFO/job_scheduler] Job 1813: RunJobThread.run
[2017-06-01 19:30:41,457: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:41,464: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:41,470: INFO/command_plan] add_jobs: created Job 1814 (Test for host connectivity)
[2017-06-01 19:30:41,548: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:30:41,571: INFO/job_scheduler] Job 1814: RunJobThread.run
[2017-06-01 19:30:43,319: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:30:43,570: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:30:43,656: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:30:43,889: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:30:44,654: INFO/job_scheduler] Job 1811 finished 1 steps successfully
[2017-06-01 19:30:44,670: INFO/job_scheduler] Job 1811: complete_job: Updating cache
[2017-06-01 19:30:44,670: INFO/job_scheduler] Job 1811 completing (errored=False, cancelled=False)
[2017-06-01 19:30:44,761: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:30:44,940: INFO/job_scheduler] Job 1812 finished 1 steps successfully
[2017-06-01 19:30:45,004: INFO/job_scheduler] Job 1812: complete_job: Updating cache
[2017-06-01 19:30:45,005: INFO/job_scheduler] Job 1812 completing (errored=False, cancelled=False)
[2017-06-01 19:30:45,110: INFO/job_scheduler] Job 1813 finished 1 steps successfully
[2017-06-01 19:30:45,128: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:30:45,145: INFO/job_scheduler] Job 1813: complete_job: Updating cache
[2017-06-01 19:30:45,145: INFO/job_scheduler] Job 1813 completing (errored=False, cancelled=False)
[2017-06-01 19:30:45,228: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:30:45,368: INFO/job_scheduler] Job 1814 finished 1 steps successfully
[2017-06-01 19:30:45,428: INFO/job_scheduler] Job 1814: complete_job: Updating cache
[2017-06-01 19:30:45,428: INFO/job_scheduler] Job 1814 completing (errored=False, cancelled=False)
[2017-06-01 19:30:45,479: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:30:46,510: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:46,520: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:46,538: INFO/command_plan] add_jobs: created Job 1815 (Test for host connectivity)
[2017-06-01 19:30:46,555: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:30:46,567: INFO/job_scheduler] Job 1815: RunJobThread.run
[2017-06-01 19:30:46,677: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:46,677: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:46,729: INFO/command_plan] add_jobs: created Job 1816 (Test for host connectivity)
[2017-06-01 19:30:46,792: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:30:46,817: INFO/job_scheduler] Job 1816: RunJobThread.run
[2017-06-01 19:30:46,859: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:46,861: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:46,882: INFO/command_plan] add_jobs: created Job 1817 (Test for host connectivity)
[2017-06-01 19:30:46,922: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:30:46,958: INFO/job_scheduler] Job 1817: RunJobThread.run
[2017-06-01 19:30:47,091: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:47,092: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:47,139: INFO/command_plan] add_jobs: created Job 1818 (Test for host connectivity)
[2017-06-01 19:30:47,208: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:30:47,231: INFO/job_scheduler] Job 1818: RunJobThread.run
[2017-06-01 19:30:48,963: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:30:49,153: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:30:49,245: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:30:49,432: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:30:50,313: INFO/job_scheduler] Job 1815 finished 1 steps successfully
[2017-06-01 19:30:50,353: INFO/job_scheduler] Job 1815: complete_job: Updating cache
[2017-06-01 19:30:50,353: INFO/job_scheduler] Job 1815 completing (errored=False, cancelled=False)
[2017-06-01 19:30:50,423: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:30:50,552: INFO/job_scheduler] Job 1816 finished 1 steps successfully
[2017-06-01 19:30:50,610: INFO/job_scheduler] Job 1816: complete_job: Updating cache
[2017-06-01 19:30:50,610: INFO/job_scheduler] Job 1816 completing (errored=False, cancelled=False)
[2017-06-01 19:30:50,662: INFO/job_scheduler] Job 1817 finished 1 steps successfully
[2017-06-01 19:30:50,668: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:30:50,722: INFO/job_scheduler] Job 1817: complete_job: Updating cache
[2017-06-01 19:30:50,722: INFO/job_scheduler] Job 1817 completing (errored=False, cancelled=False)
[2017-06-01 19:30:50,796: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:30:50,814: INFO/job_scheduler] Job 1818 finished 1 steps successfully
[2017-06-01 19:30:50,853: INFO/job_scheduler] Job 1818: complete_job: Updating cache
[2017-06-01 19:30:50,854: INFO/job_scheduler] Job 1818 completing (errored=False, cancelled=False)
[2017-06-01 19:30:50,913: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:30:52,979: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:52,979: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:53,010: INFO/command_plan] add_jobs: created Job 1819 (Test for host connectivity)
[2017-06-01 19:30:53,033: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:30:53,052: INFO/job_scheduler] Job 1819: RunJobThread.run
[2017-06-01 19:30:53,146: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:53,153: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:53,169: INFO/command_plan] add_jobs: created Job 1820 (Test for host connectivity)
[2017-06-01 19:30:53,250: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:30:53,258: INFO/job_scheduler] Job 1820: RunJobThread.run
[2017-06-01 19:30:53,320: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:53,321: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:53,336: INFO/command_plan] add_jobs: created Job 1821 (Test for host connectivity)
[2017-06-01 19:30:53,446: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:53,446: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:53,462: INFO/command_plan] add_jobs: created Job 1822 (Test for host connectivity)
[2017-06-01 19:30:53,488: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:30:53,554: INFO/job_scheduler] Job 1821: RunJobThread.run
[2017-06-01 19:30:53,588: INFO/job_scheduler] Job 1822: RunJobThread.run
[2017-06-01 19:30:53,605: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:30:55,391: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:30:55,541: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:30:55,938: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:30:55,961: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:30:56,690: INFO/job_scheduler] Job 1819 finished 1 steps successfully
[2017-06-01 19:30:56,717: INFO/job_scheduler] Job 1819: complete_job: Updating cache
[2017-06-01 19:30:56,717: INFO/job_scheduler] Job 1819 completing (errored=False, cancelled=False)
[2017-06-01 19:30:56,753: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:30:56,891: INFO/job_scheduler] Job 1820 finished 1 steps successfully
[2017-06-01 19:30:56,919: INFO/job_scheduler] Job 1820: complete_job: Updating cache
[2017-06-01 19:30:56,919: INFO/job_scheduler] Job 1820 completing (errored=False, cancelled=False)
[2017-06-01 19:30:56,986: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:30:57,412: INFO/job_scheduler] Job 1821 finished 1 steps successfully
[2017-06-01 19:30:57,414: INFO/job_scheduler] Job 1822 finished 1 steps successfully
[2017-06-01 19:30:57,459: INFO/job_scheduler] Job 1821: complete_job: Updating cache
[2017-06-01 19:30:57,459: INFO/job_scheduler] Job 1821 completing (errored=False, cancelled=False)
[2017-06-01 19:30:57,494: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:30:57,524: INFO/job_scheduler] Job 1822: complete_job: Updating cache
[2017-06-01 19:30:57,525: INFO/job_scheduler] Job 1822 completing (errored=False, cancelled=False)
[2017-06-01 19:30:57,594: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:30:59,144: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:59,146: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:59,161: INFO/command_plan] add_jobs: created Job 1823 (Test for host connectivity)
[2017-06-01 19:30:59,216: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:30:59,222: INFO/job_scheduler] Job 1823: RunJobThread.run
[2017-06-01 19:30:59,283: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:59,283: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:59,302: INFO/command_plan] add_jobs: created Job 1824 (Test for host connectivity)
[2017-06-01 19:30:59,321: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:30:59,354: INFO/job_scheduler] Job 1824: RunJobThread.run
[2017-06-01 19:30:59,434: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:59,442: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:59,475: INFO/command_plan] add_jobs: created Job 1825 (Test for host connectivity)
[2017-06-01 19:30:59,555: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:30:59,583: INFO/job_scheduler] Job 1825: RunJobThread.run
[2017-06-01 19:30:59,640: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:30:59,642: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:30:59,661: INFO/command_plan] add_jobs: created Job 1826 (Test for host connectivity)
[2017-06-01 19:30:59,692: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:30:59,717: INFO/job_scheduler] Job 1826: RunJobThread.run
[2017-06-01 19:31:01,692: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:31:01,712: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:31:01,816: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:31:02,012: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:31:03,110: INFO/job_scheduler] Job 1823 finished 1 steps successfully
[2017-06-01 19:31:03,111: INFO/job_scheduler] Job 1824 finished 1 steps successfully
[2017-06-01 19:31:03,149: INFO/job_scheduler] Job 1823: complete_job: Updating cache
[2017-06-01 19:31:03,149: INFO/job_scheduler] Job 1823 completing (errored=False, cancelled=False)
[2017-06-01 19:31:03,184: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:31:03,188: INFO/job_scheduler] Job 1824: complete_job: Updating cache
[2017-06-01 19:31:03,188: INFO/job_scheduler] Job 1824 completing (errored=False, cancelled=False)
[2017-06-01 19:31:03,275: INFO/job_scheduler] Job 1825 finished 1 steps successfully
[2017-06-01 19:31:03,278: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:31:03,304: INFO/job_scheduler] Job 1825: complete_job: Updating cache
[2017-06-01 19:31:03,304: INFO/job_scheduler] Job 1825 completing (errored=False, cancelled=False)
[2017-06-01 19:31:03,368: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:31:03,397: INFO/job_scheduler] Job 1826 finished 1 steps successfully
[2017-06-01 19:31:03,430: INFO/job_scheduler] Job 1826: complete_job: Updating cache
[2017-06-01 19:31:03,430: INFO/job_scheduler] Job 1826 completing (errored=False, cancelled=False)
[2017-06-01 19:31:03,492: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:31:05,260: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:05,267: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:05,282: INFO/command_plan] add_jobs: created Job 1827 (Test for host connectivity)
[2017-06-01 19:31:05,310: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:31:05,333: INFO/job_scheduler] Job 1827: RunJobThread.run
[2017-06-01 19:31:05,395: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:05,395: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:05,414: INFO/command_plan] add_jobs: created Job 1828 (Test for host connectivity)
[2017-06-01 19:31:05,463: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:31:05,499: INFO/job_scheduler] Job 1828: RunJobThread.run
[2017-06-01 19:31:05,589: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:05,595: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:05,625: INFO/command_plan] add_jobs: created Job 1829 (Test for host connectivity)
[2017-06-01 19:31:05,680: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:31:05,684: INFO/job_scheduler] Job 1829: RunJobThread.run
[2017-06-01 19:31:05,779: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:05,779: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:05,793: INFO/command_plan] add_jobs: created Job 1830 (Test for host connectivity)
[2017-06-01 19:31:05,871: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:31:05,885: INFO/job_scheduler] Job 1830: RunJobThread.run
[2017-06-01 19:31:07,742: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:31:07,899: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:31:07,977: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:31:08,190: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:31:09,088: INFO/job_scheduler] Job 1827 finished 1 steps successfully
[2017-06-01 19:31:09,141: INFO/job_scheduler] Job 1827: complete_job: Updating cache
[2017-06-01 19:31:09,141: INFO/job_scheduler] Job 1827 completing (errored=False, cancelled=False)
[2017-06-01 19:31:09,248: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:31:09,255: INFO/job_scheduler] Job 1828 finished 1 steps successfully
[2017-06-01 19:31:09,282: INFO/job_scheduler] Job 1828: complete_job: Updating cache
[2017-06-01 19:31:09,283: INFO/job_scheduler] Job 1828 completing (errored=False, cancelled=False)
[2017-06-01 19:31:09,346: INFO/job_scheduler] Job 1829 finished 1 steps successfully
[2017-06-01 19:31:09,365: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:31:09,392: INFO/job_scheduler] Job 1829: complete_job: Updating cache
[2017-06-01 19:31:09,392: INFO/job_scheduler] Job 1829 completing (errored=False, cancelled=False)
[2017-06-01 19:31:09,460: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:31:09,640: INFO/job_scheduler] Job 1830 finished 1 steps successfully
[2017-06-01 19:31:09,673: INFO/job_scheduler] Job 1830: complete_job: Updating cache
[2017-06-01 19:31:09,673: INFO/job_scheduler] Job 1830 completing (errored=False, cancelled=False)
[2017-06-01 19:31:09,700: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:31:11,474: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:11,475: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:11,491: INFO/command_plan] add_jobs: created Job 1831 (Test for host connectivity)
[2017-06-01 19:31:11,526: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:31:11,597: INFO/job_scheduler] Job 1831: RunJobThread.run
[2017-06-01 19:31:11,673: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:11,675: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:11,706: INFO/command_plan] add_jobs: created Job 1832 (Test for host connectivity)
[2017-06-01 19:31:11,787: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:31:11,814: INFO/job_scheduler] Job 1832: RunJobThread.run
[2017-06-01 19:31:11,867: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:11,868: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:11,887: INFO/command_plan] add_jobs: created Job 1833 (Test for host connectivity)
[2017-06-01 19:31:11,980: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:31:11,987: INFO/job_scheduler] Job 1833: RunJobThread.run
[2017-06-01 19:31:12,063: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:12,064: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:12,078: INFO/command_plan] add_jobs: created Job 1834 (Test for host connectivity)
[2017-06-01 19:31:12,154: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:31:12,165: INFO/job_scheduler] Job 1834: RunJobThread.run
[2017-06-01 19:31:13,953: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:31:14,041: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:31:14,293: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:31:14,610: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:31:15,272: INFO/job_scheduler] Job 1831 finished 1 steps successfully
[2017-06-01 19:31:15,281: INFO/job_scheduler] Job 1831: complete_job: Updating cache
[2017-06-01 19:31:15,281: INFO/job_scheduler] Job 1831 completing (errored=False, cancelled=False)
[2017-06-01 19:31:15,354: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:31:15,362: INFO/job_scheduler] Job 1832 finished 1 steps successfully
[2017-06-01 19:31:15,391: INFO/job_scheduler] Job 1832: complete_job: Updating cache
[2017-06-01 19:31:15,391: INFO/job_scheduler] Job 1832 completing (errored=False, cancelled=False)
[2017-06-01 19:31:15,522: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:31:15,776: INFO/job_scheduler] Job 1833 finished 1 steps successfully
[2017-06-01 19:31:15,800: INFO/job_scheduler] Job 1833: complete_job: Updating cache
[2017-06-01 19:31:15,801: INFO/job_scheduler] Job 1833 completing (errored=False, cancelled=False)
[2017-06-01 19:31:15,856: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:31:16,034: INFO/job_scheduler] Job 1834 finished 1 steps successfully
[2017-06-01 19:31:16,063: INFO/job_scheduler] Job 1834: complete_job: Updating cache
[2017-06-01 19:31:16,063: INFO/job_scheduler] Job 1834 completing (errored=False, cancelled=False)
[2017-06-01 19:31:16,095: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:31:17,718: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:17,720: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:17,735: INFO/command_plan] add_jobs: created Job 1835 (Test for host connectivity)
[2017-06-01 19:31:17,832: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:31:17,839: INFO/job_scheduler] Job 1835: RunJobThread.run
[2017-06-01 19:31:17,876: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:17,876: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:17,901: INFO/command_plan] add_jobs: created Job 1836 (Test for host connectivity)
[2017-06-01 19:31:17,940: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:31:17,962: INFO/job_scheduler] Job 1836: RunJobThread.run
[2017-06-01 19:31:18,044: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:18,044: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:18,074: INFO/command_plan] add_jobs: created Job 1837 (Test for host connectivity)
[2017-06-01 19:31:18,100: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:31:18,189: INFO/job_scheduler] Job 1837: RunJobThread.run
[2017-06-01 19:31:18,256: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:18,257: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:18,292: INFO/command_plan] add_jobs: created Job 1838 (Test for host connectivity)
[2017-06-01 19:31:18,353: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:31:18,367: INFO/job_scheduler] Job 1838: RunJobThread.run
[2017-06-01 19:31:20,274: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:31:20,362: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:31:20,454: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:31:20,675: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:31:21,612: INFO/job_scheduler] Job 1835 finished 1 steps successfully
[2017-06-01 19:31:21,640: INFO/job_scheduler] Job 1835: complete_job: Updating cache
[2017-06-01 19:31:21,641: INFO/job_scheduler] Job 1835 completing (errored=False, cancelled=False)
[2017-06-01 19:31:21,677: INFO/job_scheduler] Job 1836 finished 1 steps successfully
[2017-06-01 19:31:21,748: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:31:21,791: INFO/job_scheduler] Job 1836: complete_job: Updating cache
[2017-06-01 19:31:21,791: INFO/job_scheduler] Job 1836 completing (errored=False, cancelled=False)
[2017-06-01 19:31:21,842: INFO/job_scheduler] Job 1837 finished 1 steps successfully
[2017-06-01 19:31:21,852: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:31:21,877: INFO/job_scheduler] Job 1837: complete_job: Updating cache
[2017-06-01 19:31:21,877: INFO/job_scheduler] Job 1837 completing (errored=False, cancelled=False)
[2017-06-01 19:31:21,936: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:31:22,085: INFO/job_scheduler] Job 1838 finished 1 steps successfully
[2017-06-01 19:31:22,124: INFO/job_scheduler] Job 1838: complete_job: Updating cache
[2017-06-01 19:31:22,125: INFO/job_scheduler] Job 1838 completing (errored=False, cancelled=False)
[2017-06-01 19:31:22,182: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:31:23,928: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:23,929: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:23,941: INFO/command_plan] add_jobs: created Job 1839 (Test for host connectivity)
[2017-06-01 19:31:24,007: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:24,008: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:24,054: INFO/command_plan] add_jobs: created Job 1840 (Test for host connectivity)
[2017-06-01 19:31:24,082: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:31:24,128: INFO/job_scheduler] Job 1840: RunJobThread.run
[2017-06-01 19:31:24,157: INFO/job_scheduler] Job 1839: RunJobThread.run
[2017-06-01 19:31:24,166: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:31:24,289: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:24,290: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:24,326: INFO/command_plan] add_jobs: created Job 1841 (Test for host connectivity)
[2017-06-01 19:31:24,407: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:31:24,424: INFO/job_scheduler] Job 1841: RunJobThread.run
[2017-06-01 19:31:24,468: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:24,468: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:24,499: INFO/command_plan] add_jobs: created Job 1842 (Test for host connectivity)
[2017-06-01 19:31:24,534: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:31:24,558: INFO/job_scheduler] Job 1842: RunJobThread.run
[2017-06-01 19:31:26,584: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:31:26,628: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:31:26,739: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:31:26,894: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:31:27,933: INFO/job_scheduler] Job 1840 finished 1 steps successfully
[2017-06-01 19:31:27,990: INFO/job_scheduler] Job 1840: complete_job: Updating cache
[2017-06-01 19:31:27,990: INFO/job_scheduler] Job 1840 completing (errored=False, cancelled=False)
[2017-06-01 19:31:28,011: INFO/job_scheduler] Job 1839 finished 1 steps successfully
[2017-06-01 19:31:28,131: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:31:28,171: INFO/job_scheduler] Job 1839: complete_job: Updating cache
[2017-06-01 19:31:28,171: INFO/job_scheduler] Job 1839 completing (errored=False, cancelled=False)
[2017-06-01 19:31:28,209: INFO/job_scheduler] Job 1841 finished 1 steps successfully
[2017-06-01 19:31:28,210: INFO/job_scheduler] Job 1842 finished 1 steps successfully
[2017-06-01 19:31:28,273: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:31:28,294: INFO/job_scheduler] Job 1841: complete_job: Updating cache
[2017-06-01 19:31:28,295: INFO/job_scheduler] Job 1841 completing (errored=False, cancelled=False)
[2017-06-01 19:31:28,351: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:31:28,384: INFO/job_scheduler] Job 1842: complete_job: Updating cache
[2017-06-01 19:31:28,385: INFO/job_scheduler] Job 1842 completing (errored=False, cancelled=False)
[2017-06-01 19:31:28,495: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:31:30,120: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:30,122: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:30,147: INFO/command_plan] add_jobs: created Job 1843 (Test for host connectivity)
[2017-06-01 19:31:30,282: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:31:30,315: INFO/job_scheduler] Job 1843: RunJobThread.run
[2017-06-01 19:31:30,417: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:30,428: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:30,446: INFO/command_plan] add_jobs: created Job 1844 (Test for host connectivity)
[2017-06-01 19:31:30,495: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:31:30,536: INFO/job_scheduler] Job 1844: RunJobThread.run
[2017-06-01 19:31:30,607: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:30,608: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:30,649: INFO/command_plan] add_jobs: created Job 1845 (Test for host connectivity)
[2017-06-01 19:31:30,715: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:31:30,753: INFO/job_scheduler] Job 1845: RunJobThread.run
[2017-06-01 19:31:30,816: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:30,819: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:30,845: INFO/command_plan] add_jobs: created Job 1846 (Test for host connectivity)
[2017-06-01 19:31:30,973: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:31:30,986: INFO/job_scheduler] Job 1846: RunJobThread.run
[2017-06-01 19:31:32,629: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:31:32,869: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:31:33,180: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:31:33,236: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:31:34,003: INFO/job_scheduler] Job 1843 finished 1 steps successfully
[2017-06-01 19:31:34,040: INFO/job_scheduler] Job 1843: complete_job: Updating cache
[2017-06-01 19:31:34,040: INFO/job_scheduler] Job 1843 completing (errored=False, cancelled=False)
[2017-06-01 19:31:34,090: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:31:34,188: INFO/job_scheduler] Job 1844 finished 1 steps successfully
[2017-06-01 19:31:34,239: INFO/job_scheduler] Job 1844: complete_job: Updating cache
[2017-06-01 19:31:34,240: INFO/job_scheduler] Job 1844 completing (errored=False, cancelled=False)
[2017-06-01 19:31:34,377: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:31:34,571: INFO/job_scheduler] Job 1845 finished 1 steps successfully
[2017-06-01 19:31:34,597: INFO/job_scheduler] Job 1845: complete_job: Updating cache
[2017-06-01 19:31:34,597: INFO/job_scheduler] Job 1845 completing (errored=False, cancelled=False)
[2017-06-01 19:31:34,631: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:31:34,746: INFO/job_scheduler] Job 1846 finished 1 steps successfully
[2017-06-01 19:31:34,789: INFO/job_scheduler] Job 1846: complete_job: Updating cache
[2017-06-01 19:31:34,790: INFO/job_scheduler] Job 1846 completing (errored=False, cancelled=False)
[2017-06-01 19:31:34,839: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:31:36,545: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:36,545: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:36,578: INFO/command_plan] add_jobs: created Job 1847 (Test for host connectivity)
[2017-06-01 19:31:36,607: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:31:36,609: INFO/job_scheduler] Job 1847: RunJobThread.run
[2017-06-01 19:31:36,694: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:36,695: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:36,766: INFO/command_plan] add_jobs: created Job 1848 (Test for host connectivity)
[2017-06-01 19:31:36,832: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:31:36,836: INFO/job_scheduler] Job 1848: RunJobThread.run
[2017-06-01 19:31:36,959: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:36,961: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:36,981: INFO/command_plan] add_jobs: created Job 1849 (Test for host connectivity)
[2017-06-01 19:31:37,078: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:37,079: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:37,142: INFO/command_plan] add_jobs: created Job 1850 (Test for host connectivity)
[2017-06-01 19:31:37,184: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:31:37,190: INFO/job_scheduler] Job 1849: RunJobThread.run
[2017-06-01 19:31:37,225: INFO/job_scheduler] Job 1850: RunJobThread.run
[2017-06-01 19:31:37,249: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:31:39,003: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:31:39,242: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:31:39,534: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:31:39,614: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:31:40,340: INFO/job_scheduler] Job 1847 finished 1 steps successfully
[2017-06-01 19:31:40,355: INFO/job_scheduler] Job 1847: complete_job: Updating cache
[2017-06-01 19:31:40,355: INFO/job_scheduler] Job 1847 completing (errored=False, cancelled=False)
[2017-06-01 19:31:40,423: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:31:40,580: INFO/job_scheduler] Job 1848 finished 1 steps successfully
[2017-06-01 19:31:40,639: INFO/job_scheduler] Job 1848: complete_job: Updating cache
[2017-06-01 19:31:40,639: INFO/job_scheduler] Job 1848 completing (errored=False, cancelled=False)
[2017-06-01 19:31:40,780: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:31:40,977: INFO/job_scheduler] Job 1850 finished 1 steps successfully
[2017-06-01 19:31:41,030: INFO/job_scheduler] Job 1850: complete_job: Updating cache
[2017-06-01 19:31:41,030: INFO/job_scheduler] Job 1850 completing (errored=False, cancelled=False)
[2017-06-01 19:31:41,033: INFO/job_scheduler] Job 1849 finished 1 steps successfully
[2017-06-01 19:31:41,081: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:31:41,099: INFO/job_scheduler] Job 1849: complete_job: Updating cache
[2017-06-01 19:31:41,099: INFO/job_scheduler] Job 1849 completing (errored=False, cancelled=False)
[2017-06-01 19:31:41,197: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:31:42,819: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:42,819: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:42,832: INFO/command_plan] add_jobs: created Job 1851 (Test for host connectivity)
[2017-06-01 19:31:42,943: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:31:42,980: INFO/job_scheduler] Job 1851: RunJobThread.run
[2017-06-01 19:31:43,042: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:43,042: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:43,070: INFO/command_plan] add_jobs: created Job 1852 (Test for host connectivity)
[2017-06-01 19:31:43,103: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:31:43,129: INFO/job_scheduler] Job 1852: RunJobThread.run
[2017-06-01 19:31:43,241: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:43,244: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:43,304: INFO/command_plan] add_jobs: created Job 1853 (Test for host connectivity)
[2017-06-01 19:31:43,368: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:31:43,428: INFO/job_scheduler] Job 1853: RunJobThread.run
[2017-06-01 19:31:43,496: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:43,498: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:43,539: INFO/command_plan] add_jobs: created Job 1854 (Test for host connectivity)
[2017-06-01 19:31:43,627: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:31:43,662: INFO/job_scheduler] Job 1854: RunJobThread.run
[2017-06-01 19:31:45,349: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:31:45,609: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:31:45,819: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:31:45,902: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:31:46,704: INFO/job_scheduler] Job 1851 finished 1 steps successfully
[2017-06-01 19:31:46,737: INFO/job_scheduler] Job 1851: complete_job: Updating cache
[2017-06-01 19:31:46,738: INFO/job_scheduler] Job 1851 completing (errored=False, cancelled=False)
[2017-06-01 19:31:46,816: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:31:47,069: INFO/job_scheduler] Job 1852 finished 1 steps successfully
[2017-06-01 19:31:47,132: INFO/job_scheduler] Job 1852: complete_job: Updating cache
[2017-06-01 19:31:47,133: INFO/job_scheduler] Job 1852 completing (errored=False, cancelled=False)
[2017-06-01 19:31:47,231: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:31:47,326: INFO/job_scheduler] Job 1853 finished 1 steps successfully
[2017-06-01 19:31:47,328: INFO/job_scheduler] Job 1854 finished 1 steps successfully
[2017-06-01 19:31:47,371: INFO/job_scheduler] Job 1853: complete_job: Updating cache
[2017-06-01 19:31:47,371: INFO/job_scheduler] Job 1853 completing (errored=False, cancelled=False)
[2017-06-01 19:31:47,438: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:31:47,458: INFO/job_scheduler] Job 1854: complete_job: Updating cache
[2017-06-01 19:31:47,467: INFO/job_scheduler] Job 1854 completing (errored=False, cancelled=False)
[2017-06-01 19:31:47,572: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:31:49,158: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:49,158: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:49,219: INFO/command_plan] add_jobs: created Job 1855 (Test for host connectivity)
[2017-06-01 19:31:49,363: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:49,364: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:49,370: INFO/command_plan] add_jobs: created Job 1856 (Test for host connectivity)
[2017-06-01 19:31:49,389: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:31:49,456: INFO/job_scheduler] Job 1856: RunJobThread.run
[2017-06-01 19:31:49,486: INFO/job_scheduler] Job 1855: RunJobThread.run
[2017-06-01 19:31:49,541: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:49,541: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:49,555: INFO/command_plan] add_jobs: created Job 1857 (Test for host connectivity)
[2017-06-01 19:31:49,683: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:31:49,687: INFO/job_scheduler] Job 1857: RunJobThread.run
[2017-06-01 19:31:49,805: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:49,809: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:49,849: INFO/command_plan] add_jobs: created Job 1858 (Test for host connectivity)
[2017-06-01 19:31:49,880: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:31:49,897: INFO/job_scheduler] Job 1858: RunJobThread.run
[2017-06-01 19:31:49,929: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:31:51,841: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:31:51,874: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:31:52,048: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:31:52,254: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:31:53,246: INFO/job_scheduler] Job 1856 finished 1 steps successfully
[2017-06-01 19:31:53,295: INFO/job_scheduler] Job 1856: complete_job: Updating cache
[2017-06-01 19:31:53,295: INFO/job_scheduler] Job 1856 completing (errored=False, cancelled=False)
[2017-06-01 19:31:53,329: INFO/job_scheduler] Job 1855 finished 1 steps successfully
[2017-06-01 19:31:53,364: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:31:53,398: INFO/job_scheduler] Job 1857 finished 1 steps successfully
[2017-06-01 19:31:53,399: INFO/job_scheduler] Job 1855: complete_job: Updating cache
[2017-06-01 19:31:53,399: INFO/job_scheduler] Job 1855 completing (errored=False, cancelled=False)
[2017-06-01 19:31:53,521: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:31:53,537: INFO/job_scheduler] Job 1857: complete_job: Updating cache
[2017-06-01 19:31:53,537: INFO/job_scheduler] Job 1857 completing (errored=False, cancelled=False)
[2017-06-01 19:31:53,581: INFO/job_scheduler] Job 1858 finished 1 steps successfully
[2017-06-01 19:31:53,626: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:31:53,666: INFO/job_scheduler] Job 1858: complete_job: Updating cache
[2017-06-01 19:31:53,666: INFO/job_scheduler] Job 1858 completing (errored=False, cancelled=False)
[2017-06-01 19:31:53,739: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:31:55,502: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:55,503: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:55,520: INFO/command_plan] add_jobs: created Job 1859 (Test for host connectivity)
[2017-06-01 19:31:55,637: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:31:55,642: INFO/job_scheduler] Job 1859: RunJobThread.run
[2017-06-01 19:31:55,731: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:55,731: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:55,745: INFO/command_plan] add_jobs: created Job 1860 (Test for host connectivity)
[2017-06-01 19:31:55,807: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:31:55,847: INFO/job_scheduler] Job 1860: RunJobThread.run
[2017-06-01 19:31:55,894: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:55,895: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:55,922: INFO/command_plan] add_jobs: created Job 1861 (Test for host connectivity)
[2017-06-01 19:31:55,994: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:31:56,015: INFO/job_scheduler] Job 1861: RunJobThread.run
[2017-06-01 19:31:56,097: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:31:56,098: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:31:56,112: INFO/command_plan] add_jobs: created Job 1862 (Test for host connectivity)
[2017-06-01 19:31:56,139: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:31:56,187: INFO/job_scheduler] Job 1862: RunJobThread.run
[2017-06-01 19:31:58,050: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:31:58,204: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:31:58,246: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:31:58,442: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:31:59,385: INFO/job_scheduler] Job 1859 finished 1 steps successfully
[2017-06-01 19:31:59,526: INFO/job_scheduler] Job 1859: complete_job: Updating cache
[2017-06-01 19:31:59,526: INFO/job_scheduler] Job 1859 completing (errored=False, cancelled=False)
[2017-06-01 19:31:59,585: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:31:59,594: INFO/job_scheduler] Job 1860 finished 1 steps successfully
[2017-06-01 19:31:59,614: INFO/job_scheduler] Job 1860: complete_job: Updating cache
[2017-06-01 19:31:59,614: INFO/job_scheduler] Job 1860 completing (errored=False, cancelled=False)
[2017-06-01 19:31:59,651: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:31:59,672: INFO/job_scheduler] Job 1861 finished 1 steps successfully
[2017-06-01 19:31:59,738: INFO/job_scheduler] Job 1861: complete_job: Updating cache
[2017-06-01 19:31:59,738: INFO/job_scheduler] Job 1861 completing (errored=False, cancelled=False)
[2017-06-01 19:31:59,784: INFO/job_scheduler] Job 1862 finished 1 steps successfully
[2017-06-01 19:31:59,821: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:31:59,851: INFO/job_scheduler] Job 1862: complete_job: Updating cache
[2017-06-01 19:31:59,851: INFO/job_scheduler] Job 1862 completing (errored=False, cancelled=False)
[2017-06-01 19:31:59,968: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:32:01,729: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:01,731: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:01,746: INFO/command_plan] add_jobs: created Job 1863 (Test for host connectivity)
[2017-06-01 19:32:01,787: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:32:01,792: INFO/job_scheduler] Job 1863: RunJobThread.run
[2017-06-01 19:32:01,934: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:01,934: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:01,987: INFO/command_plan] add_jobs: created Job 1864 (Test for host connectivity)
[2017-06-01 19:32:02,016: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:32:02,024: INFO/job_scheduler] Job 1864: RunJobThread.run
[2017-06-01 19:32:02,137: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:02,138: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:02,177: INFO/command_plan] add_jobs: created Job 1865 (Test for host connectivity)
[2017-06-01 19:32:02,263: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:32:02,305: INFO/job_scheduler] Job 1865: RunJobThread.run
[2017-06-01 19:32:02,321: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:02,321: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:02,344: INFO/command_plan] add_jobs: created Job 1866 (Test for host connectivity)
[2017-06-01 19:32:02,387: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:32:02,438: INFO/job_scheduler] Job 1866: RunJobThread.run
[2017-06-01 19:32:04,112: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:32:04,389: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:32:04,618: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:32:04,730: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:32:05,439: INFO/job_scheduler] Job 1863 finished 1 steps successfully
[2017-06-01 19:32:05,500: INFO/job_scheduler] Job 1863: complete_job: Updating cache
[2017-06-01 19:32:05,500: INFO/job_scheduler] Job 1863 completing (errored=False, cancelled=False)
[2017-06-01 19:32:05,550: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:32:05,719: INFO/job_scheduler] Job 1864 finished 1 steps successfully
[2017-06-01 19:32:05,800: INFO/job_scheduler] Job 1864: complete_job: Updating cache
[2017-06-01 19:32:05,801: INFO/job_scheduler] Job 1864 completing (errored=False, cancelled=False)
[2017-06-01 19:32:05,917: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:32:06,168: INFO/job_scheduler] Job 1865 finished 1 steps successfully
[2017-06-01 19:32:06,169: INFO/job_scheduler] Job 1866 finished 1 steps successfully
[2017-06-01 19:32:06,208: INFO/job_scheduler] Job 1865: complete_job: Updating cache
[2017-06-01 19:32:06,208: INFO/job_scheduler] Job 1865 completing (errored=False, cancelled=False)
[2017-06-01 19:32:06,239: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:32:06,258: INFO/job_scheduler] Job 1866: complete_job: Updating cache
[2017-06-01 19:32:06,258: INFO/job_scheduler] Job 1866 completing (errored=False, cancelled=False)
[2017-06-01 19:32:06,303: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:32:08,021: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:08,022: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:08,049: INFO/command_plan] add_jobs: created Job 1867 (Test for host connectivity)
[2017-06-01 19:32:08,076: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:32:08,107: INFO/job_scheduler] Job 1867: RunJobThread.run
[2017-06-01 19:32:08,195: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:08,196: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:08,216: INFO/command_plan] add_jobs: created Job 1868 (Test for host connectivity)
[2017-06-01 19:32:08,282: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:32:08,325: INFO/job_scheduler] Job 1868: RunJobThread.run
[2017-06-01 19:32:08,362: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:08,363: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:08,383: INFO/command_plan] add_jobs: created Job 1869 (Test for host connectivity)
[2017-06-01 19:32:08,453: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:32:08,462: INFO/job_scheduler] Job 1869: RunJobThread.run
[2017-06-01 19:32:08,577: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:08,577: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:08,600: INFO/command_plan] add_jobs: created Job 1870 (Test for host connectivity)
[2017-06-01 19:32:08,666: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:32:08,695: INFO/job_scheduler] Job 1870: RunJobThread.run
[2017-06-01 19:32:10,449: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:32:10,676: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:32:10,824: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:32:10,946: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:32:11,777: INFO/job_scheduler] Job 1867 finished 1 steps successfully
[2017-06-01 19:32:11,819: INFO/job_scheduler] Job 1867: complete_job: Updating cache
[2017-06-01 19:32:11,819: INFO/job_scheduler] Job 1867 completing (errored=False, cancelled=False)
[2017-06-01 19:32:11,901: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:32:11,948: INFO/job_scheduler] Job 1868 finished 1 steps successfully
[2017-06-01 19:32:12,010: INFO/job_scheduler] Job 1868: complete_job: Updating cache
[2017-06-01 19:32:12,010: INFO/job_scheduler] Job 1868 completing (errored=False, cancelled=False)
[2017-06-01 19:32:12,049: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:32:12,220: INFO/job_scheduler] Job 1869 finished 1 steps successfully
[2017-06-01 19:32:12,265: INFO/job_scheduler] Job 1869: complete_job: Updating cache
[2017-06-01 19:32:12,265: INFO/job_scheduler] Job 1869 completing (errored=False, cancelled=False)
[2017-06-01 19:32:12,333: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:32:12,498: INFO/job_scheduler] Job 1870 finished 1 steps successfully
[2017-06-01 19:32:12,556: INFO/job_scheduler] Job 1870: complete_job: Updating cache
[2017-06-01 19:32:12,557: INFO/job_scheduler] Job 1870 completing (errored=False, cancelled=False)
[2017-06-01 19:32:12,586: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:32:14,934: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:14,935: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:14,949: INFO/command_plan] add_jobs: created Job 1871 (Test for host connectivity)
[2017-06-01 19:32:15,072: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:15,073: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:15,090: INFO/command_plan] add_jobs: created Job 1872 (Test for host connectivity)
[2017-06-01 19:32:15,136: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:32:15,143: INFO/job_scheduler] Job 1872: RunJobThread.run
[2017-06-01 19:32:15,144: INFO/job_scheduler] Job 1871: RunJobThread.run
[2017-06-01 19:32:15,148: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:32:15,292: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:15,293: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:15,314: INFO/command_plan] add_jobs: created Job 1873 (Test for host connectivity)
[2017-06-01 19:32:15,404: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:32:15,431: INFO/job_scheduler] Job 1873: RunJobThread.run
[2017-06-01 19:32:15,515: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:15,517: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:15,531: INFO/command_plan] add_jobs: created Job 1874 (Test for host connectivity)
[2017-06-01 19:32:15,607: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:32:15,642: INFO/job_scheduler] Job 1874: RunJobThread.run
[2017-06-01 19:32:17,578: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:32:17,643: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:32:17,794: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:32:17,979: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:32:18,915: INFO/job_scheduler] Job 1871 finished 1 steps successfully
[2017-06-01 19:32:18,932: INFO/job_scheduler] Job 1871: complete_job: Updating cache
[2017-06-01 19:32:18,932: INFO/job_scheduler] Job 1871 completing (errored=False, cancelled=False)
[2017-06-01 19:32:18,990: INFO/job_scheduler] Job 1872 finished 1 steps successfully
[2017-06-01 19:32:19,015: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:32:19,049: INFO/job_scheduler] Job 1872: complete_job: Updating cache
[2017-06-01 19:32:19,053: INFO/job_scheduler] Job 1872 completing (errored=False, cancelled=False)
[2017-06-01 19:32:19,207: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:32:19,346: INFO/job_scheduler] Job 1873 finished 1 steps successfully
[2017-06-01 19:32:19,388: INFO/job_scheduler] Job 1873: complete_job: Updating cache
[2017-06-01 19:32:19,389: INFO/job_scheduler] Job 1873 completing (errored=False, cancelled=False)
[2017-06-01 19:32:19,451: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:32:19,484: INFO/job_scheduler] Job 1874 finished 1 steps successfully
[2017-06-01 19:32:19,506: INFO/job_scheduler] Job 1874: complete_job: Updating cache
[2017-06-01 19:32:19,507: INFO/job_scheduler] Job 1874 completing (errored=False, cancelled=False)
[2017-06-01 19:32:19,591: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:32:21,319: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:21,320: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:21,331: INFO/command_plan] add_jobs: created Job 1875 (Test for host connectivity)
[2017-06-01 19:32:21,367: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:32:21,370: INFO/job_scheduler] Job 1875: RunJobThread.run
[2017-06-01 19:32:21,476: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:21,479: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:21,488: INFO/command_plan] add_jobs: created Job 1876 (Test for host connectivity)
[2017-06-01 19:32:21,524: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:32:21,531: INFO/job_scheduler] Job 1876: RunJobThread.run
[2017-06-01 19:32:21,648: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:21,649: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:21,665: INFO/command_plan] add_jobs: created Job 1877 (Test for host connectivity)
[2017-06-01 19:32:21,726: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:32:21,748: INFO/job_scheduler] Job 1877: RunJobThread.run
[2017-06-01 19:32:21,855: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:21,856: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:21,872: INFO/command_plan] add_jobs: created Job 1878 (Test for host connectivity)
[2017-06-01 19:32:21,978: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:32:22,023: INFO/job_scheduler] Job 1878: RunJobThread.run
[2017-06-01 19:32:23,852: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:32:23,972: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:32:24,229: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:32:24,232: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:32:25,201: INFO/job_scheduler] Job 1875 finished 1 steps successfully
[2017-06-01 19:32:25,248: INFO/job_scheduler] Job 1875: complete_job: Updating cache
[2017-06-01 19:32:25,248: INFO/job_scheduler] Job 1875 completing (errored=False, cancelled=False)
[2017-06-01 19:32:25,313: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:32:25,332: INFO/job_scheduler] Job 1876 finished 1 steps successfully
[2017-06-01 19:32:25,407: INFO/job_scheduler] Job 1876: complete_job: Updating cache
[2017-06-01 19:32:25,407: INFO/job_scheduler] Job 1876 completing (errored=False, cancelled=False)
[2017-06-01 19:32:25,497: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:32:25,612: INFO/job_scheduler] Job 1878 finished 1 steps successfully
[2017-06-01 19:32:25,648: INFO/job_scheduler] Job 1878: complete_job: Updating cache
[2017-06-01 19:32:25,649: INFO/job_scheduler] Job 1878 completing (errored=False, cancelled=False)
[2017-06-01 19:32:25,722: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:32:25,772: INFO/job_scheduler] Job 1877 finished 1 steps successfully
[2017-06-01 19:32:25,780: INFO/job_scheduler] Job 1877: complete_job: Updating cache
[2017-06-01 19:32:25,780: INFO/job_scheduler] Job 1877 completing (errored=False, cancelled=False)
[2017-06-01 19:32:25,854: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:32:27,700: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:27,701: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:27,720: INFO/command_plan] add_jobs: created Job 1879 (Test for host connectivity)
[2017-06-01 19:32:27,782: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:32:27,821: INFO/job_scheduler] Job 1879: RunJobThread.run
[2017-06-01 19:32:27,864: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:27,866: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:27,879: INFO/command_plan] add_jobs: created Job 1880 (Test for host connectivity)
[2017-06-01 19:32:27,953: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:32:28,002: INFO/job_scheduler] Job 1880: RunJobThread.run
[2017-06-01 19:32:28,064: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:28,064: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:28,085: INFO/command_plan] add_jobs: created Job 1881 (Test for host connectivity)
[2017-06-01 19:32:28,174: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:32:28,187: INFO/job_scheduler] Job 1881: RunJobThread.run
[2017-06-01 19:32:28,276: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:28,277: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:28,292: INFO/command_plan] add_jobs: created Job 1882 (Test for host connectivity)
[2017-06-01 19:32:28,387: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:32:28,402: INFO/job_scheduler] Job 1882: RunJobThread.run
[2017-06-01 19:32:30,257: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:32:30,346: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:32:30,602: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:32:30,712: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:32:31,599: INFO/job_scheduler] Job 1879 finished 1 steps successfully
[2017-06-01 19:32:31,611: INFO/job_scheduler] Job 1880 finished 1 steps successfully
[2017-06-01 19:32:31,644: INFO/job_scheduler] Job 1879: complete_job: Updating cache
[2017-06-01 19:32:31,644: INFO/job_scheduler] Job 1879 completing (errored=False, cancelled=False)
[2017-06-01 19:32:31,738: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:32:31,783: INFO/job_scheduler] Job 1880: complete_job: Updating cache
[2017-06-01 19:32:31,783: INFO/job_scheduler] Job 1880 completing (errored=False, cancelled=False)
[2017-06-01 19:32:31,886: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:32:31,961: INFO/job_scheduler] Job 1881 finished 1 steps successfully
[2017-06-01 19:32:32,003: INFO/job_scheduler] Job 1881: complete_job: Updating cache
[2017-06-01 19:32:32,004: INFO/job_scheduler] Job 1881 completing (errored=False, cancelled=False)
[2017-06-01 19:32:32,062: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:32:32,160: INFO/job_scheduler] Job 1882 finished 1 steps successfully
[2017-06-01 19:32:32,204: INFO/job_scheduler] Job 1882: complete_job: Updating cache
[2017-06-01 19:32:32,205: INFO/job_scheduler] Job 1882 completing (errored=False, cancelled=False)
[2017-06-01 19:32:32,279: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:32:34,073: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:34,073: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:34,093: INFO/command_plan] add_jobs: created Job 1883 (Test for host connectivity)
[2017-06-01 19:32:34,156: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:32:34,200: INFO/job_scheduler] Job 1883: RunJobThread.run
[2017-06-01 19:32:34,240: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:34,241: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:34,260: INFO/command_plan] add_jobs: created Job 1884 (Test for host connectivity)
[2017-06-01 19:32:34,298: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:32:34,300: INFO/job_scheduler] Job 1884: RunJobThread.run
[2017-06-01 19:32:34,409: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:34,410: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:34,428: INFO/command_plan] add_jobs: created Job 1885 (Test for host connectivity)
[2017-06-01 19:32:34,489: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:32:34,531: INFO/job_scheduler] Job 1885: RunJobThread.run
[2017-06-01 19:32:34,580: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:34,580: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:34,605: INFO/command_plan] add_jobs: created Job 1886 (Test for host connectivity)
[2017-06-01 19:32:34,699: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:32:34,739: INFO/job_scheduler] Job 1886: RunJobThread.run
[2017-06-01 19:32:36,570: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:32:36,706: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:32:36,862: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:32:37,033: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:32:37,910: INFO/job_scheduler] Job 1883 finished 1 steps successfully
[2017-06-01 19:32:37,970: INFO/job_scheduler] Job 1883: complete_job: Updating cache
[2017-06-01 19:32:37,971: INFO/job_scheduler] Job 1883 completing (errored=False, cancelled=False)
[2017-06-01 19:32:38,025: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:32:38,067: INFO/job_scheduler] Job 1884 finished 1 steps successfully
[2017-06-01 19:32:38,100: INFO/job_scheduler] Job 1884: complete_job: Updating cache
[2017-06-01 19:32:38,101: INFO/job_scheduler] Job 1884 completing (errored=False, cancelled=False)
[2017-06-01 19:32:38,173: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:32:38,252: INFO/job_scheduler] Job 1885 finished 1 steps successfully
[2017-06-01 19:32:38,293: INFO/job_scheduler] Job 1885: complete_job: Updating cache
[2017-06-01 19:32:38,305: INFO/job_scheduler] Job 1885 completing (errored=False, cancelled=False)
[2017-06-01 19:32:38,346: INFO/job_scheduler] Job 1886 finished 1 steps successfully
[2017-06-01 19:32:38,394: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:32:38,429: INFO/job_scheduler] Job 1886: complete_job: Updating cache
[2017-06-01 19:32:38,430: INFO/job_scheduler] Job 1886 completing (errored=False, cancelled=False)
[2017-06-01 19:32:38,542: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:32:40,367: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:40,368: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:40,392: INFO/command_plan] add_jobs: created Job 1887 (Test for host connectivity)
[2017-06-01 19:32:40,480: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:32:40,496: INFO/job_scheduler] Job 1887: RunJobThread.run
[2017-06-01 19:32:40,557: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:40,557: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:40,566: INFO/command_plan] add_jobs: created Job 1888 (Test for host connectivity)
[2017-06-01 19:32:40,582: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:32:40,629: INFO/job_scheduler] Job 1888: RunJobThread.run
[2017-06-01 19:32:40,717: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:40,717: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:40,736: INFO/command_plan] add_jobs: created Job 1889 (Test for host connectivity)
[2017-06-01 19:32:40,819: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:32:40,848: INFO/job_scheduler] Job 1889: RunJobThread.run
[2017-06-01 19:32:40,906: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:40,907: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:40,917: INFO/command_plan] add_jobs: created Job 1890 (Test for host connectivity)
[2017-06-01 19:32:40,959: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:32:41,012: INFO/job_scheduler] Job 1890: RunJobThread.run
[2017-06-01 19:32:42,929: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:32:43,062: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:32:43,339: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:32:43,489: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:32:44,248: INFO/job_scheduler] Job 1887 finished 1 steps successfully
[2017-06-01 19:32:44,274: INFO/job_scheduler] Job 1887: complete_job: Updating cache
[2017-06-01 19:32:44,275: INFO/job_scheduler] Job 1887 completing (errored=False, cancelled=False)
[2017-06-01 19:32:44,308: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:32:44,382: INFO/job_scheduler] Job 1888 finished 1 steps successfully
[2017-06-01 19:32:44,437: INFO/job_scheduler] Job 1888: complete_job: Updating cache
[2017-06-01 19:32:44,437: INFO/job_scheduler] Job 1888 completing (errored=False, cancelled=False)
[2017-06-01 19:32:44,576: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:32:44,857: INFO/job_scheduler] Job 1889 finished 1 steps successfully
[2017-06-01 19:32:44,899: INFO/job_scheduler] Job 1889: complete_job: Updating cache
[2017-06-01 19:32:44,899: INFO/job_scheduler] Job 1889 completing (errored=False, cancelled=False)
[2017-06-01 19:32:44,926: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:32:44,927: INFO/job_scheduler] Job 1890 finished 1 steps successfully
[2017-06-01 19:32:45,001: INFO/job_scheduler] Job 1890: complete_job: Updating cache
[2017-06-01 19:32:45,001: INFO/job_scheduler] Job 1890 completing (errored=False, cancelled=False)
[2017-06-01 19:32:45,074: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:32:46,781: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:46,781: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:46,814: INFO/command_plan] add_jobs: created Job 1891 (Test for host connectivity)
[2017-06-01 19:32:46,851: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:32:46,871: INFO/job_scheduler] Job 1891: RunJobThread.run
[2017-06-01 19:32:46,973: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:46,975: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:46,999: INFO/command_plan] add_jobs: created Job 1892 (Test for host connectivity)
[2017-06-01 19:32:47,123: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:32:47,131: INFO/job_scheduler] Job 1892: RunJobThread.run
[2017-06-01 19:32:47,194: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:47,196: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:47,207: INFO/command_plan] add_jobs: created Job 1893 (Test for host connectivity)
[2017-06-01 19:32:47,233: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:32:47,274: INFO/job_scheduler] Job 1893: RunJobThread.run
[2017-06-01 19:32:47,333: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:47,334: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:47,353: INFO/command_plan] add_jobs: created Job 1894 (Test for host connectivity)
[2017-06-01 19:32:47,398: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:32:47,420: INFO/job_scheduler] Job 1894: RunJobThread.run
[2017-06-01 19:32:49,297: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:32:49,486: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:32:49,652: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:32:49,745: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:32:50,624: INFO/job_scheduler] Job 1891 finished 1 steps successfully
[2017-06-01 19:32:50,665: INFO/job_scheduler] Job 1891: complete_job: Updating cache
[2017-06-01 19:32:50,665: INFO/job_scheduler] Job 1891 completing (errored=False, cancelled=False)
[2017-06-01 19:32:50,721: INFO/job_scheduler] Job 1892 finished 1 steps successfully
[2017-06-01 19:32:50,775: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:32:50,797: INFO/job_scheduler] Job 1892: complete_job: Updating cache
[2017-06-01 19:32:50,798: INFO/job_scheduler] Job 1892 completing (errored=False, cancelled=False)
[2017-06-01 19:32:50,906: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:32:51,076: INFO/job_scheduler] Job 1893 finished 1 steps successfully
[2017-06-01 19:32:51,117: INFO/job_scheduler] Job 1893: complete_job: Updating cache
[2017-06-01 19:32:51,117: INFO/job_scheduler] Job 1893 completing (errored=False, cancelled=False)
[2017-06-01 19:32:51,148: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:32:51,195: INFO/job_scheduler] Job 1894 finished 1 steps successfully
[2017-06-01 19:32:51,248: INFO/job_scheduler] Job 1894: complete_job: Updating cache
[2017-06-01 19:32:51,248: INFO/job_scheduler] Job 1894 completing (errored=False, cancelled=False)
[2017-06-01 19:32:51,323: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:32:53,181: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:53,182: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:53,198: INFO/command_plan] add_jobs: created Job 1895 (Test for host connectivity)
[2017-06-01 19:32:53,250: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:32:53,285: INFO/job_scheduler] Job 1895: RunJobThread.run
[2017-06-01 19:32:53,325: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:53,330: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:53,347: INFO/command_plan] add_jobs: created Job 1896 (Test for host connectivity)
[2017-06-01 19:32:53,442: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:32:53,445: INFO/job_scheduler] Job 1896: RunJobThread.run
[2017-06-01 19:32:53,512: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:53,513: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:53,545: INFO/command_plan] add_jobs: created Job 1897 (Test for host connectivity)
[2017-06-01 19:32:53,564: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:32:53,581: INFO/job_scheduler] Job 1897: RunJobThread.run
[2017-06-01 19:32:53,707: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:53,716: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:53,747: INFO/command_plan] add_jobs: created Job 1898 (Test for host connectivity)
[2017-06-01 19:32:53,782: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:32:53,870: INFO/job_scheduler] Job 1898: RunJobThread.run
[2017-06-01 19:32:55,736: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:32:55,869: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:32:55,997: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:32:56,100: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:32:57,035: INFO/job_scheduler] Job 1895 finished 1 steps successfully
[2017-06-01 19:32:57,080: INFO/job_scheduler] Job 1895: complete_job: Updating cache
[2017-06-01 19:32:57,080: INFO/job_scheduler] Job 1895 completing (errored=False, cancelled=False)
[2017-06-01 19:32:57,172: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:32:57,191: INFO/job_scheduler] Job 1896 finished 1 steps successfully
[2017-06-01 19:32:57,298: INFO/job_scheduler] Job 1896: complete_job: Updating cache
[2017-06-01 19:32:57,298: INFO/job_scheduler] Job 1896 completing (errored=False, cancelled=False)
[2017-06-01 19:32:57,372: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:32:57,446: INFO/job_scheduler] Job 1897 finished 1 steps successfully
[2017-06-01 19:32:57,488: INFO/job_scheduler] Job 1897: complete_job: Updating cache
[2017-06-01 19:32:57,489: INFO/job_scheduler] Job 1897 completing (errored=False, cancelled=False)
[2017-06-01 19:32:57,547: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:32:57,583: INFO/job_scheduler] Job 1898 finished 1 steps successfully
[2017-06-01 19:32:57,662: INFO/job_scheduler] Job 1898: complete_job: Updating cache
[2017-06-01 19:32:57,670: INFO/job_scheduler] Job 1898 completing (errored=False, cancelled=False)
[2017-06-01 19:32:57,730: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:32:59,487: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:59,488: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:59,513: INFO/command_plan] add_jobs: created Job 1899 (Test for host connectivity)
[2017-06-01 19:32:59,576: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:32:59,602: INFO/job_scheduler] Job 1899: RunJobThread.run
[2017-06-01 19:32:59,678: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:59,680: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:59,704: INFO/command_plan] add_jobs: created Job 1900 (Test for host connectivity)
[2017-06-01 19:32:59,818: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:32:59,824: INFO/job_scheduler] Job 1900: RunJobThread.run
[2017-06-01 19:32:59,880: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:59,881: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:32:59,908: INFO/command_plan] add_jobs: created Job 1901 (Test for host connectivity)
[2017-06-01 19:32:59,998: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:32:59,998: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:00,036: INFO/command_plan] add_jobs: created Job 1902 (Test for host connectivity)
[2017-06-01 19:33:00,091: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:33:00,113: INFO/job_scheduler] Job 1901: RunJobThread.run
[2017-06-01 19:33:00,129: INFO/job_scheduler] Job 1902: RunJobThread.run
[2017-06-01 19:33:00,172: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:33:02,011: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:33:02,152: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:33:02,510: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:33:02,525: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:33:03,398: INFO/job_scheduler] Job 1899 finished 1 steps successfully
[2017-06-01 19:33:03,438: INFO/job_scheduler] Job 1899: complete_job: Updating cache
[2017-06-01 19:33:03,438: INFO/job_scheduler] Job 1899 completing (errored=False, cancelled=False)
[2017-06-01 19:33:03,491: INFO/job_scheduler] Job 1900 finished 1 steps successfully
[2017-06-01 19:33:03,517: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:33:03,564: INFO/job_scheduler] Job 1900: complete_job: Updating cache
[2017-06-01 19:33:03,564: INFO/job_scheduler] Job 1900 completing (errored=False, cancelled=False)
[2017-06-01 19:33:03,690: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:33:03,779: INFO/job_scheduler] Job 1901 finished 1 steps successfully
[2017-06-01 19:33:03,845: INFO/job_scheduler] Job 1901: complete_job: Updating cache
[2017-06-01 19:33:03,845: INFO/job_scheduler] Job 1901 completing (errored=False, cancelled=False)
[2017-06-01 19:33:03,899: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:33:04,026: INFO/job_scheduler] Job 1902 finished 1 steps successfully
[2017-06-01 19:33:04,070: INFO/job_scheduler] Job 1902: complete_job: Updating cache
[2017-06-01 19:33:04,070: INFO/job_scheduler] Job 1902 completing (errored=False, cancelled=False)
[2017-06-01 19:33:04,107: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:33:05,830: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:05,831: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:05,857: INFO/command_plan] add_jobs: created Job 1903 (Test for host connectivity)
[2017-06-01 19:33:05,945: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:33:05,974: INFO/job_scheduler] Job 1903: RunJobThread.run
[2017-06-01 19:33:06,003: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:06,003: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:06,032: INFO/command_plan] add_jobs: created Job 1904 (Test for host connectivity)
[2017-06-01 19:33:06,059: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:33:06,081: INFO/job_scheduler] Job 1904: RunJobThread.run
[2017-06-01 19:33:06,178: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:06,178: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:06,186: INFO/command_plan] add_jobs: created Job 1905 (Test for host connectivity)
[2017-06-01 19:33:06,215: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:33:06,250: INFO/job_scheduler] Job 1905: RunJobThread.run
[2017-06-01 19:33:06,336: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:06,337: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:06,361: INFO/command_plan] add_jobs: created Job 1906 (Test for host connectivity)
[2017-06-01 19:33:06,399: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:33:06,500: INFO/job_scheduler] Job 1906: RunJobThread.run
[2017-06-01 19:33:08,379: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:33:08,469: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:33:08,654: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:33:08,812: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:33:09,720: INFO/job_scheduler] Job 1903 finished 1 steps successfully
[2017-06-01 19:33:09,802: INFO/job_scheduler] Job 1904 finished 1 steps successfully
[2017-06-01 19:33:09,822: INFO/job_scheduler] Job 1903: complete_job: Updating cache
[2017-06-01 19:33:09,824: INFO/job_scheduler] Job 1903 completing (errored=False, cancelled=False)
[2017-06-01 19:33:09,871: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:33:09,914: INFO/job_scheduler] Job 1904: complete_job: Updating cache
[2017-06-01 19:33:09,914: INFO/job_scheduler] Job 1904 completing (errored=False, cancelled=False)
[2017-06-01 19:33:10,018: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:33:10,020: INFO/job_scheduler] Job 1905 finished 1 steps successfully
[2017-06-01 19:33:10,054: INFO/job_scheduler] Job 1905: complete_job: Updating cache
[2017-06-01 19:33:10,054: INFO/job_scheduler] Job 1905 completing (errored=False, cancelled=False)
[2017-06-01 19:33:10,135: INFO/job_scheduler] Job 1906 finished 1 steps successfully
[2017-06-01 19:33:10,145: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:33:10,180: INFO/job_scheduler] Job 1906: complete_job: Updating cache
[2017-06-01 19:33:10,180: INFO/job_scheduler] Job 1906 completing (errored=False, cancelled=False)
[2017-06-01 19:33:10,271: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:33:12,154: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:12,155: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:12,177: INFO/command_plan] add_jobs: created Job 1907 (Test for host connectivity)
[2017-06-01 19:33:12,272: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:33:12,281: INFO/job_scheduler] Job 1907: RunJobThread.run
[2017-06-01 19:33:12,317: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:12,323: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:12,347: INFO/command_plan] add_jobs: created Job 1908 (Test for host connectivity)
[2017-06-01 19:33:12,467: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:33:12,485: INFO/job_scheduler] Job 1908: RunJobThread.run
[2017-06-01 19:33:12,531: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:12,532: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:12,562: INFO/command_plan] add_jobs: created Job 1909 (Test for host connectivity)
[2017-06-01 19:33:12,639: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:33:12,665: INFO/job_scheduler] Job 1909: RunJobThread.run
[2017-06-01 19:33:12,731: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:12,748: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:12,771: INFO/command_plan] add_jobs: created Job 1910 (Test for host connectivity)
[2017-06-01 19:33:12,834: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:33:12,873: INFO/job_scheduler] Job 1910: RunJobThread.run
[2017-06-01 19:33:14,729: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:33:14,871: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:33:14,992: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:33:15,189: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:33:16,081: INFO/job_scheduler] Job 1907 finished 1 steps successfully
[2017-06-01 19:33:16,143: INFO/job_scheduler] Job 1907: complete_job: Updating cache
[2017-06-01 19:33:16,143: INFO/job_scheduler] Job 1907 completing (errored=False, cancelled=False)
[2017-06-01 19:33:16,187: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:33:16,252: INFO/job_scheduler] Job 1908 finished 1 steps successfully
[2017-06-01 19:33:16,296: INFO/job_scheduler] Job 1908: complete_job: Updating cache
[2017-06-01 19:33:16,298: INFO/job_scheduler] Job 1908 completing (errored=False, cancelled=False)
[2017-06-01 19:33:16,386: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:33:16,447: INFO/job_scheduler] Job 1909 finished 1 steps successfully
[2017-06-01 19:33:16,518: INFO/job_scheduler] Job 1909: complete_job: Updating cache
[2017-06-01 19:33:16,520: INFO/job_scheduler] Job 1909 completing (errored=False, cancelled=False)
[2017-06-01 19:33:16,569: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:33:16,667: INFO/job_scheduler] Job 1910 finished 1 steps successfully
[2017-06-01 19:33:16,701: INFO/job_scheduler] Job 1910: complete_job: Updating cache
[2017-06-01 19:33:16,702: INFO/job_scheduler] Job 1910 completing (errored=False, cancelled=False)
[2017-06-01 19:33:16,768: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:33:18,580: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:18,582: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:18,594: INFO/command_plan] add_jobs: created Job 1911 (Test for host connectivity)
[2017-06-01 19:33:18,636: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:33:18,683: INFO/job_scheduler] Job 1911: RunJobThread.run
[2017-06-01 19:33:18,752: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:18,753: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:18,767: INFO/command_plan] add_jobs: created Job 1912 (Test for host connectivity)
[2017-06-01 19:33:18,800: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:33:18,830: INFO/job_scheduler] Job 1912: RunJobThread.run
[2017-06-01 19:33:18,921: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:18,921: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:18,956: INFO/command_plan] add_jobs: created Job 1913 (Test for host connectivity)
[2017-06-01 19:33:19,034: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:33:19,043: INFO/job_scheduler] Job 1913: RunJobThread.run
[2017-06-01 19:33:19,107: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:19,108: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:19,121: INFO/command_plan] add_jobs: created Job 1914 (Test for host connectivity)
[2017-06-01 19:33:19,211: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:33:19,214: INFO/job_scheduler] Job 1914: RunJobThread.run
[2017-06-01 19:33:21,018: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:33:21,216: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:33:21,309: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:33:21,542: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:33:22,359: INFO/job_scheduler] Job 1911 finished 1 steps successfully
[2017-06-01 19:33:22,425: INFO/job_scheduler] Job 1911: complete_job: Updating cache
[2017-06-01 19:33:22,425: INFO/job_scheduler] Job 1911 completing (errored=False, cancelled=False)
[2017-06-01 19:33:22,464: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:33:22,708: INFO/job_scheduler] Job 1913 finished 1 steps successfully
[2017-06-01 19:33:22,708: INFO/job_scheduler] Job 1912 finished 1 steps successfully
[2017-06-01 19:33:22,749: INFO/job_scheduler] Job 1913: complete_job: Updating cache
[2017-06-01 19:33:22,750: INFO/job_scheduler] Job 1913 completing (errored=False, cancelled=False)
[2017-06-01 19:33:22,775: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:33:22,811: INFO/job_scheduler] Job 1912: complete_job: Updating cache
[2017-06-01 19:33:22,811: INFO/job_scheduler] Job 1912 completing (errored=False, cancelled=False)
[2017-06-01 19:33:22,926: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:33:22,988: INFO/job_scheduler] Job 1914 finished 1 steps successfully
[2017-06-01 19:33:23,030: INFO/job_scheduler] Job 1914: complete_job: Updating cache
[2017-06-01 19:33:23,030: INFO/job_scheduler] Job 1914 completing (errored=False, cancelled=False)
[2017-06-01 19:33:23,093: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:33:24,870: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:24,870: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:24,882: INFO/command_plan] add_jobs: created Job 1915 (Test for host connectivity)
[2017-06-01 19:33:24,970: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:33:24,982: INFO/job_scheduler] Job 1915: RunJobThread.run
[2017-06-01 19:33:25,025: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:25,026: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:25,060: INFO/command_plan] add_jobs: created Job 1916 (Test for host connectivity)
[2017-06-01 19:33:25,200: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:25,202: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:25,220: INFO/command_plan] add_jobs: created Job 1917 (Test for host connectivity)
[2017-06-01 19:33:25,271: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:33:25,312: INFO/job_scheduler] Job 1916: RunJobThread.run
[2017-06-01 19:33:25,330: INFO/job_scheduler] Job 1917: RunJobThread.run
[2017-06-01 19:33:25,361: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:33:25,521: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:25,541: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:25,599: INFO/command_plan] add_jobs: created Job 1918 (Test for host connectivity)
[2017-06-01 19:33:25,667: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:33:25,697: INFO/job_scheduler] Job 1918: RunJobThread.run
[2017-06-01 19:33:27,265: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:33:27,558: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:33:27,705: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:33:27,940: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:33:28,618: INFO/job_scheduler] Job 1915 finished 1 steps successfully
[2017-06-01 19:33:28,659: INFO/job_scheduler] Job 1915: complete_job: Updating cache
[2017-06-01 19:33:28,659: INFO/job_scheduler] Job 1915 completing (errored=False, cancelled=False)
[2017-06-01 19:33:28,697: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:33:28,886: INFO/job_scheduler] Job 1916 finished 1 steps successfully
[2017-06-01 19:33:28,925: INFO/job_scheduler] Job 1916: complete_job: Updating cache
[2017-06-01 19:33:28,926: INFO/job_scheduler] Job 1916 completing (errored=False, cancelled=False)
[2017-06-01 19:33:28,977: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:33:29,152: INFO/job_scheduler] Job 1917 finished 1 steps successfully
[2017-06-01 19:33:29,185: INFO/job_scheduler] Job 1917: complete_job: Updating cache
[2017-06-01 19:33:29,185: INFO/job_scheduler] Job 1917 completing (errored=False, cancelled=False)
[2017-06-01 19:33:29,243: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:33:29,382: INFO/job_scheduler] Job 1918 finished 1 steps successfully
[2017-06-01 19:33:29,442: INFO/job_scheduler] Job 1918: complete_job: Updating cache
[2017-06-01 19:33:29,443: INFO/job_scheduler] Job 1918 completing (errored=False, cancelled=False)
[2017-06-01 19:33:29,502: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:33:31,333: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:31,335: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:31,342: INFO/command_plan] add_jobs: created Job 1919 (Test for host connectivity)
[2017-06-01 19:33:31,373: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:33:31,419: INFO/job_scheduler] Job 1919: RunJobThread.run
[2017-06-01 19:33:31,495: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:31,496: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:31,540: INFO/command_plan] add_jobs: created Job 1920 (Test for host connectivity)
[2017-06-01 19:33:31,636: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:33:31,661: INFO/job_scheduler] Job 1920: RunJobThread.run
[2017-06-01 19:33:31,743: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:31,745: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:31,770: INFO/command_plan] add_jobs: created Job 1921 (Test for host connectivity)
[2017-06-01 19:33:31,855: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:33:31,859: INFO/job_scheduler] Job 1921: RunJobThread.run
[2017-06-01 19:33:31,918: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:31,918: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:31,934: INFO/command_plan] add_jobs: created Job 1922 (Test for host connectivity)
[2017-06-01 19:33:32,008: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:33:32,066: INFO/job_scheduler] Job 1922: RunJobThread.run
[2017-06-01 19:33:33,981: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:33:34,088: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:33:34,175: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:33:34,515: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:33:35,295: INFO/job_scheduler] Job 1919 finished 1 steps successfully
[2017-06-01 19:33:35,342: INFO/job_scheduler] Job 1919: complete_job: Updating cache
[2017-06-01 19:33:35,342: INFO/job_scheduler] Job 1919 completing (errored=False, cancelled=False)
[2017-06-01 19:33:35,449: INFO/job_scheduler] Job 1920 finished 1 steps successfully
[2017-06-01 19:33:35,470: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:33:35,519: INFO/job_scheduler] Job 1920: complete_job: Updating cache
[2017-06-01 19:33:35,520: INFO/job_scheduler] Job 1920 completing (errored=False, cancelled=False)
[2017-06-01 19:33:35,525: INFO/job_scheduler] Job 1921 finished 1 steps successfully
[2017-06-01 19:33:35,561: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:33:35,574: INFO/job_scheduler] Job 1921: complete_job: Updating cache
[2017-06-01 19:33:35,574: INFO/job_scheduler] Job 1921 completing (errored=False, cancelled=False)
[2017-06-01 19:33:35,659: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:33:35,955: INFO/job_scheduler] Job 1922 finished 1 steps successfully
[2017-06-01 19:33:35,983: INFO/job_scheduler] Job 1922: complete_job: Updating cache
[2017-06-01 19:33:35,983: INFO/job_scheduler] Job 1922 completing (errored=False, cancelled=False)
[2017-06-01 19:33:36,042: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:33:37,690: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:37,693: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:37,707: INFO/command_plan] add_jobs: created Job 1923 (Test for host connectivity)
[2017-06-01 19:33:37,763: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:33:37,790: INFO/job_scheduler] Job 1923: RunJobThread.run
[2017-06-01 19:33:37,846: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:37,846: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:37,858: INFO/command_plan] add_jobs: created Job 1924 (Test for host connectivity)
[2017-06-01 19:33:37,999: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:38,001: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:38,132: INFO/command_plan] add_jobs: created Job 1925 (Test for host connectivity)
[2017-06-01 19:33:38,142: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:33:38,147: INFO/job_scheduler] Job 1924: RunJobThread.run
[2017-06-01 19:33:38,152: INFO/job_scheduler] Job 1925: RunJobThread.run
[2017-06-01 19:33:38,161: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:33:38,265: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:38,265: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:38,323: INFO/command_plan] add_jobs: created Job 1926 (Test for host connectivity)
[2017-06-01 19:33:38,369: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:33:38,469: INFO/job_scheduler] Job 1926: RunJobThread.run
[2017-06-01 19:33:40,134: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:33:40,417: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:33:40,508: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:33:40,689: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:33:41,412: INFO/job_scheduler] Job 1923 finished 1 steps successfully
[2017-06-01 19:33:41,465: INFO/job_scheduler] Job 1923: complete_job: Updating cache
[2017-06-01 19:33:41,465: INFO/job_scheduler] Job 1923 completing (errored=False, cancelled=False)
[2017-06-01 19:33:41,565: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:33:41,845: INFO/job_scheduler] Job 1924 finished 1 steps successfully
[2017-06-01 19:33:41,907: INFO/job_scheduler] Job 1924: complete_job: Updating cache
[2017-06-01 19:33:41,908: INFO/job_scheduler] Job 1924 completing (errored=False, cancelled=False)
[2017-06-01 19:33:41,973: INFO/job_scheduler] Job 1925 finished 1 steps successfully
[2017-06-01 19:33:41,982: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:33:42,009: INFO/job_scheduler] Job 1925: complete_job: Updating cache
[2017-06-01 19:33:42,009: INFO/job_scheduler] Job 1925 completing (errored=False, cancelled=False)
[2017-06-01 19:33:42,092: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:33:42,161: INFO/job_scheduler] Job 1926 finished 1 steps successfully
[2017-06-01 19:33:42,207: INFO/job_scheduler] Job 1926: complete_job: Updating cache
[2017-06-01 19:33:42,207: INFO/job_scheduler] Job 1926 completing (errored=False, cancelled=False)
[2017-06-01 19:33:42,250: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:33:44,687: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:44,687: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:44,706: INFO/command_plan] add_jobs: created Job 1927 (Test for host connectivity)
[2017-06-01 19:33:44,781: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:33:44,806: INFO/job_scheduler] Job 1927: RunJobThread.run
[2017-06-01 19:33:44,865: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:44,865: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:44,884: INFO/command_plan] add_jobs: created Job 1928 (Test for host connectivity)
[2017-06-01 19:33:44,927: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:33:44,964: INFO/job_scheduler] Job 1928: RunJobThread.run
[2017-06-01 19:33:45,115: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:45,131: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:45,151: INFO/command_plan] add_jobs: created Job 1929 (Test for host connectivity)
[2017-06-01 19:33:45,186: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:33:45,227: INFO/job_scheduler] Job 1929: RunJobThread.run
[2017-06-01 19:33:45,337: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:45,340: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:45,381: INFO/command_plan] add_jobs: created Job 1930 (Test for host connectivity)
[2017-06-01 19:33:45,415: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:33:45,494: INFO/job_scheduler] Job 1930: RunJobThread.run
[2017-06-01 19:33:47,219: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:33:47,413: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:33:47,498: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:33:47,731: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:33:48,602: INFO/job_scheduler] Job 1927 finished 1 steps successfully
[2017-06-01 19:33:48,663: INFO/job_scheduler] Job 1927: complete_job: Updating cache
[2017-06-01 19:33:48,663: INFO/job_scheduler] Job 1927 completing (errored=False, cancelled=False)
[2017-06-01 19:33:48,714: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:33:48,838: INFO/job_scheduler] Job 1928 finished 1 steps successfully
[2017-06-01 19:33:48,871: INFO/job_scheduler] Job 1929 finished 1 steps successfully
[2017-06-01 19:33:48,880: INFO/job_scheduler] Job 1928: complete_job: Updating cache
[2017-06-01 19:33:48,881: INFO/job_scheduler] Job 1928 completing (errored=False, cancelled=False)
[2017-06-01 19:33:48,979: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:33:49,007: INFO/job_scheduler] Job 1929: complete_job: Updating cache
[2017-06-01 19:33:49,007: INFO/job_scheduler] Job 1929 completing (errored=False, cancelled=False)
[2017-06-01 19:33:49,071: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:33:49,155: INFO/job_scheduler] Job 1930 finished 1 steps successfully
[2017-06-01 19:33:49,181: INFO/job_scheduler] Job 1930: complete_job: Updating cache
[2017-06-01 19:33:49,181: INFO/job_scheduler] Job 1930 completing (errored=False, cancelled=False)
[2017-06-01 19:33:49,287: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:33:51,028: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:51,028: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:51,049: INFO/command_plan] add_jobs: created Job 1931 (Test for host connectivity)
[2017-06-01 19:33:51,108: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:33:51,111: INFO/job_scheduler] Job 1931: RunJobThread.run
[2017-06-01 19:33:51,185: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:51,185: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:51,207: INFO/command_plan] add_jobs: created Job 1932 (Test for host connectivity)
[2017-06-01 19:33:51,301: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:51,301: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:51,358: INFO/command_plan] add_jobs: created Job 1933 (Test for host connectivity)
[2017-06-01 19:33:51,419: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:33:51,495: INFO/job_scheduler] Job 1932: RunJobThread.run
[2017-06-01 19:33:51,507: INFO/job_scheduler] Job 1933: RunJobThread.run
[2017-06-01 19:33:51,520: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:33:51,587: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:51,589: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:51,635: INFO/command_plan] add_jobs: created Job 1934 (Test for host connectivity)
[2017-06-01 19:33:51,709: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:33:51,753: INFO/job_scheduler] Job 1934: RunJobThread.run
[2017-06-01 19:33:53,466: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:33:53,722: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:33:53,741: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:33:54,089: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:33:54,842: INFO/job_scheduler] Job 1931 finished 1 steps successfully
[2017-06-01 19:33:54,870: INFO/job_scheduler] Job 1931: complete_job: Updating cache
[2017-06-01 19:33:54,870: INFO/job_scheduler] Job 1931 completing (errored=False, cancelled=False)
[2017-06-01 19:33:54,911: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:33:55,116: INFO/job_scheduler] Job 1932 finished 1 steps successfully
[2017-06-01 19:33:55,172: INFO/job_scheduler] Job 1932: complete_job: Updating cache
[2017-06-01 19:33:55,172: INFO/job_scheduler] Job 1932 completing (errored=False, cancelled=False)
[2017-06-01 19:33:55,229: INFO/job_scheduler] Job 1933 finished 1 steps successfully
[2017-06-01 19:33:55,239: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:33:55,254: INFO/job_scheduler] Job 1933: complete_job: Updating cache
[2017-06-01 19:33:55,282: INFO/job_scheduler] Job 1933 completing (errored=False, cancelled=False)
[2017-06-01 19:33:55,368: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:33:55,616: INFO/job_scheduler] Job 1934 finished 1 steps successfully
[2017-06-01 19:33:55,637: INFO/job_scheduler] Job 1934: complete_job: Updating cache
[2017-06-01 19:33:55,637: INFO/job_scheduler] Job 1934 completing (errored=False, cancelled=False)
[2017-06-01 19:33:55,673: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:33:57,290: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:57,292: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:57,304: INFO/command_plan] add_jobs: created Job 1935 (Test for host connectivity)
[2017-06-01 19:33:57,380: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:57,380: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:57,406: INFO/command_plan] add_jobs: created Job 1936 (Test for host connectivity)
[2017-06-01 19:33:57,427: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:33:57,431: INFO/job_scheduler] Job 1936: RunJobThread.run
[2017-06-01 19:33:57,465: INFO/job_scheduler] Job 1935: RunJobThread.run
[2017-06-01 19:33:57,489: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:33:57,611: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:57,611: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:57,677: INFO/command_plan] add_jobs: created Job 1937 (Test for host connectivity)
[2017-06-01 19:33:57,728: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:33:57,733: INFO/job_scheduler] Job 1937: RunJobThread.run
[2017-06-01 19:33:57,850: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:33:57,850: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:33:57,877: INFO/command_plan] add_jobs: created Job 1938 (Test for host connectivity)
[2017-06-01 19:33:58,002: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:33:58,035: INFO/job_scheduler] Job 1938: RunJobThread.run
[2017-06-01 19:33:59,839: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:34:00,035: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:34:00,097: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:34:00,359: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:34:01,153: INFO/job_scheduler] Job 1935 finished 1 steps successfully
[2017-06-01 19:34:01,203: INFO/job_scheduler] Job 1935: complete_job: Updating cache
[2017-06-01 19:34:01,203: INFO/job_scheduler] Job 1935 completing (errored=False, cancelled=False)
[2017-06-01 19:34:01,269: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:34:01,352: INFO/job_scheduler] Job 1936 finished 1 steps successfully
[2017-06-01 19:34:01,487: INFO/job_scheduler] Job 1936: complete_job: Updating cache
[2017-06-01 19:34:01,487: INFO/job_scheduler] Job 1936 completing (errored=False, cancelled=False)
[2017-06-01 19:34:01,549: INFO/job_scheduler] Job 1937 finished 1 steps successfully
[2017-06-01 19:34:01,553: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:34:01,598: INFO/job_scheduler] Job 1937: complete_job: Updating cache
[2017-06-01 19:34:01,599: INFO/job_scheduler] Job 1937 completing (errored=False, cancelled=False)
[2017-06-01 19:34:01,673: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:34:01,745: INFO/job_scheduler] Job 1938 finished 1 steps successfully
[2017-06-01 19:34:01,812: INFO/job_scheduler] Job 1938: complete_job: Updating cache
[2017-06-01 19:34:01,814: INFO/job_scheduler] Job 1938 completing (errored=False, cancelled=False)
[2017-06-01 19:34:01,862: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:34:03,591: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:03,591: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:03,603: INFO/command_plan] add_jobs: created Job 1939 (Test for host connectivity)
[2017-06-01 19:34:03,637: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:34:03,643: INFO/job_scheduler] Job 1939: RunJobThread.run
[2017-06-01 19:34:03,724: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:03,725: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:03,746: INFO/command_plan] add_jobs: created Job 1940 (Test for host connectivity)
[2017-06-01 19:34:03,840: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:34:03,851: INFO/job_scheduler] Job 1940: RunJobThread.run
[2017-06-01 19:34:03,909: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:03,910: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:03,968: INFO/command_plan] add_jobs: created Job 1941 (Test for host connectivity)
[2017-06-01 19:34:04,062: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:04,063: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:04,070: INFO/command_plan] add_jobs: created Job 1942 (Test for host connectivity)
[2017-06-01 19:34:04,085: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:34:04,118: INFO/job_scheduler] Job 1941: RunJobThread.run
[2017-06-01 19:34:04,130: INFO/job_scheduler] Job 1942: RunJobThread.run
[2017-06-01 19:34:04,141: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:34:06,044: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:34:06,179: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:34:06,457: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:34:06,501: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:34:07,379: INFO/job_scheduler] Job 1939 finished 1 steps successfully
[2017-06-01 19:34:07,401: INFO/job_scheduler] Job 1939: complete_job: Updating cache
[2017-06-01 19:34:07,401: INFO/job_scheduler] Job 1939 completing (errored=False, cancelled=False)
[2017-06-01 19:34:07,460: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:34:07,513: INFO/job_scheduler] Job 1940 finished 1 steps successfully
[2017-06-01 19:34:07,528: INFO/job_scheduler] Job 1940: complete_job: Updating cache
[2017-06-01 19:34:07,528: INFO/job_scheduler] Job 1940 completing (errored=False, cancelled=False)
[2017-06-01 19:34:07,660: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:34:07,890: INFO/job_scheduler] Job 1942 finished 1 steps successfully
[2017-06-01 19:34:07,935: INFO/job_scheduler] Job 1942: complete_job: Updating cache
[2017-06-01 19:34:07,935: INFO/job_scheduler] Job 1942 completing (errored=False, cancelled=False)
[2017-06-01 19:34:07,978: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:34:08,055: INFO/job_scheduler] Job 1941 finished 1 steps successfully
[2017-06-01 19:34:08,068: INFO/job_scheduler] Job 1941: complete_job: Updating cache
[2017-06-01 19:34:08,068: INFO/job_scheduler] Job 1941 completing (errored=False, cancelled=False)
[2017-06-01 19:34:08,129: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:34:09,757: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:09,758: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:09,774: INFO/command_plan] add_jobs: created Job 1943 (Test for host connectivity)
[2017-06-01 19:34:09,855: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:09,855: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:09,867: INFO/command_plan] add_jobs: created Job 1944 (Test for host connectivity)
[2017-06-01 19:34:09,953: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:34:09,979: INFO/job_scheduler] Job 1944: RunJobThread.run
[2017-06-01 19:34:09,998: INFO/job_scheduler] Job 1943: RunJobThread.run
[2017-06-01 19:34:10,012: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:34:10,069: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:10,071: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:10,103: INFO/command_plan] add_jobs: created Job 1945 (Test for host connectivity)
[2017-06-01 19:34:10,151: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:34:10,199: INFO/job_scheduler] Job 1945: RunJobThread.run
[2017-06-01 19:34:10,328: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:10,330: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:10,365: INFO/command_plan] add_jobs: created Job 1946 (Test for host connectivity)
[2017-06-01 19:34:10,396: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:34:10,408: INFO/job_scheduler] Job 1946: RunJobThread.run
[2017-06-01 19:34:12,377: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:34:12,405: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:34:12,611: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:34:12,743: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:34:13,709: INFO/job_scheduler] Job 1944 finished 1 steps successfully
[2017-06-01 19:34:13,761: INFO/job_scheduler] Job 1943 finished 1 steps successfully
[2017-06-01 19:34:13,767: INFO/job_scheduler] Job 1944: complete_job: Updating cache
[2017-06-01 19:34:13,767: INFO/job_scheduler] Job 1944 completing (errored=False, cancelled=False)
[2017-06-01 19:34:13,810: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:34:13,850: INFO/job_scheduler] Job 1943: complete_job: Updating cache
[2017-06-01 19:34:13,850: INFO/job_scheduler] Job 1943 completing (errored=False, cancelled=False)
[2017-06-01 19:34:13,911: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:34:13,981: INFO/job_scheduler] Job 1945 finished 1 steps successfully
[2017-06-01 19:34:14,032: INFO/job_scheduler] Job 1945: complete_job: Updating cache
[2017-06-01 19:34:14,032: INFO/job_scheduler] Job 1945 completing (errored=False, cancelled=False)
[2017-06-01 19:34:14,065: INFO/job_scheduler] Job 1946 finished 1 steps successfully
[2017-06-01 19:34:14,093: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:34:14,110: INFO/job_scheduler] Job 1946: complete_job: Updating cache
[2017-06-01 19:34:14,129: INFO/job_scheduler] Job 1946 completing (errored=False, cancelled=False)
[2017-06-01 19:34:14,189: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:34:16,028: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:16,030: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:16,041: INFO/command_plan] add_jobs: created Job 1947 (Test for host connectivity)
[2017-06-01 19:34:16,060: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:34:16,063: INFO/job_scheduler] Job 1947: RunJobThread.run
[2017-06-01 19:34:16,179: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:16,180: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:16,194: INFO/command_plan] add_jobs: created Job 1948 (Test for host connectivity)
[2017-06-01 19:34:16,231: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:34:16,278: INFO/job_scheduler] Job 1948: RunJobThread.run
[2017-06-01 19:34:16,344: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:16,346: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:16,361: INFO/command_plan] add_jobs: created Job 1949 (Test for host connectivity)
[2017-06-01 19:34:16,435: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:34:16,470: INFO/job_scheduler] Job 1949: RunJobThread.run
[2017-06-01 19:34:16,522: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:16,528: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:16,541: INFO/command_plan] add_jobs: created Job 1950 (Test for host connectivity)
[2017-06-01 19:34:16,578: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:34:16,583: INFO/job_scheduler] Job 1950: RunJobThread.run
[2017-06-01 19:34:18,556: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:34:18,670: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:34:18,820: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:34:18,915: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:34:19,932: INFO/job_scheduler] Job 1947 finished 1 steps successfully
[2017-06-01 19:34:19,960: INFO/job_scheduler] Job 1947: complete_job: Updating cache
[2017-06-01 19:34:19,960: INFO/job_scheduler] Job 1947 completing (errored=False, cancelled=False)
[2017-06-01 19:34:20,008: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:34:20,012: INFO/job_scheduler] Job 1948 finished 1 steps successfully
[2017-06-01 19:34:20,053: INFO/job_scheduler] Job 1948: complete_job: Updating cache
[2017-06-01 19:34:20,053: INFO/job_scheduler] Job 1948 completing (errored=False, cancelled=False)
[2017-06-01 19:34:20,142: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:34:20,264: INFO/job_scheduler] Job 1949 finished 1 steps successfully
[2017-06-01 19:34:20,357: INFO/job_scheduler] Job 1949: complete_job: Updating cache
[2017-06-01 19:34:20,357: INFO/job_scheduler] Job 1949 completing (errored=False, cancelled=False)
[2017-06-01 19:34:20,371: INFO/job_scheduler] Job 1950 finished 1 steps successfully
[2017-06-01 19:34:20,409: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:34:20,423: INFO/job_scheduler] Job 1950: complete_job: Updating cache
[2017-06-01 19:34:20,424: INFO/job_scheduler] Job 1950 completing (errored=False, cancelled=False)
[2017-06-01 19:34:20,505: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:34:22,267: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:22,267: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:22,285: INFO/command_plan] add_jobs: created Job 1951 (Test for host connectivity)
[2017-06-01 19:34:22,336: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:34:22,348: INFO/job_scheduler] Job 1951: RunJobThread.run
[2017-06-01 19:34:22,403: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:22,403: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:22,411: INFO/command_plan] add_jobs: created Job 1952 (Test for host connectivity)
[2017-06-01 19:34:22,463: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:34:22,524: INFO/job_scheduler] Job 1952: RunJobThread.run
[2017-06-01 19:34:22,564: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:22,566: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:22,593: INFO/command_plan] add_jobs: created Job 1953 (Test for host connectivity)
[2017-06-01 19:34:22,707: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:34:22,709: INFO/job_scheduler] Job 1953: RunJobThread.run
[2017-06-01 19:34:22,794: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:22,794: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:22,917: INFO/command_plan] add_jobs: created Job 1954 (Test for host connectivity)
[2017-06-01 19:34:23,035: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:34:23,072: INFO/job_scheduler] Job 1954: RunJobThread.run
[2017-06-01 19:34:24,644: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:34:24,940: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:34:25,279: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:34:25,890: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:34:25,940: INFO/job_scheduler] Job 1951 finished 1 steps successfully
[2017-06-01 19:34:25,983: INFO/job_scheduler] Job 1951: complete_job: Updating cache
[2017-06-01 19:34:25,983: INFO/job_scheduler] Job 1951 completing (errored=False, cancelled=False)
[2017-06-01 19:34:26,018: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:34:26,691: INFO/job_scheduler] Job 1953 finished 1 steps successfully
[2017-06-01 19:34:26,709: INFO/job_scheduler] Job 1952 finished 1 steps successfully
[2017-06-01 19:34:26,725: INFO/job_scheduler] Job 1953: complete_job: Updating cache
[2017-06-01 19:34:26,725: INFO/job_scheduler] Job 1953 completing (errored=False, cancelled=False)
[2017-06-01 19:34:26,753: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:34:26,779: INFO/job_scheduler] Job 1952: complete_job: Updating cache
[2017-06-01 19:34:26,779: INFO/job_scheduler] Job 1952 completing (errored=False, cancelled=False)
[2017-06-01 19:34:26,817: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:34:27,306: INFO/job_scheduler] Job 1954 finished 1 steps successfully
[2017-06-01 19:34:27,350: INFO/job_scheduler] Job 1954: complete_job: Updating cache
[2017-06-01 19:34:27,350: INFO/job_scheduler] Job 1954 completing (errored=False, cancelled=False)
[2017-06-01 19:34:27,412: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:34:28,636: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:28,638: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:28,654: INFO/command_plan] add_jobs: created Job 1955 (Test for host connectivity)
[2017-06-01 19:34:28,709: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:34:28,755: INFO/job_scheduler] Job 1955: RunJobThread.run
[2017-06-01 19:34:28,791: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:28,792: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:28,825: INFO/command_plan] add_jobs: created Job 1956 (Test for host connectivity)
[2017-06-01 19:34:28,865: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:34:28,876: INFO/job_scheduler] Job 1956: RunJobThread.run
[2017-06-01 19:34:28,969: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:28,970: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:28,991: INFO/command_plan] add_jobs: created Job 1957 (Test for host connectivity)
[2017-06-01 19:34:29,127: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:34:29,133: INFO/job_scheduler] Job 1957: RunJobThread.run
[2017-06-01 19:34:29,190: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:29,191: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:29,212: INFO/command_plan] add_jobs: created Job 1958 (Test for host connectivity)
[2017-06-01 19:34:29,281: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:34:29,343: INFO/job_scheduler] Job 1958: RunJobThread.run
[2017-06-01 19:34:31,207: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:34:31,253: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:34:31,361: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:34:31,716: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:34:32,538: INFO/job_scheduler] Job 1955 finished 1 steps successfully
[2017-06-01 19:34:32,562: INFO/job_scheduler] Job 1956 finished 1 steps successfully
[2017-06-01 19:34:32,591: INFO/job_scheduler] Job 1955: complete_job: Updating cache
[2017-06-01 19:34:32,591: INFO/job_scheduler] Job 1955 completing (errored=False, cancelled=False)
[2017-06-01 19:34:32,626: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:34:32,647: INFO/job_scheduler] Job 1956: complete_job: Updating cache
[2017-06-01 19:34:32,647: INFO/job_scheduler] Job 1956 completing (errored=False, cancelled=False)
[2017-06-01 19:34:32,723: INFO/job_scheduler] Job 1957 finished 1 steps successfully
[2017-06-01 19:34:32,788: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:34:32,813: INFO/job_scheduler] Job 1957: complete_job: Updating cache
[2017-06-01 19:34:32,813: INFO/job_scheduler] Job 1957 completing (errored=False, cancelled=False)
[2017-06-01 19:34:32,920: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:34:33,201: INFO/job_scheduler] Job 1958 finished 1 steps successfully
[2017-06-01 19:34:33,216: INFO/job_scheduler] Job 1958: complete_job: Updating cache
[2017-06-01 19:34:33,216: INFO/job_scheduler] Job 1958 completing (errored=False, cancelled=False)
[2017-06-01 19:34:33,241: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:34:34,878: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:34,879: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:34,891: INFO/command_plan] add_jobs: created Job 1959 (Test for host connectivity)
[2017-06-01 19:34:34,910: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:34:34,912: INFO/job_scheduler] Job 1959: RunJobThread.run
[2017-06-01 19:34:35,081: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:35,082: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:35,094: INFO/command_plan] add_jobs: created Job 1960 (Test for host connectivity)
[2017-06-01 19:34:35,172: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:34:35,220: INFO/job_scheduler] Job 1960: RunJobThread.run
[2017-06-01 19:34:35,302: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:35,304: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:35,328: INFO/command_plan] add_jobs: created Job 1961 (Test for host connectivity)
[2017-06-01 19:34:35,357: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:34:35,360: INFO/job_scheduler] Job 1961: RunJobThread.run
[2017-06-01 19:34:35,464: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:35,465: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:35,482: INFO/command_plan] add_jobs: created Job 1962 (Test for host connectivity)
[2017-06-01 19:34:35,519: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:34:35,572: INFO/job_scheduler] Job 1962: RunJobThread.run
[2017-06-01 19:34:37,492: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:34:37,689: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:34:37,724: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:34:37,936: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:34:38,853: INFO/job_scheduler] Job 1959 finished 1 steps successfully
[2017-06-01 19:34:38,907: INFO/job_scheduler] Job 1959: complete_job: Updating cache
[2017-06-01 19:34:38,907: INFO/job_scheduler] Job 1959 completing (errored=False, cancelled=False)
[2017-06-01 19:34:38,941: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:34:39,062: INFO/job_scheduler] Job 1960 finished 1 steps successfully
[2017-06-01 19:34:39,123: INFO/job_scheduler] Job 1960: complete_job: Updating cache
[2017-06-01 19:34:39,129: INFO/job_scheduler] Job 1960 completing (errored=False, cancelled=False)
[2017-06-01 19:34:39,136: INFO/job_scheduler] Job 1961 finished 1 steps successfully
[2017-06-01 19:34:39,216: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:34:39,232: INFO/job_scheduler] Job 1961: complete_job: Updating cache
[2017-06-01 19:34:39,232: INFO/job_scheduler] Job 1961 completing (errored=False, cancelled=False)
[2017-06-01 19:34:39,375: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:34:39,463: INFO/job_scheduler] Job 1962 finished 1 steps successfully
[2017-06-01 19:34:39,532: INFO/job_scheduler] Job 1962: complete_job: Updating cache
[2017-06-01 19:34:39,532: INFO/job_scheduler] Job 1962 completing (errored=False, cancelled=False)
[2017-06-01 19:34:39,558: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:34:41,158: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:41,159: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:41,173: INFO/command_plan] add_jobs: created Job 1963 (Test for host connectivity)
[2017-06-01 19:34:41,231: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:34:41,264: INFO/job_scheduler] Job 1963: RunJobThread.run
[2017-06-01 19:34:41,338: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:41,340: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:41,357: INFO/command_plan] add_jobs: created Job 1964 (Test for host connectivity)
[2017-06-01 19:34:41,390: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:34:41,408: INFO/job_scheduler] Job 1964: RunJobThread.run
[2017-06-01 19:34:41,533: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:41,535: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:41,559: INFO/command_plan] add_jobs: created Job 1965 (Test for host connectivity)
[2017-06-01 19:34:41,607: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:34:41,697: INFO/job_scheduler] Job 1965: RunJobThread.run
[2017-06-01 19:34:41,737: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:41,738: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:41,894: INFO/command_plan] add_jobs: created Job 1966 (Test for host connectivity)
[2017-06-01 19:34:41,962: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:34:41,990: INFO/job_scheduler] Job 1966: RunJobThread.run
[2017-06-01 19:34:43,484: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:34:43,833: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:34:44,124: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:34:44,279: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:34:44,869: INFO/job_scheduler] Job 1963 finished 1 steps successfully
[2017-06-01 19:34:44,881: INFO/job_scheduler] Job 1963: complete_job: Updating cache
[2017-06-01 19:34:44,881: INFO/job_scheduler] Job 1963 completing (errored=False, cancelled=False)
[2017-06-01 19:34:44,932: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:34:45,181: INFO/job_scheduler] Job 1964 finished 1 steps successfully
[2017-06-01 19:34:45,239: INFO/job_scheduler] Job 1964: complete_job: Updating cache
[2017-06-01 19:34:45,239: INFO/job_scheduler] Job 1964 completing (errored=False, cancelled=False)
[2017-06-01 19:34:45,294: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:34:45,618: INFO/job_scheduler] Job 1965 finished 1 steps successfully
[2017-06-01 19:34:45,656: INFO/job_scheduler] Job 1965: complete_job: Updating cache
[2017-06-01 19:34:45,656: INFO/job_scheduler] Job 1965 completing (errored=False, cancelled=False)
[2017-06-01 19:34:45,690: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:34:45,709: INFO/job_scheduler] Job 1966 finished 1 steps successfully
[2017-06-01 19:34:45,740: INFO/job_scheduler] Job 1966: complete_job: Updating cache
[2017-06-01 19:34:45,740: INFO/job_scheduler] Job 1966 completing (errored=False, cancelled=False)
[2017-06-01 19:34:45,791: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:34:47,619: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:47,619: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:47,631: INFO/command_plan] add_jobs: created Job 1967 (Test for host connectivity)
[2017-06-01 19:34:47,672: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:34:47,692: INFO/job_scheduler] Job 1967: RunJobThread.run
[2017-06-01 19:34:47,734: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:47,734: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:47,757: INFO/command_plan] add_jobs: created Job 1968 (Test for host connectivity)
[2017-06-01 19:34:47,803: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:34:47,840: INFO/job_scheduler] Job 1968: RunJobThread.run
[2017-06-01 19:34:47,948: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:47,951: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:47,990: INFO/command_plan] add_jobs: created Job 1969 (Test for host connectivity)
[2017-06-01 19:34:48,044: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:34:48,051: INFO/job_scheduler] Job 1969: RunJobThread.run
[2017-06-01 19:34:48,130: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:48,130: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:48,149: INFO/command_plan] add_jobs: created Job 1970 (Test for host connectivity)
[2017-06-01 19:34:48,168: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:34:48,181: INFO/job_scheduler] Job 1970: RunJobThread.run
[2017-06-01 19:34:50,175: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:34:50,306: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:34:50,495: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:34:50,632: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:34:51,487: INFO/job_scheduler] Job 1967 finished 1 steps successfully
[2017-06-01 19:34:51,522: INFO/job_scheduler] Job 1967: complete_job: Updating cache
[2017-06-01 19:34:51,528: INFO/job_scheduler] Job 1967 completing (errored=False, cancelled=False)
[2017-06-01 19:34:51,572: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:34:51,626: INFO/job_scheduler] Job 1968 finished 1 steps successfully
[2017-06-01 19:34:51,689: INFO/job_scheduler] Job 1968: complete_job: Updating cache
[2017-06-01 19:34:51,690: INFO/job_scheduler] Job 1968 completing (errored=False, cancelled=False)
[2017-06-01 19:34:51,750: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:34:51,887: INFO/job_scheduler] Job 1969 finished 1 steps successfully
[2017-06-01 19:34:51,958: INFO/job_scheduler] Job 1969: complete_job: Updating cache
[2017-06-01 19:34:51,958: INFO/job_scheduler] Job 1969 completing (errored=False, cancelled=False)
[2017-06-01 19:34:52,015: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:34:52,119: INFO/job_scheduler] Job 1970 finished 1 steps successfully
[2017-06-01 19:34:52,164: INFO/job_scheduler] Job 1970: complete_job: Updating cache
[2017-06-01 19:34:52,166: INFO/job_scheduler] Job 1970 completing (errored=False, cancelled=False)
[2017-06-01 19:34:52,235: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:34:53,913: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:53,913: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:53,922: INFO/command_plan] add_jobs: created Job 1971 (Test for host connectivity)
[2017-06-01 19:34:54,010: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:34:54,026: INFO/job_scheduler] Job 1971: RunJobThread.run
[2017-06-01 19:34:54,118: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:54,121: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:54,139: INFO/command_plan] add_jobs: created Job 1972 (Test for host connectivity)
[2017-06-01 19:34:54,168: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:34:54,183: INFO/job_scheduler] Job 1972: RunJobThread.run
[2017-06-01 19:34:54,267: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:54,267: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:54,284: INFO/command_plan] add_jobs: created Job 1973 (Test for host connectivity)
[2017-06-01 19:34:54,336: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:34:54,349: INFO/job_scheduler] Job 1973: RunJobThread.run
[2017-06-01 19:34:54,464: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:34:54,464: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:34:54,480: INFO/command_plan] add_jobs: created Job 1974 (Test for host connectivity)
[2017-06-01 19:34:54,584: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:34:54,605: INFO/job_scheduler] Job 1974: RunJobThread.run
[2017-06-01 19:34:56,439: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:34:56,547: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:34:56,843: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:34:56,963: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:34:57,787: INFO/job_scheduler] Job 1971 finished 1 steps successfully
[2017-06-01 19:34:57,820: INFO/job_scheduler] Job 1971: complete_job: Updating cache
[2017-06-01 19:34:57,820: INFO/job_scheduler] Job 1971 completing (errored=False, cancelled=False)
[2017-06-01 19:34:57,868: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:34:57,884: INFO/job_scheduler] Job 1972 finished 1 steps successfully
[2017-06-01 19:34:58,015: INFO/job_scheduler] Job 1972: complete_job: Updating cache
[2017-06-01 19:34:58,016: INFO/job_scheduler] Job 1972 completing (errored=False, cancelled=False)
[2017-06-01 19:34:58,178: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:34:58,376: INFO/job_scheduler] Job 1973 finished 1 steps successfully
[2017-06-01 19:34:58,403: INFO/job_scheduler] Job 1973: complete_job: Updating cache
[2017-06-01 19:34:58,403: INFO/job_scheduler] Job 1973 completing (errored=False, cancelled=False)
[2017-06-01 19:34:58,457: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:34:58,468: INFO/job_scheduler] Job 1974 finished 1 steps successfully
[2017-06-01 19:34:58,492: INFO/job_scheduler] Job 1974: complete_job: Updating cache
[2017-06-01 19:34:58,492: INFO/job_scheduler] Job 1974 completing (errored=False, cancelled=False)
[2017-06-01 19:34:58,554: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:35:00,220: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:00,220: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:00,239: INFO/command_plan] add_jobs: created Job 1975 (Test for host connectivity)
[2017-06-01 19:35:00,281: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:35:00,283: INFO/job_scheduler] Job 1975: RunJobThread.run
[2017-06-01 19:35:00,367: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:00,368: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:00,390: INFO/command_plan] add_jobs: created Job 1976 (Test for host connectivity)
[2017-06-01 19:35:00,458: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:35:00,460: INFO/job_scheduler] Job 1976: RunJobThread.run
[2017-06-01 19:35:00,561: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:00,561: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:00,589: INFO/command_plan] add_jobs: created Job 1977 (Test for host connectivity)
[2017-06-01 19:35:00,672: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:35:00,710: INFO/job_scheduler] Job 1977: RunJobThread.run
[2017-06-01 19:35:00,745: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:00,746: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:00,761: INFO/command_plan] add_jobs: created Job 1978 (Test for host connectivity)
[2017-06-01 19:35:00,923: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:35:00,926: INFO/job_scheduler] Job 1978: RunJobThread.run
[2017-06-01 19:35:02,706: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:35:02,862: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:35:02,945: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:35:03,185: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:35:04,037: INFO/job_scheduler] Job 1975 finished 1 steps successfully
[2017-06-01 19:35:04,069: INFO/job_scheduler] Job 1975: complete_job: Updating cache
[2017-06-01 19:35:04,069: INFO/job_scheduler] Job 1975 completing (errored=False, cancelled=False)
[2017-06-01 19:35:04,150: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:35:04,261: INFO/job_scheduler] Job 1976 finished 1 steps successfully
[2017-06-01 19:35:04,295: INFO/job_scheduler] Job 1977 finished 1 steps successfully
[2017-06-01 19:35:04,317: INFO/job_scheduler] Job 1976: complete_job: Updating cache
[2017-06-01 19:35:04,320: INFO/job_scheduler] Job 1976 completing (errored=False, cancelled=False)
[2017-06-01 19:35:04,436: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:35:04,453: INFO/job_scheduler] Job 1977: complete_job: Updating cache
[2017-06-01 19:35:04,454: INFO/job_scheduler] Job 1977 completing (errored=False, cancelled=False)
[2017-06-01 19:35:04,510: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:35:04,680: INFO/job_scheduler] Job 1978 finished 1 steps successfully
[2017-06-01 19:35:04,753: INFO/job_scheduler] Job 1978: complete_job: Updating cache
[2017-06-01 19:35:04,754: INFO/job_scheduler] Job 1978 completing (errored=False, cancelled=False)
[2017-06-01 19:35:04,811: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:35:06,509: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:06,510: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:06,518: INFO/command_plan] add_jobs: created Job 1979 (Test for host connectivity)
[2017-06-01 19:35:06,663: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:06,664: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:06,714: INFO/command_plan] add_jobs: created Job 1980 (Test for host connectivity)
[2017-06-01 19:35:06,732: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:35:06,787: INFO/job_scheduler] Job 1979: RunJobThread.run
[2017-06-01 19:35:06,817: INFO/job_scheduler] Job 1980: RunJobThread.run
[2017-06-01 19:35:06,883: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:06,884: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:06,903: INFO/command_plan] add_jobs: created Job 1981 (Test for host connectivity)
[2017-06-01 19:35:06,922: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:35:07,022: INFO/job_scheduler] Job 1981: RunJobThread.run
[2017-06-01 19:35:07,143: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:07,147: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:07,175: INFO/command_plan] add_jobs: created Job 1982 (Test for host connectivity)
[2017-06-01 19:35:07,210: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:35:07,249: INFO/job_scheduler] Job 1982: RunJobThread.run
[2017-06-01 19:35:07,279: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:35:09,181: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:35:09,316: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:35:09,355: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:35:09,505: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:35:10,561: INFO/job_scheduler] Job 1979 finished 1 steps successfully
[2017-06-01 19:35:10,592: INFO/job_scheduler] Job 1979: complete_job: Updating cache
[2017-06-01 19:35:10,592: INFO/job_scheduler] Job 1979 completing (errored=False, cancelled=False)
[2017-06-01 19:35:10,663: INFO/job_scheduler] Job 1980 finished 1 steps successfully
[2017-06-01 19:35:10,668: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:35:10,806: INFO/job_scheduler] Job 1981 finished 1 steps successfully
[2017-06-01 19:35:10,822: INFO/job_scheduler] Job 1980: complete_job: Updating cache
[2017-06-01 19:35:10,824: INFO/job_scheduler] Job 1980 completing (errored=False, cancelled=False)
[2017-06-01 19:35:10,887: INFO/job_scheduler] Job 1982 finished 1 steps successfully
[2017-06-01 19:35:10,893: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:35:10,909: INFO/job_scheduler] Job 1981: complete_job: Updating cache
[2017-06-01 19:35:10,909: INFO/job_scheduler] Job 1981 completing (errored=False, cancelled=False)
[2017-06-01 19:35:10,977: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:35:10,992: INFO/job_scheduler] Job 1982: complete_job: Updating cache
[2017-06-01 19:35:10,992: INFO/job_scheduler] Job 1982 completing (errored=False, cancelled=False)
[2017-06-01 19:35:11,082: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:35:12,915: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:12,916: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:12,948: INFO/command_plan] add_jobs: created Job 1983 (Test for host connectivity)
[2017-06-01 19:35:13,017: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:35:13,059: INFO/job_scheduler] Job 1983: RunJobThread.run
[2017-06-01 19:35:13,094: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:13,094: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:13,110: INFO/command_plan] add_jobs: created Job 1984 (Test for host connectivity)
[2017-06-01 19:35:13,159: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:35:13,197: INFO/job_scheduler] Job 1984: RunJobThread.run
[2017-06-01 19:35:13,275: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:13,276: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:13,307: INFO/command_plan] add_jobs: created Job 1985 (Test for host connectivity)
[2017-06-01 19:35:13,334: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:35:13,426: INFO/job_scheduler] Job 1985: RunJobThread.run
[2017-06-01 19:35:13,469: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:13,473: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:13,592: INFO/command_plan] add_jobs: created Job 1986 (Test for host connectivity)
[2017-06-01 19:35:13,644: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:35:13,681: INFO/job_scheduler] Job 1986: RunJobThread.run
[2017-06-01 19:35:15,406: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:35:15,613: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:35:15,672: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:35:15,887: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:35:16,733: INFO/job_scheduler] Job 1983 finished 1 steps successfully
[2017-06-01 19:35:16,774: INFO/job_scheduler] Job 1983: complete_job: Updating cache
[2017-06-01 19:35:16,775: INFO/job_scheduler] Job 1983 completing (errored=False, cancelled=False)
[2017-06-01 19:35:16,810: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:35:16,955: INFO/job_scheduler] Job 1984 finished 1 steps successfully
[2017-06-01 19:35:17,016: INFO/job_scheduler] Job 1984: complete_job: Updating cache
[2017-06-01 19:35:17,016: INFO/job_scheduler] Job 1984 completing (errored=False, cancelled=False)
[2017-06-01 19:35:17,068: INFO/job_scheduler] Job 1985 finished 1 steps successfully
[2017-06-01 19:35:17,103: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:35:17,155: INFO/job_scheduler] Job 1985: complete_job: Updating cache
[2017-06-01 19:35:17,156: INFO/job_scheduler] Job 1985 completing (errored=False, cancelled=False)
[2017-06-01 19:35:17,233: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:35:17,375: INFO/job_scheduler] Job 1986 finished 1 steps successfully
[2017-06-01 19:35:17,424: INFO/job_scheduler] Job 1986: complete_job: Updating cache
[2017-06-01 19:35:17,424: INFO/job_scheduler] Job 1986 completing (errored=False, cancelled=False)
[2017-06-01 19:35:17,508: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:35:18,722: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:18,724: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:18,732: INFO/command_plan] add_jobs: created Job 1987 (Test for host connectivity)
[2017-06-01 19:35:18,807: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:35:18,810: INFO/job_scheduler] Job 1987: RunJobThread.run
[2017-06-01 19:35:18,890: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:18,891: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:18,899: INFO/command_plan] add_jobs: created Job 1988 (Test for host connectivity)
[2017-06-01 19:35:18,958: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:35:18,984: INFO/job_scheduler] Job 1988: RunJobThread.run
[2017-06-01 19:35:19,015: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:19,017: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:19,037: INFO/command_plan] add_jobs: created Job 1989 (Test for host connectivity)
[2017-06-01 19:35:19,088: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:35:19,118: INFO/job_scheduler] Job 1989: RunJobThread.run
[2017-06-01 19:35:19,226: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:19,227: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:19,245: INFO/command_plan] add_jobs: created Job 1990 (Test for host connectivity)
[2017-06-01 19:35:19,347: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:35:19,364: INFO/job_scheduler] Job 1990: RunJobThread.run
[2017-06-01 19:35:21,418: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:35:21,530: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:35:21,629: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:35:21,755: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:35:22,783: INFO/job_scheduler] Job 1988 finished 1 steps successfully
[2017-06-01 19:35:22,848: INFO/job_scheduler] Job 1988: complete_job: Updating cache
[2017-06-01 19:35:22,848: INFO/job_scheduler] Job 1988 completing (errored=False, cancelled=False)
[2017-06-01 19:35:22,920: INFO/job_scheduler] Job 1987 finished 1 steps successfully
[2017-06-01 19:35:22,923: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:35:22,956: INFO/job_scheduler] Job 1989 finished 1 steps successfully
[2017-06-01 19:35:23,021: INFO/job_scheduler] Job 1987: complete_job: Updating cache
[2017-06-01 19:35:23,022: INFO/job_scheduler] Job 1987 completing (errored=False, cancelled=False)
[2017-06-01 19:35:23,095: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:35:23,101: INFO/job_scheduler] Job 1990 finished 1 steps successfully
[2017-06-01 19:35:23,115: INFO/job_scheduler] Job 1989: complete_job: Updating cache
[2017-06-01 19:35:23,147: INFO/job_scheduler] Job 1989 completing (errored=False, cancelled=False)
[2017-06-01 19:35:23,233: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:35:23,288: INFO/job_scheduler] Job 1990: complete_job: Updating cache
[2017-06-01 19:35:23,288: INFO/job_scheduler] Job 1990 completing (errored=False, cancelled=False)
[2017-06-01 19:35:23,448: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:35:25,017: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:25,017: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:25,032: INFO/command_plan] add_jobs: created Job 1991 (Test for host connectivity)
[2017-06-01 19:35:25,105: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:35:25,147: INFO/job_scheduler] Job 1991: RunJobThread.run
[2017-06-01 19:35:25,216: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:25,216: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:25,241: INFO/command_plan] add_jobs: created Job 1992 (Test for host connectivity)
[2017-06-01 19:35:25,278: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:35:25,307: INFO/job_scheduler] Job 1992: RunJobThread.run
[2017-06-01 19:35:25,501: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:25,501: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:25,532: INFO/command_plan] add_jobs: created Job 1993 (Test for host connectivity)
[2017-06-01 19:35:25,579: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:35:25,620: INFO/job_scheduler] Job 1993: RunJobThread.run
[2017-06-01 19:35:25,719: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:25,719: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:25,751: INFO/command_plan] add_jobs: created Job 1994 (Test for host connectivity)
[2017-06-01 19:35:25,843: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:35:25,889: INFO/job_scheduler] Job 1994: RunJobThread.run
[2017-06-01 19:35:27,581: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:35:27,804: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:35:27,920: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:35:28,221: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:35:28,907: INFO/job_scheduler] Job 1991 finished 1 steps successfully
[2017-06-01 19:35:28,938: INFO/job_scheduler] Job 1991: complete_job: Updating cache
[2017-06-01 19:35:28,939: INFO/job_scheduler] Job 1991 completing (errored=False, cancelled=False)
[2017-06-01 19:35:28,972: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:35:29,233: INFO/job_scheduler] Job 1992 finished 1 steps successfully
[2017-06-01 19:35:29,322: INFO/job_scheduler] Job 1992: complete_job: Updating cache
[2017-06-01 19:35:29,322: INFO/job_scheduler] Job 1992 completing (errored=False, cancelled=False)
[2017-06-01 19:35:29,357: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:35:29,497: INFO/job_scheduler] Job 1993 finished 1 steps successfully
[2017-06-01 19:35:29,565: INFO/job_scheduler] Job 1993: complete_job: Updating cache
[2017-06-01 19:35:29,565: INFO/job_scheduler] Job 1993 completing (errored=False, cancelled=False)
[2017-06-01 19:35:29,621: INFO/job_scheduler] Job 1994 finished 1 steps successfully
[2017-06-01 19:35:29,630: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:35:29,645: INFO/job_scheduler] Job 1994: complete_job: Updating cache
[2017-06-01 19:35:29,645: INFO/job_scheduler] Job 1994 completing (errored=False, cancelled=False)
[2017-06-01 19:35:29,725: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:35:31,545: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:31,546: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:31,554: INFO/command_plan] add_jobs: created Job 1995 (Test for host connectivity)
[2017-06-01 19:35:31,657: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:31,658: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:31,673: INFO/command_plan] add_jobs: created Job 1996 (Test for host connectivity)
[2017-06-01 19:35:31,697: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:35:31,709: INFO/job_scheduler] Job 1995: RunJobThread.run
[2017-06-01 19:35:31,726: INFO/job_scheduler] Job 1996: RunJobThread.run
[2017-06-01 19:35:31,754: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:35:31,825: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:31,825: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:31,859: INFO/command_plan] add_jobs: created Job 1997 (Test for host connectivity)
[2017-06-01 19:35:31,932: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:35:31,972: INFO/job_scheduler] Job 1997: RunJobThread.run
[2017-06-01 19:35:32,044: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:32,049: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:32,085: INFO/command_plan] add_jobs: created Job 1998 (Test for host connectivity)
[2017-06-01 19:35:32,130: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:35:32,209: INFO/job_scheduler] Job 1998: RunJobThread.run
[2017-06-01 19:35:34,118: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:35:34,232: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:35:34,353: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:35:34,511: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:35:35,471: INFO/job_scheduler] Job 1995 finished 1 steps successfully
[2017-06-01 19:35:35,512: INFO/job_scheduler] Job 1995: complete_job: Updating cache
[2017-06-01 19:35:35,512: INFO/job_scheduler] Job 1995 completing (errored=False, cancelled=False)
[2017-06-01 19:35:35,555: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:35:35,616: INFO/job_scheduler] Job 1996 finished 1 steps successfully
[2017-06-01 19:35:35,663: INFO/job_scheduler] Job 1996: complete_job: Updating cache
[2017-06-01 19:35:35,663: INFO/job_scheduler] Job 1996 completing (errored=False, cancelled=False)
[2017-06-01 19:35:35,711: INFO/job_scheduler] Job 1997 finished 1 steps successfully
[2017-06-01 19:35:35,751: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:35:35,763: INFO/job_scheduler] Job 1997: complete_job: Updating cache
[2017-06-01 19:35:35,763: INFO/job_scheduler] Job 1997 completing (errored=False, cancelled=False)
[2017-06-01 19:35:35,863: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:35:35,967: INFO/job_scheduler] Job 1998 finished 1 steps successfully
[2017-06-01 19:35:35,987: INFO/job_scheduler] Job 1998: complete_job: Updating cache
[2017-06-01 19:35:35,987: INFO/job_scheduler] Job 1998 completing (errored=False, cancelled=False)
[2017-06-01 19:35:36,039: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:35:37,869: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:37,869: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:37,886: INFO/command_plan] add_jobs: created Job 1999 (Test for host connectivity)
[2017-06-01 19:35:37,947: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:35:37,973: INFO/job_scheduler] Job 1999: RunJobThread.run
[2017-06-01 19:35:38,067: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:38,073: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:38,087: INFO/command_plan] add_jobs: created Job 2000 (Test for host connectivity)
[2017-06-01 19:35:38,141: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:35:38,182: INFO/job_scheduler] Job 2000: RunJobThread.run
[2017-06-01 19:35:38,272: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:38,272: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:38,288: INFO/command_plan] add_jobs: created Job 2001 (Test for host connectivity)
[2017-06-01 19:35:38,326: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:35:38,379: INFO/job_scheduler] Job 2001: RunJobThread.run
[2017-06-01 19:35:38,426: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:38,426: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:38,450: INFO/command_plan] add_jobs: created Job 2002 (Test for host connectivity)
[2017-06-01 19:35:38,483: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:35:38,524: INFO/job_scheduler] Job 2002: RunJobThread.run
[2017-06-01 19:35:40,456: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:35:40,553: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:35:40,716: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:35:40,898: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:35:41,809: INFO/job_scheduler] Job 1999 finished 1 steps successfully
[2017-06-01 19:35:41,864: INFO/job_scheduler] Job 2000 finished 1 steps successfully
[2017-06-01 19:35:41,885: INFO/job_scheduler] Job 1999: complete_job: Updating cache
[2017-06-01 19:35:41,887: INFO/job_scheduler] Job 1999 completing (errored=False, cancelled=False)
[2017-06-01 19:35:41,964: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:35:41,989: INFO/job_scheduler] Job 2000: complete_job: Updating cache
[2017-06-01 19:35:41,990: INFO/job_scheduler] Job 2000 completing (errored=False, cancelled=False)
[2017-06-01 19:35:42,082: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:35:42,207: INFO/job_scheduler] Job 2001 finished 1 steps successfully
[2017-06-01 19:35:42,244: INFO/job_scheduler] Job 2001: complete_job: Updating cache
[2017-06-01 19:35:42,244: INFO/job_scheduler] Job 2001 completing (errored=False, cancelled=False)
[2017-06-01 19:35:42,260: INFO/job_scheduler] Job 2002 finished 1 steps successfully
[2017-06-01 19:35:42,394: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:35:42,438: INFO/job_scheduler] Job 2002: complete_job: Updating cache
[2017-06-01 19:35:42,438: INFO/job_scheduler] Job 2002 completing (errored=False, cancelled=False)
[2017-06-01 19:35:42,511: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:35:44,176: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:44,177: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:44,202: INFO/command_plan] add_jobs: created Job 2003 (Test for host connectivity)
[2017-06-01 19:35:44,232: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:35:44,242: INFO/job_scheduler] Job 2003: RunJobThread.run
[2017-06-01 19:35:44,346: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:44,346: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:44,365: INFO/command_plan] add_jobs: created Job 2004 (Test for host connectivity)
[2017-06-01 19:35:44,411: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:35:44,449: INFO/job_scheduler] Job 2004: RunJobThread.run
[2017-06-01 19:35:44,503: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:44,503: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:44,634: INFO/command_plan] add_jobs: created Job 2005 (Test for host connectivity)
[2017-06-01 19:35:44,688: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:35:44,741: INFO/job_scheduler] Job 2005: RunJobThread.run
[2017-06-01 19:35:44,804: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:44,804: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:44,839: INFO/command_plan] add_jobs: created Job 2006 (Test for host connectivity)
[2017-06-01 19:35:44,925: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:35:44,945: INFO/job_scheduler] Job 2006: RunJobThread.run
[2017-06-01 19:35:46,568: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:35:46,768: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:35:47,100: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:35:47,125: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:35:47,912: INFO/job_scheduler] Job 2003 finished 1 steps successfully
[2017-06-01 19:35:47,926: INFO/job_scheduler] Job 2003: complete_job: Updating cache
[2017-06-01 19:35:47,926: INFO/job_scheduler] Job 2003 completing (errored=False, cancelled=False)
[2017-06-01 19:35:47,962: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:35:48,080: INFO/job_scheduler] Job 2004 finished 1 steps successfully
[2017-06-01 19:35:48,111: INFO/job_scheduler] Job 2004: complete_job: Updating cache
[2017-06-01 19:35:48,111: INFO/job_scheduler] Job 2004 completing (errored=False, cancelled=False)
[2017-06-01 19:35:48,197: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:35:48,557: INFO/job_scheduler] Job 2006 finished 1 steps successfully
[2017-06-01 19:35:48,584: INFO/job_scheduler] Job 2006: complete_job: Updating cache
[2017-06-01 19:35:48,585: INFO/job_scheduler] Job 2006 completing (errored=False, cancelled=False)
[2017-06-01 19:35:48,649: INFO/job_scheduler] Job 2005 finished 1 steps successfully
[2017-06-01 19:35:48,686: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:35:48,719: INFO/job_scheduler] Job 2005: complete_job: Updating cache
[2017-06-01 19:35:48,730: INFO/job_scheduler] Job 2005 completing (errored=False, cancelled=False)
[2017-06-01 19:35:48,827: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:35:50,539: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:50,539: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:50,561: INFO/command_plan] add_jobs: created Job 2007 (Test for host connectivity)
[2017-06-01 19:35:50,606: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:35:50,651: INFO/job_scheduler] Job 2007: RunJobThread.run
[2017-06-01 19:35:50,710: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:50,711: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:50,726: INFO/command_plan] add_jobs: created Job 2008 (Test for host connectivity)
[2017-06-01 19:35:50,809: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:35:50,827: INFO/job_scheduler] Job 2008: RunJobThread.run
[2017-06-01 19:35:50,924: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:50,925: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:50,960: INFO/command_plan] add_jobs: created Job 2009 (Test for host connectivity)
[2017-06-01 19:35:51,017: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:35:51,051: INFO/job_scheduler] Job 2009: RunJobThread.run
[2017-06-01 19:35:51,118: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:51,119: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:51,134: INFO/command_plan] add_jobs: created Job 2010 (Test for host connectivity)
[2017-06-01 19:35:51,188: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:35:51,245: INFO/job_scheduler] Job 2010: RunJobThread.run
[2017-06-01 19:35:53,161: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:35:53,245: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:35:53,291: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:35:53,557: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:35:54,448: INFO/job_scheduler] Job 2007 finished 1 steps successfully
[2017-06-01 19:35:54,516: INFO/job_scheduler] Job 2007: complete_job: Updating cache
[2017-06-01 19:35:54,517: INFO/job_scheduler] Job 2007 completing (errored=False, cancelled=False)
[2017-06-01 19:35:54,609: INFO/job_scheduler] Job 2008 finished 1 steps successfully
[2017-06-01 19:35:54,618: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:35:54,650: INFO/job_scheduler] Job 2008: complete_job: Updating cache
[2017-06-01 19:35:54,650: INFO/job_scheduler] Job 2008 completing (errored=False, cancelled=False)
[2017-06-01 19:35:54,788: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:35:54,844: INFO/job_scheduler] Job 2009 finished 1 steps successfully
[2017-06-01 19:35:54,873: INFO/job_scheduler] Job 2009: complete_job: Updating cache
[2017-06-01 19:35:54,873: INFO/job_scheduler] Job 2009 completing (errored=False, cancelled=False)
[2017-06-01 19:35:54,920: INFO/job_scheduler] Job 2010 finished 1 steps successfully
[2017-06-01 19:35:54,953: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:35:54,977: INFO/job_scheduler] Job 2010: complete_job: Updating cache
[2017-06-01 19:35:54,977: INFO/job_scheduler] Job 2010 completing (errored=False, cancelled=False)
[2017-06-01 19:35:55,016: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:35:56,905: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:56,906: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:56,925: INFO/command_plan] add_jobs: created Job 2011 (Test for host connectivity)
[2017-06-01 19:35:57,028: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:35:57,054: INFO/job_scheduler] Job 2011: RunJobThread.run
[2017-06-01 19:35:57,100: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:57,102: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:57,118: INFO/command_plan] add_jobs: created Job 2012 (Test for host connectivity)
[2017-06-01 19:35:57,188: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:35:57,245: INFO/job_scheduler] Job 2012: RunJobThread.run
[2017-06-01 19:35:57,409: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:57,409: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:57,436: INFO/command_plan] add_jobs: created Job 2013 (Test for host connectivity)
[2017-06-01 19:35:57,496: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:35:57,498: INFO/job_scheduler] Job 2013: RunJobThread.run
[2017-06-01 19:35:57,624: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:35:57,626: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:35:57,652: INFO/command_plan] add_jobs: created Job 2014 (Test for host connectivity)
[2017-06-01 19:35:57,761: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:35:57,772: INFO/job_scheduler] Job 2014: RunJobThread.run
[2017-06-01 19:35:59,450: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:35:59,653: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:35:59,895: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:36:00,046: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:36:00,822: INFO/job_scheduler] Job 2011 finished 1 steps successfully
[2017-06-01 19:36:00,849: INFO/job_scheduler] Job 2011: complete_job: Updating cache
[2017-06-01 19:36:00,849: INFO/job_scheduler] Job 2011 completing (errored=False, cancelled=False)
[2017-06-01 19:36:00,892: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:36:00,999: INFO/job_scheduler] Job 2012 finished 1 steps successfully
[2017-06-01 19:36:01,064: INFO/job_scheduler] Job 2012: complete_job: Updating cache
[2017-06-01 19:36:01,065: INFO/job_scheduler] Job 2012 completing (errored=False, cancelled=False)
[2017-06-01 19:36:01,225: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:36:01,280: INFO/job_scheduler] Job 2013 finished 1 steps successfully
[2017-06-01 19:36:01,292: INFO/job_scheduler] Job 2013: complete_job: Updating cache
[2017-06-01 19:36:01,292: INFO/job_scheduler] Job 2013 completing (errored=False, cancelled=False)
[2017-06-01 19:36:01,405: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:36:01,518: INFO/job_scheduler] Job 2014 finished 1 steps successfully
[2017-06-01 19:36:01,559: INFO/job_scheduler] Job 2014: complete_job: Updating cache
[2017-06-01 19:36:01,559: INFO/job_scheduler] Job 2014 completing (errored=False, cancelled=False)
[2017-06-01 19:36:01,608: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:36:03,455: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:03,456: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:03,465: INFO/command_plan] add_jobs: created Job 2015 (Test for host connectivity)
[2017-06-01 19:36:03,527: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:03,527: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:03,542: INFO/command_plan] add_jobs: created Job 2016 (Test for host connectivity)
[2017-06-01 19:36:03,549: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:36:03,613: INFO/job_scheduler] Job 2016: RunJobThread.run
[2017-06-01 19:36:03,654: INFO/job_scheduler] Job 2015: RunJobThread.run
[2017-06-01 19:36:03,702: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:03,707: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:03,726: INFO/command_plan] add_jobs: created Job 2017 (Test for host connectivity)
[2017-06-01 19:36:03,754: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:36:03,827: INFO/job_scheduler] Job 2017: RunJobThread.run
[2017-06-01 19:36:04,016: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:04,016: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:04,045: INFO/command_plan] add_jobs: created Job 2018 (Test for host connectivity)
[2017-06-01 19:36:04,074: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:36:04,106: INFO/job_scheduler] Job 2018: RunJobThread.run
[2017-06-01 19:36:04,167: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:36:06,012: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:36:06,025: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:36:06,175: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:36:06,515: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:36:07,355: INFO/job_scheduler] Job 2015 finished 1 steps successfully
[2017-06-01 19:36:07,381: INFO/job_scheduler] Job 2015: complete_job: Updating cache
[2017-06-01 19:36:07,381: INFO/job_scheduler] Job 2015 completing (errored=False, cancelled=False)
[2017-06-01 19:36:07,451: INFO/job_scheduler] Job 2016 finished 1 steps successfully
[2017-06-01 19:36:07,476: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:36:07,498: INFO/job_scheduler] Job 2016: complete_job: Updating cache
[2017-06-01 19:36:07,498: INFO/job_scheduler] Job 2016 completing (errored=False, cancelled=False)
[2017-06-01 19:36:07,564: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:36:07,720: INFO/job_scheduler] Job 2017 finished 1 steps successfully
[2017-06-01 19:36:07,764: INFO/job_scheduler] Job 2017: complete_job: Updating cache
[2017-06-01 19:36:07,764: INFO/job_scheduler] Job 2017 completing (errored=False, cancelled=False)
[2017-06-01 19:36:07,831: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:36:07,922: INFO/job_scheduler] Job 2018 finished 1 steps successfully
[2017-06-01 19:36:07,966: INFO/job_scheduler] Job 2018: complete_job: Updating cache
[2017-06-01 19:36:07,966: INFO/job_scheduler] Job 2018 completing (errored=False, cancelled=False)
[2017-06-01 19:36:08,050: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:36:09,783: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:09,785: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:09,797: INFO/command_plan] add_jobs: created Job 2019 (Test for host connectivity)
[2017-06-01 19:36:09,821: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:36:09,835: INFO/job_scheduler] Job 2019: RunJobThread.run
[2017-06-01 19:36:09,893: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:09,893: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:09,914: INFO/command_plan] add_jobs: created Job 2020 (Test for host connectivity)
[2017-06-01 19:36:09,954: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:36:09,987: INFO/job_scheduler] Job 2020: RunJobThread.run
[2017-06-01 19:36:10,091: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:10,092: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:10,118: INFO/command_plan] add_jobs: created Job 2021 (Test for host connectivity)
[2017-06-01 19:36:10,212: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:36:10,286: INFO/job_scheduler] Job 2021: RunJobThread.run
[2017-06-01 19:36:10,343: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:10,344: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:10,375: INFO/command_plan] add_jobs: created Job 2022 (Test for host connectivity)
[2017-06-01 19:36:10,502: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:36:10,520: INFO/job_scheduler] Job 2022: RunJobThread.run
[2017-06-01 19:36:12,352: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:36:12,435: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:36:12,766: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:36:12,782: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:36:13,680: INFO/job_scheduler] Job 2019 finished 1 steps successfully
[2017-06-01 19:36:13,713: INFO/job_scheduler] Job 2019: complete_job: Updating cache
[2017-06-01 19:36:13,714: INFO/job_scheduler] Job 2019 completing (errored=False, cancelled=False)
[2017-06-01 19:36:13,755: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:36:13,765: INFO/job_scheduler] Job 2020 finished 1 steps successfully
[2017-06-01 19:36:13,793: INFO/job_scheduler] Job 2020: complete_job: Updating cache
[2017-06-01 19:36:13,793: INFO/job_scheduler] Job 2020 completing (errored=False, cancelled=False)
[2017-06-01 19:36:13,880: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:36:14,163: INFO/job_scheduler] Job 2022 finished 1 steps successfully
[2017-06-01 19:36:14,188: INFO/job_scheduler] Job 2022: complete_job: Updating cache
[2017-06-01 19:36:14,188: INFO/job_scheduler] Job 2022 completing (errored=False, cancelled=False)
[2017-06-01 19:36:14,239: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:36:14,317: INFO/job_scheduler] Job 2021 finished 1 steps successfully
[2017-06-01 19:36:14,353: INFO/job_scheduler] Job 2021: complete_job: Updating cache
[2017-06-01 19:36:14,359: INFO/job_scheduler] Job 2021 completing (errored=False, cancelled=False)
[2017-06-01 19:36:14,396: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:36:16,135: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:16,145: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:16,155: INFO/command_plan] add_jobs: created Job 2023 (Test for host connectivity)
[2017-06-01 19:36:16,165: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:36:16,187: INFO/job_scheduler] Job 2023: RunJobThread.run
[2017-06-01 19:36:16,227: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:16,227: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:16,265: INFO/command_plan] add_jobs: created Job 2024 (Test for host connectivity)
[2017-06-01 19:36:16,367: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:36:16,384: INFO/job_scheduler] Job 2024: RunJobThread.run
[2017-06-01 19:36:16,456: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:16,457: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:16,477: INFO/command_plan] add_jobs: created Job 2025 (Test for host connectivity)
[2017-06-01 19:36:16,569: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:36:16,583: INFO/job_scheduler] Job 2025: RunJobThread.run
[2017-06-01 19:36:16,614: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:16,615: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:16,648: INFO/command_plan] add_jobs: created Job 2026 (Test for host connectivity)
[2017-06-01 19:36:16,701: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:36:16,749: INFO/job_scheduler] Job 2026: RunJobThread.run
[2017-06-01 19:36:18,556: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:36:18,814: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:36:18,880: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:36:18,983: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:36:19,890: INFO/job_scheduler] Job 2023 finished 1 steps successfully
[2017-06-01 19:36:19,945: INFO/job_scheduler] Job 2023: complete_job: Updating cache
[2017-06-01 19:36:19,945: INFO/job_scheduler] Job 2023 completing (errored=False, cancelled=False)
[2017-06-01 19:36:20,020: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:36:20,229: INFO/job_scheduler] Job 2024 finished 1 steps successfully
[2017-06-01 19:36:20,260: INFO/job_scheduler] Job 2025 finished 1 steps successfully
[2017-06-01 19:36:20,360: INFO/job_scheduler] Job 2026 finished 1 steps successfully
[2017-06-01 19:36:20,403: INFO/job_scheduler] Job 2024: complete_job: Updating cache
[2017-06-01 19:36:20,403: INFO/job_scheduler] Job 2024 completing (errored=False, cancelled=False)
[2017-06-01 19:36:20,446: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:36:20,477: INFO/job_scheduler] Job 2025: complete_job: Updating cache
[2017-06-01 19:36:20,478: INFO/job_scheduler] Job 2025 completing (errored=False, cancelled=False)
[2017-06-01 19:36:20,644: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:36:20,675: INFO/job_scheduler] Job 2026: complete_job: Updating cache
[2017-06-01 19:36:20,675: INFO/job_scheduler] Job 2026 completing (errored=False, cancelled=False)
[2017-06-01 19:36:20,764: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:36:22,521: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:22,521: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:22,540: INFO/command_plan] add_jobs: created Job 2027 (Test for host connectivity)
[2017-06-01 19:36:22,635: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:22,636: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:22,646: INFO/command_plan] add_jobs: created Job 2028 (Test for host connectivity)
[2017-06-01 19:36:22,659: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:36:22,680: INFO/job_scheduler] Job 2027: RunJobThread.run
[2017-06-01 19:36:22,717: INFO/job_scheduler] Job 2028: RunJobThread.run
[2017-06-01 19:36:22,816: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:22,816: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:22,843: INFO/command_plan] add_jobs: created Job 2029 (Test for host connectivity)
[2017-06-01 19:36:22,859: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:36:22,861: INFO/job_scheduler] Job 2029: RunJobThread.run
[2017-06-01 19:36:22,977: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:22,984: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:23,008: INFO/command_plan] add_jobs: created Job 2030 (Test for host connectivity)
[2017-06-01 19:36:23,051: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:36:23,105: INFO/job_scheduler] Job 2030: RunJobThread.run
[2017-06-01 19:36:23,158: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:36:25,081: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:36:25,122: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:36:25,270: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:36:25,427: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:36:26,462: INFO/job_scheduler] Job 2027 finished 1 steps successfully
[2017-06-01 19:36:26,514: INFO/job_scheduler] Job 2027: complete_job: Updating cache
[2017-06-01 19:36:26,515: INFO/job_scheduler] Job 2027 completing (errored=False, cancelled=False)
[2017-06-01 19:36:26,522: INFO/job_scheduler] Job 2028 finished 1 steps successfully
[2017-06-01 19:36:26,576: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:36:26,612: INFO/job_scheduler] Job 2028: complete_job: Updating cache
[2017-06-01 19:36:26,612: INFO/job_scheduler] Job 2028 completing (errored=False, cancelled=False)
[2017-06-01 19:36:26,669: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:36:26,672: INFO/job_scheduler] Job 2029 finished 1 steps successfully
[2017-06-01 19:36:26,714: INFO/job_scheduler] Job 2029: complete_job: Updating cache
[2017-06-01 19:36:26,714: INFO/job_scheduler] Job 2029 completing (errored=False, cancelled=False)
[2017-06-01 19:36:26,750: INFO/job_scheduler] Job 2030 finished 1 steps successfully
[2017-06-01 19:36:26,825: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:36:26,859: INFO/job_scheduler] Job 2030: complete_job: Updating cache
[2017-06-01 19:36:26,862: INFO/job_scheduler] Job 2030 completing (errored=False, cancelled=False)
[2017-06-01 19:36:26,949: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:36:28,866: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:28,867: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:28,886: INFO/command_plan] add_jobs: created Job 2031 (Test for host connectivity)
[2017-06-01 19:36:28,922: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:36:28,958: INFO/job_scheduler] Job 2031: RunJobThread.run
[2017-06-01 19:36:29,024: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:29,026: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:29,051: INFO/command_plan] add_jobs: created Job 2032 (Test for host connectivity)
[2017-06-01 19:36:29,071: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:36:29,133: INFO/job_scheduler] Job 2032: RunJobThread.run
[2017-06-01 19:36:29,214: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:29,222: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:29,243: INFO/command_plan] add_jobs: created Job 2033 (Test for host connectivity)
[2017-06-01 19:36:29,293: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:36:29,325: INFO/job_scheduler] Job 2033: RunJobThread.run
[2017-06-01 19:36:29,374: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:29,374: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:29,387: INFO/command_plan] add_jobs: created Job 2034 (Test for host connectivity)
[2017-06-01 19:36:29,451: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:36:29,481: INFO/job_scheduler] Job 2034: RunJobThread.run
[2017-06-01 19:36:31,463: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:36:31,576: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:36:31,711: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:36:31,794: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:36:32,825: INFO/job_scheduler] Job 2031 finished 1 steps successfully
[2017-06-01 19:36:32,878: INFO/job_scheduler] Job 2031: complete_job: Updating cache
[2017-06-01 19:36:32,878: INFO/job_scheduler] Job 2031 completing (errored=False, cancelled=False)
[2017-06-01 19:36:32,907: INFO/job_scheduler] Job 2032 finished 1 steps successfully
[2017-06-01 19:36:32,912: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:36:32,940: INFO/job_scheduler] Job 2032: complete_job: Updating cache
[2017-06-01 19:36:32,940: INFO/job_scheduler] Job 2032 completing (errored=False, cancelled=False)
[2017-06-01 19:36:33,068: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:36:33,083: INFO/job_scheduler] Job 2033 finished 1 steps successfully
[2017-06-01 19:36:33,134: INFO/job_scheduler] Job 2033: complete_job: Updating cache
[2017-06-01 19:36:33,134: INFO/job_scheduler] Job 2033 completing (errored=False, cancelled=False)
[2017-06-01 19:36:33,183: INFO/job_scheduler] Job 2034 finished 1 steps successfully
[2017-06-01 19:36:33,183: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:36:33,196: INFO/job_scheduler] Job 2034: complete_job: Updating cache
[2017-06-01 19:36:33,196: INFO/job_scheduler] Job 2034 completing (errored=False, cancelled=False)
[2017-06-01 19:36:33,286: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:36:35,093: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:35,094: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:35,102: INFO/command_plan] add_jobs: created Job 2035 (Test for host connectivity)
[2017-06-01 19:36:35,220: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:35,222: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:35,237: INFO/command_plan] add_jobs: created Job 2036 (Test for host connectivity)
[2017-06-01 19:36:35,253: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:36:35,275: INFO/job_scheduler] Job 2035: RunJobThread.run
[2017-06-01 19:36:35,276: INFO/job_scheduler] Job 2036: RunJobThread.run
[2017-06-01 19:36:35,309: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:35,309: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:35,359: INFO/command_plan] add_jobs: created Job 2037 (Test for host connectivity)
[2017-06-01 19:36:35,526: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:36:35,542: INFO/job_scheduler] Job 2037: RunJobThread.run
[2017-06-01 19:36:35,606: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:35,607: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:35,627: INFO/command_plan] add_jobs: created Job 2038 (Test for host connectivity)
[2017-06-01 19:36:35,692: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:36:35,743: INFO/job_scheduler] Job 2038: RunJobThread.run
[2017-06-01 19:36:35,779: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:36:37,605: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:36:37,768: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:36:37,789: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:36:38,106: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:36:38,934: INFO/job_scheduler] Job 2036 finished 1 steps successfully
[2017-06-01 19:36:38,960: INFO/job_scheduler] Job 2036: complete_job: Updating cache
[2017-06-01 19:36:38,960: INFO/job_scheduler] Job 2036 completing (errored=False, cancelled=False)
[2017-06-01 19:36:39,092: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:36:39,135: INFO/job_scheduler] Job 2035 finished 1 steps successfully
[2017-06-01 19:36:39,160: INFO/job_scheduler] Job 2035: complete_job: Updating cache
[2017-06-01 19:36:39,160: INFO/job_scheduler] Job 2035 completing (errored=False, cancelled=False)
[2017-06-01 19:36:39,196: INFO/job_scheduler] Job 2037 finished 1 steps successfully
[2017-06-01 19:36:39,244: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:36:39,315: INFO/job_scheduler] Job 2037: complete_job: Updating cache
[2017-06-01 19:36:39,317: INFO/job_scheduler] Job 2037 completing (errored=False, cancelled=False)
[2017-06-01 19:36:39,371: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:36:39,543: INFO/job_scheduler] Job 2038 finished 1 steps successfully
[2017-06-01 19:36:39,627: INFO/job_scheduler] Job 2038: complete_job: Updating cache
[2017-06-01 19:36:39,627: INFO/job_scheduler] Job 2038 completing (errored=False, cancelled=False)
[2017-06-01 19:36:39,678: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:36:41,451: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:41,453: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:41,459: INFO/command_plan] add_jobs: created Job 2039 (Test for host connectivity)
[2017-06-01 19:36:41,522: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:36:41,581: INFO/job_scheduler] Job 2039: RunJobThread.run
[2017-06-01 19:36:41,617: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:41,619: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:41,637: INFO/command_plan] add_jobs: created Job 2040 (Test for host connectivity)
[2017-06-01 19:36:41,692: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:36:41,722: INFO/job_scheduler] Job 2040: RunJobThread.run
[2017-06-01 19:36:41,795: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:41,797: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:41,847: INFO/command_plan] add_jobs: created Job 2041 (Test for host connectivity)
[2017-06-01 19:36:41,955: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:36:41,964: INFO/job_scheduler] Job 2041: RunJobThread.run
[2017-06-01 19:36:42,020: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:42,020: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:42,052: INFO/command_plan] add_jobs: created Job 2042 (Test for host connectivity)
[2017-06-01 19:36:42,077: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:36:42,086: INFO/job_scheduler] Job 2042: RunJobThread.run
[2017-06-01 19:36:44,114: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:36:44,202: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:36:44,345: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:36:44,400: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:36:45,434: INFO/job_scheduler] Job 2039 finished 1 steps successfully
[2017-06-01 19:36:45,450: INFO/job_scheduler] Job 2039: complete_job: Updating cache
[2017-06-01 19:36:45,451: INFO/job_scheduler] Job 2039 completing (errored=False, cancelled=False)
[2017-06-01 19:36:45,543: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:36:45,591: INFO/job_scheduler] Job 2040 finished 1 steps successfully
[2017-06-01 19:36:45,609: INFO/job_scheduler] Job 2040: complete_job: Updating cache
[2017-06-01 19:36:45,609: INFO/job_scheduler] Job 2040 completing (errored=False, cancelled=False)
[2017-06-01 19:36:45,728: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:36:45,904: INFO/job_scheduler] Job 2041 finished 1 steps successfully
[2017-06-01 19:36:45,960: INFO/job_scheduler] Job 2041: complete_job: Updating cache
[2017-06-01 19:36:45,960: INFO/job_scheduler] Job 2041 completing (errored=False, cancelled=False)
[2017-06-01 19:36:45,966: INFO/job_scheduler] Job 2042 finished 1 steps successfully
[2017-06-01 19:36:46,008: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:36:46,017: INFO/job_scheduler] Job 2042: complete_job: Updating cache
[2017-06-01 19:36:46,017: INFO/job_scheduler] Job 2042 completing (errored=False, cancelled=False)
[2017-06-01 19:36:46,165: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:36:48,463: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:48,463: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:48,480: INFO/command_plan] add_jobs: created Job 2043 (Test for host connectivity)
[2017-06-01 19:36:48,515: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:36:48,519: INFO/job_scheduler] Job 2043: RunJobThread.run
[2017-06-01 19:36:48,569: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:48,570: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:48,586: INFO/command_plan] add_jobs: created Job 2044 (Test for host connectivity)
[2017-06-01 19:36:48,624: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:36:48,642: INFO/job_scheduler] Job 2044: RunJobThread.run
[2017-06-01 19:36:48,668: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:48,668: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:48,739: INFO/command_plan] add_jobs: created Job 2045 (Test for host connectivity)
[2017-06-01 19:36:48,839: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:36:48,855: INFO/job_scheduler] Job 2045: RunJobThread.run
[2017-06-01 19:36:48,972: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:48,974: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:49,003: INFO/command_plan] add_jobs: created Job 2046 (Test for host connectivity)
[2017-06-01 19:36:49,082: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:36:49,094: INFO/job_scheduler] Job 2046: RunJobThread.run
[2017-06-01 19:36:50,835: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:36:51,011: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:36:51,227: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:36:51,258: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:36:52,172: INFO/job_scheduler] Job 2043 finished 1 steps successfully
[2017-06-01 19:36:52,199: INFO/job_scheduler] Job 2043: complete_job: Updating cache
[2017-06-01 19:36:52,199: INFO/job_scheduler] Job 2043 completing (errored=False, cancelled=False)
[2017-06-01 19:36:52,234: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:36:52,356: INFO/job_scheduler] Job 2044 finished 1 steps successfully
[2017-06-01 19:36:52,441: INFO/job_scheduler] Job 2044: complete_job: Updating cache
[2017-06-01 19:36:52,446: INFO/job_scheduler] Job 2044 completing (errored=False, cancelled=False)
[2017-06-01 19:36:52,549: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:36:52,758: INFO/job_scheduler] Job 2046 finished 1 steps successfully
[2017-06-01 19:36:52,842: INFO/job_scheduler] Job 2046: complete_job: Updating cache
[2017-06-01 19:36:52,842: INFO/job_scheduler] Job 2046 completing (errored=False, cancelled=False)
[2017-06-01 19:36:52,853: INFO/job_scheduler] Job 2045 finished 1 steps successfully
[2017-06-01 19:36:52,909: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:36:52,924: INFO/job_scheduler] Job 2045: complete_job: Updating cache
[2017-06-01 19:36:52,927: INFO/job_scheduler] Job 2045 completing (errored=False, cancelled=False)
[2017-06-01 19:36:53,016: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:36:54,805: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:54,807: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:54,824: INFO/command_plan] add_jobs: created Job 2047 (Test for host connectivity)
[2017-06-01 19:36:54,885: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:36:54,899: INFO/job_scheduler] Job 2047: RunJobThread.run
[2017-06-01 19:36:54,953: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:54,953: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:54,969: INFO/command_plan] add_jobs: created Job 2048 (Test for host connectivity)
[2017-06-01 19:36:55,006: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:36:55,015: INFO/job_scheduler] Job 2048: RunJobThread.run
[2017-06-01 19:36:55,125: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:55,126: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:55,141: INFO/command_plan] add_jobs: created Job 2049 (Test for host connectivity)
[2017-06-01 19:36:55,294: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:36:55,318: INFO/job_scheduler] Job 2049: RunJobThread.run
[2017-06-01 19:36:55,375: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:36:55,377: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:36:55,398: INFO/command_plan] add_jobs: created Job 2050 (Test for host connectivity)
[2017-06-01 19:36:55,438: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:36:55,533: INFO/job_scheduler] Job 2050: RunJobThread.run
[2017-06-01 19:36:57,227: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:36:57,528: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:36:57,676: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:36:57,745: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:36:58,549: INFO/job_scheduler] Job 2047 finished 1 steps successfully
[2017-06-01 19:36:58,589: INFO/job_scheduler] Job 2047: complete_job: Updating cache
[2017-06-01 19:36:58,589: INFO/job_scheduler] Job 2047 completing (errored=False, cancelled=False)
[2017-06-01 19:36:58,639: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:36:59,031: INFO/job_scheduler] Job 2048 finished 1 steps successfully
[2017-06-01 19:36:59,047: INFO/job_scheduler] Job 2048: complete_job: Updating cache
[2017-06-01 19:36:59,048: INFO/job_scheduler] Job 2048 completing (errored=False, cancelled=False)
[2017-06-01 19:36:59,075: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:36:59,393: INFO/job_scheduler] Job 2050 finished 1 steps successfully
[2017-06-01 19:36:59,406: INFO/job_scheduler] Job 2050: complete_job: Updating cache
[2017-06-01 19:36:59,406: INFO/job_scheduler] Job 2050 completing (errored=False, cancelled=False)
[2017-06-01 19:36:59,488: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:36:59,493: INFO/job_scheduler] Job 2049 finished 1 steps successfully
[2017-06-01 19:36:59,506: INFO/job_scheduler] Job 2049: complete_job: Updating cache
[2017-06-01 19:36:59,525: INFO/job_scheduler] Job 2049 completing (errored=False, cancelled=False)
[2017-06-01 19:36:59,704: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:37:01,231: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:01,232: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:01,248: INFO/command_plan] add_jobs: created Job 2051 (Test for host connectivity)
[2017-06-01 19:37:01,282: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:37:01,319: INFO/job_scheduler] Job 2051: RunJobThread.run
[2017-06-01 19:37:01,376: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:01,377: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:01,403: INFO/command_plan] add_jobs: created Job 2052 (Test for host connectivity)
[2017-06-01 19:37:01,489: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:37:01,531: INFO/job_scheduler] Job 2052: RunJobThread.run
[2017-06-01 19:37:01,576: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:01,577: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:01,611: INFO/command_plan] add_jobs: created Job 2053 (Test for host connectivity)
[2017-06-01 19:37:01,690: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:37:01,693: INFO/job_scheduler] Job 2053: RunJobThread.run
[2017-06-01 19:37:01,763: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:01,763: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:01,801: INFO/command_plan] add_jobs: created Job 2054 (Test for host connectivity)
[2017-06-01 19:37:01,886: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:37:01,907: INFO/job_scheduler] Job 2054: RunJobThread.run
[2017-06-01 19:37:03,616: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:37:04,033: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:37:04,184: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:37:04,239: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:37:04,969: INFO/job_scheduler] Job 2051 finished 1 steps successfully
[2017-06-01 19:37:05,005: INFO/job_scheduler] Job 2051: complete_job: Updating cache
[2017-06-01 19:37:05,005: INFO/job_scheduler] Job 2051 completing (errored=False, cancelled=False)
[2017-06-01 19:37:05,031: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:37:05,478: INFO/job_scheduler] Job 2052 finished 1 steps successfully
[2017-06-01 19:37:05,513: INFO/job_scheduler] Job 2052: complete_job: Updating cache
[2017-06-01 19:37:05,513: INFO/job_scheduler] Job 2052 completing (errored=False, cancelled=False)
[2017-06-01 19:37:05,572: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:37:05,664: INFO/job_scheduler] Job 2053 finished 1 steps successfully
[2017-06-01 19:37:05,665: INFO/job_scheduler] Job 2054 finished 1 steps successfully
[2017-06-01 19:37:05,747: INFO/job_scheduler] Job 2053: complete_job: Updating cache
[2017-06-01 19:37:05,747: INFO/job_scheduler] Job 2053 completing (errored=False, cancelled=False)
[2017-06-01 19:37:05,796: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:37:05,798: INFO/job_scheduler] Job 2054: complete_job: Updating cache
[2017-06-01 19:37:05,799: INFO/job_scheduler] Job 2054 completing (errored=False, cancelled=False)
[2017-06-01 19:37:05,896: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:37:07,581: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:07,583: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:07,596: INFO/command_plan] add_jobs: created Job 2055 (Test for host connectivity)
[2017-06-01 19:37:07,631: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:37:07,643: INFO/job_scheduler] Job 2055: RunJobThread.run
[2017-06-01 19:37:07,690: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:07,690: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:07,721: INFO/command_plan] add_jobs: created Job 2056 (Test for host connectivity)
[2017-06-01 19:37:07,794: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:37:07,824: INFO/job_scheduler] Job 2056: RunJobThread.run
[2017-06-01 19:37:07,898: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:07,898: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:07,915: INFO/command_plan] add_jobs: created Job 2057 (Test for host connectivity)
[2017-06-01 19:37:07,965: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:37:07,976: INFO/job_scheduler] Job 2057: RunJobThread.run
[2017-06-01 19:37:08,077: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:08,077: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:08,099: INFO/command_plan] add_jobs: created Job 2058 (Test for host connectivity)
[2017-06-01 19:37:08,228: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:37:08,273: INFO/job_scheduler] Job 2058: RunJobThread.run
[2017-06-01 19:37:10,111: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:37:10,262: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:37:10,512: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:37:10,627: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:37:11,445: INFO/job_scheduler] Job 2055 finished 1 steps successfully
[2017-06-01 19:37:11,491: INFO/job_scheduler] Job 2055: complete_job: Updating cache
[2017-06-01 19:37:11,492: INFO/job_scheduler] Job 2055 completing (errored=False, cancelled=False)
[2017-06-01 19:37:11,557: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:37:11,612: INFO/job_scheduler] Job 2056 finished 1 steps successfully
[2017-06-01 19:37:11,669: INFO/job_scheduler] Job 2056: complete_job: Updating cache
[2017-06-01 19:37:11,669: INFO/job_scheduler] Job 2056 completing (errored=False, cancelled=False)
[2017-06-01 19:37:11,777: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:37:11,902: INFO/job_scheduler] Job 2057 finished 1 steps successfully
[2017-06-01 19:37:11,937: INFO/job_scheduler] Job 2057: complete_job: Updating cache
[2017-06-01 19:37:11,938: INFO/job_scheduler] Job 2057 completing (errored=False, cancelled=False)
[2017-06-01 19:37:11,979: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:37:12,074: INFO/job_scheduler] Job 2058 finished 1 steps successfully
[2017-06-01 19:37:12,114: INFO/job_scheduler] Job 2058: complete_job: Updating cache
[2017-06-01 19:37:12,114: INFO/job_scheduler] Job 2058 completing (errored=False, cancelled=False)
[2017-06-01 19:37:12,172: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:37:13,947: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:13,949: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:13,981: INFO/command_plan] add_jobs: created Job 2059 (Test for host connectivity)
[2017-06-01 19:37:14,054: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:37:14,056: INFO/job_scheduler] Job 2059: RunJobThread.run
[2017-06-01 19:37:14,105: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:14,106: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:14,121: INFO/command_plan] add_jobs: created Job 2060 (Test for host connectivity)
[2017-06-01 19:37:14,149: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:37:14,184: INFO/job_scheduler] Job 2060: RunJobThread.run
[2017-06-01 19:37:14,303: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:14,304: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:14,320: INFO/command_plan] add_jobs: created Job 2061 (Test for host connectivity)
[2017-06-01 19:37:14,364: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:37:14,412: INFO/job_scheduler] Job 2061: RunJobThread.run
[2017-06-01 19:37:14,550: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:14,551: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:14,589: INFO/command_plan] add_jobs: created Job 2062 (Test for host connectivity)
[2017-06-01 19:37:14,650: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:37:14,684: INFO/job_scheduler] Job 2062: RunJobThread.run
[2017-06-01 19:37:16,400: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:37:16,613: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:37:16,714: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:37:16,895: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:37:17,765: INFO/job_scheduler] Job 2059 finished 1 steps successfully
[2017-06-01 19:37:17,778: INFO/job_scheduler] Job 2059: complete_job: Updating cache
[2017-06-01 19:37:17,778: INFO/job_scheduler] Job 2059 completing (errored=False, cancelled=False)
[2017-06-01 19:37:17,869: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:37:17,988: INFO/job_scheduler] Job 2060 finished 1 steps successfully
[2017-06-01 19:37:18,062: INFO/job_scheduler] Job 2060: complete_job: Updating cache
[2017-06-01 19:37:18,062: INFO/job_scheduler] Job 2060 completing (errored=False, cancelled=False)
[2017-06-01 19:37:18,115: INFO/job_scheduler] Job 2061 finished 1 steps successfully
[2017-06-01 19:37:18,129: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:37:18,161: INFO/job_scheduler] Job 2061: complete_job: Updating cache
[2017-06-01 19:37:18,161: INFO/job_scheduler] Job 2061 completing (errored=False, cancelled=False)
[2017-06-01 19:37:18,254: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:37:18,280: INFO/job_scheduler] Job 2062 finished 1 steps successfully
[2017-06-01 19:37:18,295: INFO/job_scheduler] Job 2062: complete_job: Updating cache
[2017-06-01 19:37:18,295: INFO/job_scheduler] Job 2062 completing (errored=False, cancelled=False)
[2017-06-01 19:37:18,354: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:37:20,348: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:20,348: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:20,361: INFO/command_plan] add_jobs: created Job 2063 (Test for host connectivity)
[2017-06-01 19:37:20,413: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:37:20,427: INFO/job_scheduler] Job 2063: RunJobThread.run
[2017-06-01 19:37:20,519: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:20,519: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:20,544: INFO/command_plan] add_jobs: created Job 2064 (Test for host connectivity)
[2017-06-01 19:37:20,627: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:37:20,655: INFO/job_scheduler] Job 2064: RunJobThread.run
[2017-06-01 19:37:20,709: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:20,710: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:20,728: INFO/command_plan] add_jobs: created Job 2065 (Test for host connectivity)
[2017-06-01 19:37:20,758: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:37:20,781: INFO/job_scheduler] Job 2065: RunJobThread.run
[2017-06-01 19:37:20,834: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:20,835: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:20,870: INFO/command_plan] add_jobs: created Job 2066 (Test for host connectivity)
[2017-06-01 19:37:20,920: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:37:20,938: INFO/job_scheduler] Job 2066: RunJobThread.run
[2017-06-01 19:37:22,809: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:37:22,944: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:37:23,160: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:37:23,250: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:37:24,118: INFO/job_scheduler] Job 2063 finished 1 steps successfully
[2017-06-01 19:37:24,161: INFO/job_scheduler] Job 2063: complete_job: Updating cache
[2017-06-01 19:37:24,161: INFO/job_scheduler] Job 2063 completing (errored=False, cancelled=False)
[2017-06-01 19:37:24,188: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:37:24,259: INFO/job_scheduler] Job 2064 finished 1 steps successfully
[2017-06-01 19:37:24,326: INFO/job_scheduler] Job 2064: complete_job: Updating cache
[2017-06-01 19:37:24,326: INFO/job_scheduler] Job 2064 completing (errored=False, cancelled=False)
[2017-06-01 19:37:24,419: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:37:24,650: INFO/job_scheduler] Job 2065 finished 1 steps successfully
[2017-06-01 19:37:24,681: INFO/job_scheduler] Job 2066 finished 1 steps successfully
[2017-06-01 19:37:24,694: INFO/job_scheduler] Job 2065: complete_job: Updating cache
[2017-06-01 19:37:24,694: INFO/job_scheduler] Job 2065 completing (errored=False, cancelled=False)
[2017-06-01 19:37:24,720: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:37:24,745: INFO/job_scheduler] Job 2066: complete_job: Updating cache
[2017-06-01 19:37:24,745: INFO/job_scheduler] Job 2066 completing (errored=False, cancelled=False)
[2017-06-01 19:37:24,848: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:37:26,644: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:26,644: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:26,661: INFO/command_plan] add_jobs: created Job 2067 (Test for host connectivity)
[2017-06-01 19:37:26,736: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:37:26,754: INFO/job_scheduler] Job 2067: RunJobThread.run
[2017-06-01 19:37:26,807: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:26,811: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:26,835: INFO/command_plan] add_jobs: created Job 2068 (Test for host connectivity)
[2017-06-01 19:37:26,917: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:37:26,921: INFO/job_scheduler] Job 2068: RunJobThread.run
[2017-06-01 19:37:26,977: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:26,980: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:26,995: INFO/command_plan] add_jobs: created Job 2069 (Test for host connectivity)
[2017-06-01 19:37:27,051: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:37:27,081: INFO/job_scheduler] Job 2069: RunJobThread.run
[2017-06-01 19:37:27,189: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:27,189: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:27,205: INFO/command_plan] add_jobs: created Job 2070 (Test for host connectivity)
[2017-06-01 19:37:27,314: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:37:27,317: INFO/job_scheduler] Job 2070: RunJobThread.run
[2017-06-01 19:37:29,213: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:37:29,271: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:37:29,380: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:37:29,548: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:37:30,521: INFO/job_scheduler] Job 2068 finished 1 steps successfully
[2017-06-01 19:37:30,542: INFO/job_scheduler] Job 2067 finished 1 steps successfully
[2017-06-01 19:37:30,552: INFO/job_scheduler] Job 2068: complete_job: Updating cache
[2017-06-01 19:37:30,552: INFO/job_scheduler] Job 2068 completing (errored=False, cancelled=False)
[2017-06-01 19:37:30,614: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:37:30,650: INFO/job_scheduler] Job 2067: complete_job: Updating cache
[2017-06-01 19:37:30,650: INFO/job_scheduler] Job 2067 completing (errored=False, cancelled=False)
[2017-06-01 19:37:30,694: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:37:30,809: INFO/job_scheduler] Job 2069 finished 1 steps successfully
[2017-06-01 19:37:30,895: INFO/job_scheduler] Job 2069: complete_job: Updating cache
[2017-06-01 19:37:30,895: INFO/job_scheduler] Job 2069 completing (errored=False, cancelled=False)
[2017-06-01 19:37:30,932: INFO/job_scheduler] Job 2070 finished 1 steps successfully
[2017-06-01 19:37:30,952: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:37:30,970: INFO/job_scheduler] Job 2070: complete_job: Updating cache
[2017-06-01 19:37:30,980: INFO/job_scheduler] Job 2070 completing (errored=False, cancelled=False)
[2017-06-01 19:37:31,069: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:37:33,069: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:33,069: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:33,076: INFO/command_plan] add_jobs: created Job 2071 (Test for host connectivity)
[2017-06-01 19:37:33,085: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:37:33,153: INFO/job_scheduler] Job 2071: RunJobThread.run
[2017-06-01 19:37:33,226: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:33,228: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:33,243: INFO/command_plan] add_jobs: created Job 2072 (Test for host connectivity)
[2017-06-01 19:37:33,260: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:37:33,276: INFO/job_scheduler] Job 2072: RunJobThread.run
[2017-06-01 19:37:33,410: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:33,411: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:33,446: INFO/command_plan] add_jobs: created Job 2073 (Test for host connectivity)
[2017-06-01 19:37:33,522: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:37:33,541: INFO/job_scheduler] Job 2073: RunJobThread.run
[2017-06-01 19:37:33,669: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:33,670: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:33,694: INFO/command_plan] add_jobs: created Job 2074 (Test for host connectivity)
[2017-06-01 19:37:33,722: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:37:33,806: INFO/job_scheduler] Job 2074: RunJobThread.run
[2017-06-01 19:37:35,598: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:37:35,744: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:37:35,927: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:37:36,110: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:37:36,950: INFO/job_scheduler] Job 2071 finished 1 steps successfully
[2017-06-01 19:37:37,009: INFO/job_scheduler] Job 2071: complete_job: Updating cache
[2017-06-01 19:37:37,009: INFO/job_scheduler] Job 2071 completing (errored=False, cancelled=False)
[2017-06-01 19:37:37,071: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:37:37,159: INFO/job_scheduler] Job 2072 finished 1 steps successfully
[2017-06-01 19:37:37,214: INFO/job_scheduler] Job 2072: complete_job: Updating cache
[2017-06-01 19:37:37,214: INFO/job_scheduler] Job 2072 completing (errored=False, cancelled=False)
[2017-06-01 19:37:37,288: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:37:37,348: INFO/job_scheduler] Job 2073 finished 1 steps successfully
[2017-06-01 19:37:37,410: INFO/job_scheduler] Job 2073: complete_job: Updating cache
[2017-06-01 19:37:37,410: INFO/job_scheduler] Job 2073 completing (errored=False, cancelled=False)
[2017-06-01 19:37:37,488: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:37:37,575: INFO/job_scheduler] Job 2074 finished 1 steps successfully
[2017-06-01 19:37:37,595: INFO/job_scheduler] Job 2074: complete_job: Updating cache
[2017-06-01 19:37:37,595: INFO/job_scheduler] Job 2074 completing (errored=False, cancelled=False)
[2017-06-01 19:37:37,651: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:37:39,462: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:39,464: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:39,477: INFO/command_plan] add_jobs: created Job 2075 (Test for host connectivity)
[2017-06-01 19:37:39,549: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:37:39,564: INFO/job_scheduler] Job 2075: RunJobThread.run
[2017-06-01 19:37:39,619: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:39,620: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:39,637: INFO/command_plan] add_jobs: created Job 2076 (Test for host connectivity)
[2017-06-01 19:37:39,737: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:37:39,760: INFO/job_scheduler] Job 2076: RunJobThread.run
[2017-06-01 19:37:39,796: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:39,797: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:39,920: INFO/command_plan] add_jobs: created Job 2077 (Test for host connectivity)
[2017-06-01 19:37:39,971: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:37:40,008: INFO/job_scheduler] Job 2077: RunJobThread.run
[2017-06-01 19:37:40,085: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:40,087: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:40,103: INFO/command_plan] add_jobs: created Job 2078 (Test for host connectivity)
[2017-06-01 19:37:40,226: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:37:40,244: INFO/job_scheduler] Job 2078: RunJobThread.run
[2017-06-01 19:37:41,943: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:37:42,061: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:37:42,299: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:37:42,536: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:37:43,261: INFO/job_scheduler] Job 2076 finished 1 steps successfully
[2017-06-01 19:37:43,292: INFO/job_scheduler] Job 2076: complete_job: Updating cache
[2017-06-01 19:37:43,292: INFO/job_scheduler] Job 2076 completing (errored=False, cancelled=False)
[2017-06-01 19:37:43,343: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:37:43,363: INFO/job_scheduler] Job 2075 finished 1 steps successfully
[2017-06-01 19:37:43,432: INFO/job_scheduler] Job 2075: complete_job: Updating cache
[2017-06-01 19:37:43,432: INFO/job_scheduler] Job 2075 completing (errored=False, cancelled=False)
[2017-06-01 19:37:43,460: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:37:43,676: INFO/job_scheduler] Job 2077 finished 1 steps successfully
[2017-06-01 19:37:43,701: INFO/job_scheduler] Job 2077: complete_job: Updating cache
[2017-06-01 19:37:43,701: INFO/job_scheduler] Job 2077 completing (errored=False, cancelled=False)
[2017-06-01 19:37:43,764: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:37:43,983: INFO/job_scheduler] Job 2078 finished 1 steps successfully
[2017-06-01 19:37:44,025: INFO/job_scheduler] Job 2078: complete_job: Updating cache
[2017-06-01 19:37:44,025: INFO/job_scheduler] Job 2078 completing (errored=False, cancelled=False)
[2017-06-01 19:37:44,076: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:37:45,946: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:45,947: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:45,958: INFO/command_plan] add_jobs: created Job 2079 (Test for host connectivity)
[2017-06-01 19:37:45,997: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:37:46,009: INFO/job_scheduler] Job 2079: RunJobThread.run
[2017-06-01 19:37:46,090: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:46,091: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:46,103: INFO/command_plan] add_jobs: created Job 2080 (Test for host connectivity)
[2017-06-01 19:37:46,146: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:37:46,151: INFO/job_scheduler] Job 2080: RunJobThread.run
[2017-06-01 19:37:46,305: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:46,306: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:46,317: INFO/command_plan] add_jobs: created Job 2081 (Test for host connectivity)
[2017-06-01 19:37:46,405: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:37:46,416: INFO/job_scheduler] Job 2081: RunJobThread.run
[2017-06-01 19:37:46,455: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:46,461: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:46,541: INFO/command_plan] add_jobs: created Job 2082 (Test for host connectivity)
[2017-06-01 19:37:46,606: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:37:46,645: INFO/job_scheduler] Job 2082: RunJobThread.run
[2017-06-01 19:37:48,394: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:37:48,609: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:37:48,762: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:37:48,911: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:37:49,692: INFO/job_scheduler] Job 2079 finished 1 steps successfully
[2017-06-01 19:37:49,749: INFO/job_scheduler] Job 2079: complete_job: Updating cache
[2017-06-01 19:37:49,750: INFO/job_scheduler] Job 2079 completing (errored=False, cancelled=False)
[2017-06-01 19:37:49,826: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:37:49,960: INFO/job_scheduler] Job 2080 finished 1 steps successfully
[2017-06-01 19:37:50,025: INFO/job_scheduler] Job 2080: complete_job: Updating cache
[2017-06-01 19:37:50,025: INFO/job_scheduler] Job 2080 completing (errored=False, cancelled=False)
[2017-06-01 19:37:50,086: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:37:50,183: INFO/job_scheduler] Job 2081 finished 1 steps successfully
[2017-06-01 19:37:50,246: INFO/job_scheduler] Job 2082 finished 1 steps successfully
[2017-06-01 19:37:50,267: INFO/job_scheduler] Job 2081: complete_job: Updating cache
[2017-06-01 19:37:50,268: INFO/job_scheduler] Job 2081 completing (errored=False, cancelled=False)
[2017-06-01 19:37:50,324: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:37:50,333: INFO/job_scheduler] Job 2082: complete_job: Updating cache
[2017-06-01 19:37:50,347: INFO/job_scheduler] Job 2082 completing (errored=False, cancelled=False)
[2017-06-01 19:37:50,409: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:37:52,321: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:52,323: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:52,360: INFO/command_plan] add_jobs: created Job 2083 (Test for host connectivity)
[2017-06-01 19:37:52,377: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:37:52,379: INFO/job_scheduler] Job 2083: RunJobThread.run
[2017-06-01 19:37:52,484: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:52,484: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:52,491: INFO/command_plan] add_jobs: created Job 2084 (Test for host connectivity)
[2017-06-01 19:37:52,541: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:37:52,591: INFO/job_scheduler] Job 2084: RunJobThread.run
[2017-06-01 19:37:52,689: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:52,690: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:52,715: INFO/command_plan] add_jobs: created Job 2085 (Test for host connectivity)
[2017-06-01 19:37:52,799: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:37:52,818: INFO/job_scheduler] Job 2085: RunJobThread.run
[2017-06-01 19:37:52,987: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:52,988: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:53,052: INFO/command_plan] add_jobs: created Job 2086 (Test for host connectivity)
[2017-06-01 19:37:53,099: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:37:53,124: INFO/job_scheduler] Job 2086: RunJobThread.run
[2017-06-01 19:37:54,861: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:37:54,962: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:37:55,096: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:37:55,508: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:37:56,279: INFO/job_scheduler] Job 2083 finished 1 steps successfully
[2017-06-01 19:37:56,310: INFO/job_scheduler] Job 2083: complete_job: Updating cache
[2017-06-01 19:37:56,310: INFO/job_scheduler] Job 2083 completing (errored=False, cancelled=False)
[2017-06-01 19:37:56,368: INFO/job_scheduler] Job 2084 finished 1 steps successfully
[2017-06-01 19:37:56,368: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:37:56,413: INFO/job_scheduler] Job 2084: complete_job: Updating cache
[2017-06-01 19:37:56,424: INFO/job_scheduler] Job 2084 completing (errored=False, cancelled=False)
[2017-06-01 19:37:56,588: INFO/job_scheduler] Job 2085 finished 1 steps successfully
[2017-06-01 19:37:56,607: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:37:56,627: INFO/job_scheduler] Job 2085: complete_job: Updating cache
[2017-06-01 19:37:56,627: INFO/job_scheduler] Job 2085 completing (errored=False, cancelled=False)
[2017-06-01 19:37:56,727: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:37:57,025: INFO/job_scheduler] Job 2086 finished 1 steps successfully
[2017-06-01 19:37:57,057: INFO/job_scheduler] Job 2086: complete_job: Updating cache
[2017-06-01 19:37:57,057: INFO/job_scheduler] Job 2086 completing (errored=False, cancelled=False)
[2017-06-01 19:37:57,083: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:37:58,848: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:58,848: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:58,876: INFO/command_plan] add_jobs: created Job 2087 (Test for host connectivity)
[2017-06-01 19:37:58,912: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:37:58,918: INFO/job_scheduler] Job 2087: RunJobThread.run
[2017-06-01 19:37:58,968: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:58,968: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:58,992: INFO/command_plan] add_jobs: created Job 2088 (Test for host connectivity)
[2017-06-01 19:37:59,078: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:37:59,114: INFO/job_scheduler] Job 2088: RunJobThread.run
[2017-06-01 19:37:59,182: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:59,183: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:59,198: INFO/command_plan] add_jobs: created Job 2089 (Test for host connectivity)
[2017-06-01 19:37:59,309: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:37:59,309: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:37:59,344: INFO/command_plan] add_jobs: created Job 2090 (Test for host connectivity)
[2017-06-01 19:37:59,367: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:37:59,394: INFO/job_scheduler] Job 2089: RunJobThread.run
[2017-06-01 19:37:59,468: INFO/job_scheduler] Job 2090: RunJobThread.run
[2017-06-01 19:37:59,495: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:38:01,381: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:38:01,495: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:38:01,736: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:38:01,814: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:38:02,702: INFO/job_scheduler] Job 2087 finished 1 steps successfully
[2017-06-01 19:38:02,721: INFO/job_scheduler] Job 2087: complete_job: Updating cache
[2017-06-01 19:38:02,721: INFO/job_scheduler] Job 2087 completing (errored=False, cancelled=False)
[2017-06-01 19:38:02,779: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:38:02,817: INFO/job_scheduler] Job 2088 finished 1 steps successfully
[2017-06-01 19:38:02,872: INFO/job_scheduler] Job 2088: complete_job: Updating cache
[2017-06-01 19:38:02,880: INFO/job_scheduler] Job 2088 completing (errored=False, cancelled=False)
[2017-06-01 19:38:03,004: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:38:03,190: INFO/job_scheduler] Job 2090 finished 1 steps successfully
[2017-06-01 19:38:03,294: INFO/job_scheduler] Job 2089 finished 1 steps successfully
[2017-06-01 19:38:03,321: INFO/job_scheduler] Job 2090: complete_job: Updating cache
[2017-06-01 19:38:03,321: INFO/job_scheduler] Job 2090 completing (errored=False, cancelled=False)
[2017-06-01 19:38:03,361: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:38:03,390: INFO/job_scheduler] Job 2089: complete_job: Updating cache
[2017-06-01 19:38:03,394: INFO/job_scheduler] Job 2089 completing (errored=False, cancelled=False)
[2017-06-01 19:38:03,463: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:38:05,118: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:05,120: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:05,129: INFO/command_plan] add_jobs: created Job 2091 (Test for host connectivity)
[2017-06-01 19:38:05,190: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:38:05,244: INFO/job_scheduler] Job 2091: RunJobThread.run
[2017-06-01 19:38:05,340: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:05,340: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:05,354: INFO/command_plan] add_jobs: created Job 2092 (Test for host connectivity)
[2017-06-01 19:38:05,389: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:38:05,442: INFO/job_scheduler] Job 2092: RunJobThread.run
[2017-06-01 19:38:05,520: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:05,523: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:05,539: INFO/command_plan] add_jobs: created Job 2093 (Test for host connectivity)
[2017-06-01 19:38:05,556: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:38:05,576: INFO/job_scheduler] Job 2093: RunJobThread.run
[2017-06-01 19:38:05,683: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:05,684: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:05,698: INFO/command_plan] add_jobs: created Job 2094 (Test for host connectivity)
[2017-06-01 19:38:05,765: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:38:05,809: INFO/job_scheduler] Job 2094: RunJobThread.run
[2017-06-01 19:38:07,605: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:38:07,771: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:38:07,920: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:38:08,116: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:38:08,917: INFO/job_scheduler] Job 2091 finished 1 steps successfully
[2017-06-01 19:38:08,948: INFO/job_scheduler] Job 2091: complete_job: Updating cache
[2017-06-01 19:38:08,948: INFO/job_scheduler] Job 2091 completing (errored=False, cancelled=False)
[2017-06-01 19:38:08,989: INFO/job_scheduler] Job 2092 finished 1 steps successfully
[2017-06-01 19:38:09,037: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:38:09,055: INFO/job_scheduler] Job 2092: complete_job: Updating cache
[2017-06-01 19:38:09,055: INFO/job_scheduler] Job 2092 completing (errored=False, cancelled=False)
[2017-06-01 19:38:09,168: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:38:09,307: INFO/job_scheduler] Job 2093 finished 1 steps successfully
[2017-06-01 19:38:09,337: INFO/job_scheduler] Job 2093: complete_job: Updating cache
[2017-06-01 19:38:09,337: INFO/job_scheduler] Job 2093 completing (errored=False, cancelled=False)
[2017-06-01 19:38:09,418: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:38:09,648: INFO/job_scheduler] Job 2094 finished 1 steps successfully
[2017-06-01 19:38:09,678: INFO/job_scheduler] Job 2094: complete_job: Updating cache
[2017-06-01 19:38:09,678: INFO/job_scheduler] Job 2094 completing (errored=False, cancelled=False)
[2017-06-01 19:38:09,709: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:38:11,585: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:11,587: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:11,602: INFO/command_plan] add_jobs: created Job 2095 (Test for host connectivity)
[2017-06-01 19:38:11,689: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:38:11,726: INFO/job_scheduler] Job 2095: RunJobThread.run
[2017-06-01 19:38:11,765: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:11,765: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:11,895: INFO/command_plan] add_jobs: created Job 2096 (Test for host connectivity)
[2017-06-01 19:38:11,941: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:38:11,970: INFO/job_scheduler] Job 2096: RunJobThread.run
[2017-06-01 19:38:11,999: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:12,002: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:12,014: INFO/command_plan] add_jobs: created Job 2097 (Test for host connectivity)
[2017-06-01 19:38:12,020: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:38:12,041: INFO/job_scheduler] Job 2097: RunJobThread.run
[2017-06-01 19:38:12,245: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:12,246: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:12,371: INFO/command_plan] add_jobs: created Job 2098 (Test for host connectivity)
[2017-06-01 19:38:12,433: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:38:12,476: INFO/job_scheduler] Job 2098: RunJobThread.run
[2017-06-01 19:38:14,033: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:38:14,400: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:38:14,458: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:38:14,724: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:38:15,369: INFO/job_scheduler] Job 2095 finished 1 steps successfully
[2017-06-01 19:38:15,427: INFO/job_scheduler] Job 2095: complete_job: Updating cache
[2017-06-01 19:38:15,427: INFO/job_scheduler] Job 2095 completing (errored=False, cancelled=False)
[2017-06-01 19:38:15,544: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:38:15,834: INFO/job_scheduler] Job 2097 finished 1 steps successfully
[2017-06-01 19:38:15,879: INFO/job_scheduler] Job 2097: complete_job: Updating cache
[2017-06-01 19:38:15,879: INFO/job_scheduler] Job 2097 completing (errored=False, cancelled=False)
[2017-06-01 19:38:15,943: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:38:15,965: INFO/job_scheduler] Job 2098 finished 1 steps successfully
[2017-06-01 19:38:16,008: INFO/job_scheduler] Job 2098: complete_job: Updating cache
[2017-06-01 19:38:16,009: INFO/job_scheduler] Job 2098 completing (errored=False, cancelled=False)
[2017-06-01 19:38:16,009: INFO/job_scheduler] Job 2096 finished 1 steps successfully
[2017-06-01 19:38:16,052: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:38:16,069: INFO/job_scheduler] Job 2096: complete_job: Updating cache
[2017-06-01 19:38:16,070: INFO/job_scheduler] Job 2096 completing (errored=False, cancelled=False)
[2017-06-01 19:38:16,260: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:38:18,839: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:18,840: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:18,851: INFO/command_plan] add_jobs: created Job 2099 (Test for host connectivity)
[2017-06-01 19:38:18,889: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:38:18,908: INFO/job_scheduler] Job 2099: RunJobThread.run
[2017-06-01 19:38:18,990: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:18,991: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:19,010: INFO/command_plan] add_jobs: created Job 2100 (Test for host connectivity)
[2017-06-01 19:38:19,031: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:38:19,094: INFO/job_scheduler] Job 2100: RunJobThread.run
[2017-06-01 19:38:19,170: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:19,172: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:19,186: INFO/command_plan] add_jobs: created Job 2101 (Test for host connectivity)
[2017-06-01 19:38:19,255: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:38:19,276: INFO/job_scheduler] Job 2101: RunJobThread.run
[2017-06-01 19:38:19,362: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:19,363: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:19,378: INFO/command_plan] add_jobs: created Job 2102 (Test for host connectivity)
[2017-06-01 19:38:19,424: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:38:19,481: INFO/job_scheduler] Job 2102: RunJobThread.run
[2017-06-01 19:38:21,249: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:38:21,328: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:38:21,586: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:38:21,830: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:38:22,577: INFO/job_scheduler] Job 2099 finished 1 steps successfully
[2017-06-01 19:38:22,625: INFO/job_scheduler] Job 2099: complete_job: Updating cache
[2017-06-01 19:38:22,625: INFO/job_scheduler] Job 2099 completing (errored=False, cancelled=False)
[2017-06-01 19:38:22,646: INFO/job_scheduler] Job 2100 finished 1 steps successfully
[2017-06-01 19:38:22,693: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:38:22,711: INFO/job_scheduler] Job 2100: complete_job: Updating cache
[2017-06-01 19:38:22,717: INFO/job_scheduler] Job 2100 completing (errored=False, cancelled=False)
[2017-06-01 19:38:22,788: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:38:22,921: INFO/job_scheduler] Job 2101 finished 1 steps successfully
[2017-06-01 19:38:22,978: INFO/job_scheduler] Job 2101: complete_job: Updating cache
[2017-06-01 19:38:22,979: INFO/job_scheduler] Job 2101 completing (errored=False, cancelled=False)
[2017-06-01 19:38:23,050: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:38:23,291: INFO/job_scheduler] Job 2102 finished 1 steps successfully
[2017-06-01 19:38:23,334: INFO/job_scheduler] Job 2102: complete_job: Updating cache
[2017-06-01 19:38:23,334: INFO/job_scheduler] Job 2102 completing (errored=False, cancelled=False)
[2017-06-01 19:38:23,461: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:38:25,257: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:25,257: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:25,266: INFO/command_plan] add_jobs: created Job 2103 (Test for host connectivity)
[2017-06-01 19:38:25,284: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:38:25,300: INFO/job_scheduler] Job 2103: RunJobThread.run
[2017-06-01 19:38:25,380: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:25,381: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:25,479: INFO/command_plan] add_jobs: created Job 2104 (Test for host connectivity)
[2017-06-01 19:38:25,543: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:38:25,566: INFO/job_scheduler] Job 2104: RunJobThread.run
[2017-06-01 19:38:25,648: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:25,650: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:25,675: INFO/command_plan] add_jobs: created Job 2105 (Test for host connectivity)
[2017-06-01 19:38:25,744: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:38:25,783: INFO/job_scheduler] Job 2105: RunJobThread.run
[2017-06-01 19:38:25,848: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:25,848: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:25,902: INFO/command_plan] add_jobs: created Job 2106 (Test for host connectivity)
[2017-06-01 19:38:26,033: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:38:26,044: INFO/job_scheduler] Job 2106: RunJobThread.run
[2017-06-01 19:38:27,694: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:38:27,928: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:38:28,253: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:38:28,306: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:38:29,085: INFO/job_scheduler] Job 2103 finished 1 steps successfully
[2017-06-01 19:38:29,099: INFO/job_scheduler] Job 2103: complete_job: Updating cache
[2017-06-01 19:38:29,099: INFO/job_scheduler] Job 2103 completing (errored=False, cancelled=False)
[2017-06-01 19:38:29,199: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:38:29,296: INFO/job_scheduler] Job 2104 finished 1 steps successfully
[2017-06-01 19:38:29,356: INFO/job_scheduler] Job 2104: complete_job: Updating cache
[2017-06-01 19:38:29,357: INFO/job_scheduler] Job 2104 completing (errored=False, cancelled=False)
[2017-06-01 19:38:29,449: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:38:29,744: INFO/job_scheduler] Job 2105 finished 1 steps successfully
[2017-06-01 19:38:29,790: INFO/job_scheduler] Job 2105: complete_job: Updating cache
[2017-06-01 19:38:29,790: INFO/job_scheduler] Job 2105 completing (errored=False, cancelled=False)
[2017-06-01 19:38:29,816: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:38:29,925: INFO/job_scheduler] Job 2106 finished 1 steps successfully
[2017-06-01 19:38:29,954: INFO/job_scheduler] Job 2106: complete_job: Updating cache
[2017-06-01 19:38:29,955: INFO/job_scheduler] Job 2106 completing (errored=False, cancelled=False)
[2017-06-01 19:38:30,016: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:38:31,747: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:31,749: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:31,784: INFO/command_plan] add_jobs: created Job 2107 (Test for host connectivity)
[2017-06-01 19:38:31,835: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:38:31,841: INFO/job_scheduler] Job 2107: RunJobThread.run
[2017-06-01 19:38:31,929: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:31,930: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:31,952: INFO/command_plan] add_jobs: created Job 2108 (Test for host connectivity)
[2017-06-01 19:38:32,006: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:38:32,009: INFO/job_scheduler] Job 2108: RunJobThread.run
[2017-06-01 19:38:32,114: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:32,116: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:32,148: INFO/command_plan] add_jobs: created Job 2109 (Test for host connectivity)
[2017-06-01 19:38:32,229: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:38:32,266: INFO/job_scheduler] Job 2109: RunJobThread.run
[2017-06-01 19:38:32,377: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:32,378: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:32,415: INFO/command_plan] add_jobs: created Job 2110 (Test for host connectivity)
[2017-06-01 19:38:32,487: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:38:32,523: INFO/job_scheduler] Job 2110: RunJobThread.run
[2017-06-01 19:38:34,236: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:38:34,585: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:38:34,732: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:38:34,931: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:38:35,565: INFO/job_scheduler] Job 2107 finished 1 steps successfully
[2017-06-01 19:38:35,616: INFO/job_scheduler] Job 2107: complete_job: Updating cache
[2017-06-01 19:38:35,616: INFO/job_scheduler] Job 2107 completing (errored=False, cancelled=False)
[2017-06-01 19:38:35,717: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:38:35,963: INFO/job_scheduler] Job 2108 finished 1 steps successfully
[2017-06-01 19:38:36,033: INFO/job_scheduler] Job 2108: complete_job: Updating cache
[2017-06-01 19:38:36,034: INFO/job_scheduler] Job 2108 completing (errored=False, cancelled=False)
[2017-06-01 19:38:36,084: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:38:36,169: INFO/job_scheduler] Job 2109 finished 1 steps successfully
[2017-06-01 19:38:36,231: INFO/job_scheduler] Job 2109: complete_job: Updating cache
[2017-06-01 19:38:36,233: INFO/job_scheduler] Job 2109 completing (errored=False, cancelled=False)
[2017-06-01 19:38:36,298: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:38:36,343: INFO/job_scheduler] Job 2110 finished 1 steps successfully
[2017-06-01 19:38:36,385: INFO/job_scheduler] Job 2110: complete_job: Updating cache
[2017-06-01 19:38:36,386: INFO/job_scheduler] Job 2110 completing (errored=False, cancelled=False)
[2017-06-01 19:38:36,439: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:38:38,205: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:38,208: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:38,232: INFO/command_plan] add_jobs: created Job 2111 (Test for host connectivity)
[2017-06-01 19:38:38,283: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:38:38,378: INFO/job_scheduler] Job 2111: RunJobThread.run
[2017-06-01 19:38:38,455: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:38,456: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:38,477: INFO/command_plan] add_jobs: created Job 2112 (Test for host connectivity)
[2017-06-01 19:38:38,610: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:38,610: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:38,641: INFO/command_plan] add_jobs: created Job 2113 (Test for host connectivity)
[2017-06-01 19:38:38,674: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:38:38,694: INFO/job_scheduler] Job 2112: RunJobThread.run
[2017-06-01 19:38:38,719: INFO/job_scheduler] Job 2113: RunJobThread.run
[2017-06-01 19:38:38,785: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:38,788: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:38,812: INFO/command_plan] add_jobs: created Job 2114 (Test for host connectivity)
[2017-06-01 19:38:38,841: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:38:38,903: INFO/job_scheduler] Job 2114: RunJobThread.run
[2017-06-01 19:38:38,989: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:38:40,792: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:38:41,125: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:38:41,161: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:38:41,258: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:38:42,109: INFO/job_scheduler] Job 2111 finished 1 steps successfully
[2017-06-01 19:38:42,153: INFO/job_scheduler] Job 2111: complete_job: Updating cache
[2017-06-01 19:38:42,157: INFO/job_scheduler] Job 2111 completing (errored=False, cancelled=False)
[2017-06-01 19:38:42,366: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:38:42,448: INFO/job_scheduler] Job 2112 finished 1 steps successfully
[2017-06-01 19:38:42,518: INFO/job_scheduler] Job 2112: complete_job: Updating cache
[2017-06-01 19:38:42,519: INFO/job_scheduler] Job 2112 completing (errored=False, cancelled=False)
[2017-06-01 19:38:42,580: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:38:42,648: INFO/job_scheduler] Job 2113 finished 1 steps successfully
[2017-06-01 19:38:42,686: INFO/job_scheduler] Job 2113: complete_job: Updating cache
[2017-06-01 19:38:42,690: INFO/job_scheduler] Job 2113 completing (errored=False, cancelled=False)
[2017-06-01 19:38:42,689: INFO/job_scheduler] Job 2114 finished 1 steps successfully
[2017-06-01 19:38:42,739: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:38:42,755: INFO/job_scheduler] Job 2114: complete_job: Updating cache
[2017-06-01 19:38:42,757: INFO/job_scheduler] Job 2114 completing (errored=False, cancelled=False)
[2017-06-01 19:38:42,866: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:38:44,615: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:44,615: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:44,647: INFO/command_plan] add_jobs: created Job 2115 (Test for host connectivity)
[2017-06-01 19:38:44,750: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:44,750: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:44,780: INFO/command_plan] add_jobs: created Job 2116 (Test for host connectivity)
[2017-06-01 19:38:44,797: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:38:44,860: INFO/job_scheduler] Job 2115: RunJobThread.run
[2017-06-01 19:38:44,882: INFO/job_scheduler] Job 2116: RunJobThread.run
[2017-06-01 19:38:44,893: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:38:44,957: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:44,957: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:44,986: INFO/command_plan] add_jobs: created Job 2117 (Test for host connectivity)
[2017-06-01 19:38:45,079: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:45,080: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:45,117: INFO/command_plan] add_jobs: created Job 2118 (Test for host connectivity)
[2017-06-01 19:38:45,130: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:38:45,143: INFO/job_scheduler] Job 2117: RunJobThread.run
[2017-06-01 19:38:45,153: INFO/job_scheduler] Job 2118: RunJobThread.run
[2017-06-01 19:38:45,164: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:38:47,255: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:38:47,346: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:38:47,446: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:38:47,508: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:38:48,597: INFO/job_scheduler] Job 2116 finished 1 steps successfully
[2017-06-01 19:38:48,655: INFO/job_scheduler] Job 2116: complete_job: Updating cache
[2017-06-01 19:38:48,656: INFO/job_scheduler] Job 2116 completing (errored=False, cancelled=False)
[2017-06-01 19:38:48,687: INFO/job_scheduler] Job 2115 finished 1 steps successfully
[2017-06-01 19:38:48,737: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:38:48,833: INFO/job_scheduler] Job 2115: complete_job: Updating cache
[2017-06-01 19:38:48,833: INFO/job_scheduler] Job 2115 completing (errored=False, cancelled=False)
[2017-06-01 19:38:48,856: INFO/job_scheduler] Job 2117 finished 1 steps successfully
[2017-06-01 19:38:48,921: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:38:48,935: INFO/job_scheduler] Job 2118 finished 1 steps successfully
[2017-06-01 19:38:48,967: INFO/job_scheduler] Job 2117: complete_job: Updating cache
[2017-06-01 19:38:48,968: INFO/job_scheduler] Job 2117 completing (errored=False, cancelled=False)
[2017-06-01 19:38:49,013: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:38:49,038: INFO/job_scheduler] Job 2118: complete_job: Updating cache
[2017-06-01 19:38:49,038: INFO/job_scheduler] Job 2118 completing (errored=False, cancelled=False)
[2017-06-01 19:38:49,111: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:38:50,895: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:50,896: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:50,922: INFO/command_plan] add_jobs: created Job 2119 (Test for host connectivity)
[2017-06-01 19:38:51,004: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:51,004: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:51,036: INFO/command_plan] add_jobs: created Job 2120 (Test for host connectivity)
[2017-06-01 19:38:51,082: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:38:51,091: INFO/job_scheduler] Job 2120: RunJobThread.run
[2017-06-01 19:38:51,155: INFO/job_scheduler] Job 2119: RunJobThread.run
[2017-06-01 19:38:51,163: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:38:51,253: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:51,256: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:51,293: INFO/command_plan] add_jobs: created Job 2121 (Test for host connectivity)
[2017-06-01 19:38:51,411: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:38:51,419: INFO/job_scheduler] Job 2121: RunJobThread.run
[2017-06-01 19:38:51,507: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:51,508: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:51,521: INFO/command_plan] add_jobs: created Job 2122 (Test for host connectivity)
[2017-06-01 19:38:51,628: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:38:51,631: INFO/job_scheduler] Job 2122: RunJobThread.run
[2017-06-01 19:38:53,545: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:38:53,629: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:38:53,743: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:38:54,097: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:38:54,882: INFO/job_scheduler] Job 2120 finished 1 steps successfully
[2017-06-01 19:38:54,944: INFO/job_scheduler] Job 2120: complete_job: Updating cache
[2017-06-01 19:38:54,944: INFO/job_scheduler] Job 2120 completing (errored=False, cancelled=False)
[2017-06-01 19:38:54,974: INFO/job_scheduler] Job 2119 finished 1 steps successfully
[2017-06-01 19:38:54,978: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:38:54,994: INFO/job_scheduler] Job 2119: complete_job: Updating cache
[2017-06-01 19:38:54,994: INFO/job_scheduler] Job 2119 completing (errored=False, cancelled=False)
[2017-06-01 19:38:55,177: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:38:55,257: INFO/job_scheduler] Job 2121 finished 1 steps successfully
[2017-06-01 19:38:55,272: INFO/job_scheduler] Job 2121: complete_job: Updating cache
[2017-06-01 19:38:55,273: INFO/job_scheduler] Job 2121 completing (errored=False, cancelled=False)
[2017-06-01 19:38:55,323: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:38:55,610: INFO/job_scheduler] Job 2122 finished 1 steps successfully
[2017-06-01 19:38:55,644: INFO/job_scheduler] Job 2122: complete_job: Updating cache
[2017-06-01 19:38:55,644: INFO/job_scheduler] Job 2122 completing (errored=False, cancelled=False)
[2017-06-01 19:38:55,671: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:38:57,356: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:57,357: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:57,395: INFO/command_plan] add_jobs: created Job 2123 (Test for host connectivity)
[2017-06-01 19:38:57,441: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:38:57,482: INFO/job_scheduler] Job 2123: RunJobThread.run
[2017-06-01 19:38:57,589: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:57,590: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:57,620: INFO/command_plan] add_jobs: created Job 2124 (Test for host connectivity)
[2017-06-01 19:38:57,650: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:38:57,708: INFO/job_scheduler] Job 2124: RunJobThread.run
[2017-06-01 19:38:57,840: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:57,843: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:57,869: INFO/command_plan] add_jobs: created Job 2125 (Test for host connectivity)
[2017-06-01 19:38:57,942: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:38:57,969: INFO/job_scheduler] Job 2125: RunJobThread.run
[2017-06-01 19:38:58,062: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:38:58,064: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:38:58,117: INFO/command_plan] add_jobs: created Job 2126 (Test for host connectivity)
[2017-06-01 19:38:58,164: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:38:58,196: INFO/job_scheduler] Job 2126: RunJobThread.run
[2017-06-01 19:38:59,799: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:39:00,048: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:39:00,393: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:39:00,423: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:39:01,114: INFO/job_scheduler] Job 2123 finished 1 steps successfully
[2017-06-01 19:39:01,143: INFO/job_scheduler] Job 2123: complete_job: Updating cache
[2017-06-01 19:39:01,143: INFO/job_scheduler] Job 2123 completing (errored=False, cancelled=False)
[2017-06-01 19:39:01,202: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:39:01,483: INFO/job_scheduler] Job 2124 finished 1 steps successfully
[2017-06-01 19:39:01,512: INFO/job_scheduler] Job 2124: complete_job: Updating cache
[2017-06-01 19:39:01,512: INFO/job_scheduler] Job 2124 completing (errored=False, cancelled=False)
[2017-06-01 19:39:01,584: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:39:01,854: INFO/job_scheduler] Job 2126 finished 1 steps successfully
[2017-06-01 19:39:01,909: INFO/job_scheduler] Job 2126: complete_job: Updating cache
[2017-06-01 19:39:01,909: INFO/job_scheduler] Job 2126 completing (errored=False, cancelled=False)
[2017-06-01 19:39:01,945: INFO/job_scheduler] Job 2125 finished 1 steps successfully
[2017-06-01 19:39:01,954: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:39:01,968: INFO/job_scheduler] Job 2125: complete_job: Updating cache
[2017-06-01 19:39:01,969: INFO/job_scheduler] Job 2125 completing (errored=False, cancelled=False)
[2017-06-01 19:39:02,072: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:39:03,930: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:03,930: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:03,954: INFO/command_plan] add_jobs: created Job 2127 (Test for host connectivity)
[2017-06-01 19:39:04,004: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:39:04,035: INFO/job_scheduler] Job 2127: RunJobThread.run
[2017-06-01 19:39:04,104: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:04,106: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:04,125: INFO/command_plan] add_jobs: created Job 2128 (Test for host connectivity)
[2017-06-01 19:39:04,157: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:39:04,199: INFO/job_scheduler] Job 2128: RunJobThread.run
[2017-06-01 19:39:04,311: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:04,312: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:04,345: INFO/command_plan] add_jobs: created Job 2129 (Test for host connectivity)
[2017-06-01 19:39:04,390: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:39:04,477: INFO/job_scheduler] Job 2129: RunJobThread.run
[2017-06-01 19:39:04,531: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:04,532: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:04,562: INFO/command_plan] add_jobs: created Job 2130 (Test for host connectivity)
[2017-06-01 19:39:04,682: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:39:04,708: INFO/job_scheduler] Job 2130: RunJobThread.run
[2017-06-01 19:39:06,378: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:39:06,578: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:39:06,781: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:39:06,916: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:39:07,719: INFO/job_scheduler] Job 2127 finished 1 steps successfully
[2017-06-01 19:39:07,742: INFO/job_scheduler] Job 2127: complete_job: Updating cache
[2017-06-01 19:39:07,742: INFO/job_scheduler] Job 2127 completing (errored=False, cancelled=False)
[2017-06-01 19:39:07,793: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:39:07,938: INFO/job_scheduler] Job 2128 finished 1 steps successfully
[2017-06-01 19:39:07,997: INFO/job_scheduler] Job 2128: complete_job: Updating cache
[2017-06-01 19:39:07,998: INFO/job_scheduler] Job 2128 completing (errored=False, cancelled=False)
[2017-06-01 19:39:08,084: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:39:08,208: INFO/job_scheduler] Job 2129 finished 1 steps successfully
[2017-06-01 19:39:08,233: INFO/job_scheduler] Job 2129: complete_job: Updating cache
[2017-06-01 19:39:08,234: INFO/job_scheduler] Job 2129 completing (errored=False, cancelled=False)
[2017-06-01 19:39:08,285: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:39:08,342: INFO/job_scheduler] Job 2130 finished 1 steps successfully
[2017-06-01 19:39:08,411: INFO/job_scheduler] Job 2130: complete_job: Updating cache
[2017-06-01 19:39:08,412: INFO/job_scheduler] Job 2130 completing (errored=False, cancelled=False)
[2017-06-01 19:39:08,486: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:39:10,447: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:10,449: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:10,467: INFO/command_plan] add_jobs: created Job 2131 (Test for host connectivity)
[2017-06-01 19:39:10,541: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:39:10,552: INFO/job_scheduler] Job 2131: RunJobThread.run
[2017-06-01 19:39:10,591: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:10,593: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:10,625: INFO/command_plan] add_jobs: created Job 2132 (Test for host connectivity)
[2017-06-01 19:39:10,716: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:39:10,720: INFO/job_scheduler] Job 2132: RunJobThread.run
[2017-06-01 19:39:10,796: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:10,797: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:10,809: INFO/command_plan] add_jobs: created Job 2133 (Test for host connectivity)
[2017-06-01 19:39:10,865: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:39:10,906: INFO/job_scheduler] Job 2133: RunJobThread.run
[2017-06-01 19:39:11,035: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:11,036: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:11,072: INFO/command_plan] add_jobs: created Job 2134 (Test for host connectivity)
[2017-06-01 19:39:11,118: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:39:11,169: INFO/job_scheduler] Job 2134: RunJobThread.run
[2017-06-01 19:39:12,852: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:39:13,147: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:39:13,253: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:39:13,436: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:39:14,175: INFO/job_scheduler] Job 2131 finished 1 steps successfully
[2017-06-01 19:39:14,207: INFO/job_scheduler] Job 2131: complete_job: Updating cache
[2017-06-01 19:39:14,207: INFO/job_scheduler] Job 2131 completing (errored=False, cancelled=False)
[2017-06-01 19:39:14,241: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:39:14,474: INFO/job_scheduler] Job 2132 finished 1 steps successfully
[2017-06-01 19:39:14,515: INFO/job_scheduler] Job 2132: complete_job: Updating cache
[2017-06-01 19:39:14,515: INFO/job_scheduler] Job 2132 completing (errored=False, cancelled=False)
[2017-06-01 19:39:14,568: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:39:14,650: INFO/job_scheduler] Job 2133 finished 1 steps successfully
[2017-06-01 19:39:14,725: INFO/job_scheduler] Job 2133: complete_job: Updating cache
[2017-06-01 19:39:14,727: INFO/job_scheduler] Job 2133 completing (errored=False, cancelled=False)
[2017-06-01 19:39:14,818: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:39:14,833: INFO/job_scheduler] Job 2134 finished 1 steps successfully
[2017-06-01 19:39:14,879: INFO/job_scheduler] Job 2134: complete_job: Updating cache
[2017-06-01 19:39:14,880: INFO/job_scheduler] Job 2134 completing (errored=False, cancelled=False)
[2017-06-01 19:39:14,940: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:39:16,883: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:16,885: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:16,907: INFO/command_plan] add_jobs: created Job 2135 (Test for host connectivity)
[2017-06-01 19:39:16,955: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:39:16,958: INFO/job_scheduler] Job 2135: RunJobThread.run
[2017-06-01 19:39:17,075: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:17,076: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:17,090: INFO/command_plan] add_jobs: created Job 2136 (Test for host connectivity)
[2017-06-01 19:39:17,122: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:39:17,160: INFO/job_scheduler] Job 2136: RunJobThread.run
[2017-06-01 19:39:17,232: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:17,233: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:17,300: INFO/command_plan] add_jobs: created Job 2137 (Test for host connectivity)
[2017-06-01 19:39:17,333: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:39:17,451: INFO/job_scheduler] Job 2137: RunJobThread.run
[2017-06-01 19:39:17,522: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:17,523: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:17,532: INFO/command_plan] add_jobs: created Job 2138 (Test for host connectivity)
[2017-06-01 19:39:17,626: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:39:17,656: INFO/job_scheduler] Job 2138: RunJobThread.run
[2017-06-01 19:39:19,283: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:39:19,569: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:39:19,909: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:39:19,950: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:39:20,631: INFO/job_scheduler] Job 2135 finished 1 steps successfully
[2017-06-01 19:39:20,672: INFO/job_scheduler] Job 2135: complete_job: Updating cache
[2017-06-01 19:39:20,673: INFO/job_scheduler] Job 2135 completing (errored=False, cancelled=False)
[2017-06-01 19:39:20,733: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:39:20,936: INFO/job_scheduler] Job 2136 finished 1 steps successfully
[2017-06-01 19:39:21,097: INFO/job_scheduler] Job 2136: complete_job: Updating cache
[2017-06-01 19:39:21,097: INFO/job_scheduler] Job 2136 completing (errored=False, cancelled=False)
[2017-06-01 19:39:21,172: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:39:21,337: INFO/job_scheduler] Job 2138 finished 1 steps successfully
[2017-06-01 19:39:21,389: INFO/job_scheduler] Job 2138: complete_job: Updating cache
[2017-06-01 19:39:21,389: INFO/job_scheduler] Job 2138 completing (errored=False, cancelled=False)
[2017-06-01 19:39:21,439: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:39:21,442: INFO/job_scheduler] Job 2137 finished 1 steps successfully
[2017-06-01 19:39:21,471: INFO/job_scheduler] Job 2137: complete_job: Updating cache
[2017-06-01 19:39:21,472: INFO/job_scheduler] Job 2137 completing (errored=False, cancelled=False)
[2017-06-01 19:39:21,552: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:39:23,372: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:23,373: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:23,391: INFO/command_plan] add_jobs: created Job 2139 (Test for host connectivity)
[2017-06-01 19:39:23,433: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:39:23,469: INFO/job_scheduler] Job 2139: RunJobThread.run
[2017-06-01 19:39:23,528: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:23,529: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:23,551: INFO/command_plan] add_jobs: created Job 2140 (Test for host connectivity)
[2017-06-01 19:39:23,649: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:39:23,694: INFO/job_scheduler] Job 2140: RunJobThread.run
[2017-06-01 19:39:23,793: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:23,794: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:23,838: INFO/command_plan] add_jobs: created Job 2141 (Test for host connectivity)
[2017-06-01 19:39:23,891: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:39:23,905: INFO/job_scheduler] Job 2141: RunJobThread.run
[2017-06-01 19:39:23,967: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:23,967: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:24,017: INFO/command_plan] add_jobs: created Job 2142 (Test for host connectivity)
[2017-06-01 19:39:24,053: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:39:24,063: INFO/job_scheduler] Job 2142: RunJobThread.run
[2017-06-01 19:39:25,940: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:39:26,021: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:39:26,296: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:39:26,381: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:39:27,263: INFO/job_scheduler] Job 2139 finished 1 steps successfully
[2017-06-01 19:39:27,279: INFO/job_scheduler] Job 2139: complete_job: Updating cache
[2017-06-01 19:39:27,279: INFO/job_scheduler] Job 2139 completing (errored=False, cancelled=False)
[2017-06-01 19:39:27,304: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:39:27,336: INFO/job_scheduler] Job 2140 finished 1 steps successfully
[2017-06-01 19:39:27,364: INFO/job_scheduler] Job 2140: complete_job: Updating cache
[2017-06-01 19:39:27,364: INFO/job_scheduler] Job 2140 completing (errored=False, cancelled=False)
[2017-06-01 19:39:27,457: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:39:27,855: INFO/job_scheduler] Job 2142 finished 1 steps successfully
[2017-06-01 19:39:27,862: INFO/job_scheduler] Job 2142: complete_job: Updating cache
[2017-06-01 19:39:27,863: INFO/job_scheduler] Job 2142 completing (errored=False, cancelled=False)
[2017-06-01 19:39:27,935: INFO/job_scheduler] Job 2141 finished 1 steps successfully
[2017-06-01 19:39:27,938: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:39:27,948: INFO/job_scheduler] Job 2141: complete_job: Updating cache
[2017-06-01 19:39:27,948: INFO/job_scheduler] Job 2141 completing (errored=False, cancelled=False)
[2017-06-01 19:39:28,073: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:39:29,828: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:29,828: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:29,855: INFO/command_plan] add_jobs: created Job 2143 (Test for host connectivity)
[2017-06-01 19:39:29,893: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:39:29,908: INFO/job_scheduler] Job 2143: RunJobThread.run
[2017-06-01 19:39:29,995: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:29,996: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:30,006: INFO/command_plan] add_jobs: created Job 2144 (Test for host connectivity)
[2017-06-01 19:39:30,093: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:39:30,128: INFO/job_scheduler] Job 2144: RunJobThread.run
[2017-06-01 19:39:30,188: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:30,189: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:30,229: INFO/command_plan] add_jobs: created Job 2145 (Test for host connectivity)
[2017-06-01 19:39:30,370: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:30,370: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:30,389: INFO/command_plan] add_jobs: created Job 2146 (Test for host connectivity)
[2017-06-01 19:39:30,398: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:39:30,454: INFO/job_scheduler] Job 2145: RunJobThread.run
[2017-06-01 19:39:30,457: INFO/job_scheduler] Job 2146: RunJobThread.run
[2017-06-01 19:39:30,475: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:39:32,288: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:39:32,522: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:39:32,740: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:39:32,890: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:39:33,627: INFO/job_scheduler] Job 2143 finished 1 steps successfully
[2017-06-01 19:39:33,692: INFO/job_scheduler] Job 2143: complete_job: Updating cache
[2017-06-01 19:39:33,693: INFO/job_scheduler] Job 2143 completing (errored=False, cancelled=False)
[2017-06-01 19:39:33,776: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:39:33,879: INFO/job_scheduler] Job 2144 finished 1 steps successfully
[2017-06-01 19:39:33,929: INFO/job_scheduler] Job 2144: complete_job: Updating cache
[2017-06-01 19:39:33,929: INFO/job_scheduler] Job 2144 completing (errored=False, cancelled=False)
[2017-06-01 19:39:34,033: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:39:34,070: INFO/job_scheduler] Job 2146 finished 1 steps successfully
[2017-06-01 19:39:34,104: INFO/job_scheduler] Job 2146: complete_job: Updating cache
[2017-06-01 19:39:34,104: INFO/job_scheduler] Job 2146 completing (errored=False, cancelled=False)
[2017-06-01 19:39:34,153: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:39:34,324: INFO/job_scheduler] Job 2145 finished 1 steps successfully
[2017-06-01 19:39:34,386: INFO/job_scheduler] Job 2145: complete_job: Updating cache
[2017-06-01 19:39:34,387: INFO/job_scheduler] Job 2145 completing (errored=False, cancelled=False)
[2017-06-01 19:39:34,428: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:39:36,237: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:36,237: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:36,253: INFO/command_plan] add_jobs: created Job 2147 (Test for host connectivity)
[2017-06-01 19:39:36,326: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:36,326: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:36,352: INFO/command_plan] add_jobs: created Job 2148 (Test for host connectivity)
[2017-06-01 19:39:36,362: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:39:36,378: INFO/job_scheduler] Job 2147: RunJobThread.run
[2017-06-01 19:39:36,449: INFO/job_scheduler] Job 2148: RunJobThread.run
[2017-06-01 19:39:36,495: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:36,495: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:36,512: INFO/command_plan] add_jobs: created Job 2149 (Test for host connectivity)
[2017-06-01 19:39:36,573: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:39:36,584: INFO/job_scheduler] Job 2149: RunJobThread.run
[2017-06-01 19:39:36,705: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:36,708: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:36,731: INFO/command_plan] add_jobs: created Job 2150 (Test for host connectivity)
[2017-06-01 19:39:36,759: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:39:36,795: INFO/job_scheduler] Job 2150: RunJobThread.run
[2017-06-01 19:39:36,842: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:39:38,714: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:39:38,895: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:39:38,951: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:39:39,168: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:39:40,064: INFO/job_scheduler] Job 2147 finished 1 steps successfully
[2017-06-01 19:39:40,104: INFO/job_scheduler] Job 2147: complete_job: Updating cache
[2017-06-01 19:39:40,104: INFO/job_scheduler] Job 2147 completing (errored=False, cancelled=False)
[2017-06-01 19:39:40,130: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:39:40,246: INFO/job_scheduler] Job 2148 finished 1 steps successfully
[2017-06-01 19:39:40,271: INFO/job_scheduler] Job 2148: complete_job: Updating cache
[2017-06-01 19:39:40,272: INFO/job_scheduler] Job 2148 completing (errored=False, cancelled=False)
[2017-06-01 19:39:40,348: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:39:40,377: INFO/job_scheduler] Job 2149 finished 1 steps successfully
[2017-06-01 19:39:40,442: INFO/job_scheduler] Job 2149: complete_job: Updating cache
[2017-06-01 19:39:40,442: INFO/job_scheduler] Job 2149 completing (errored=False, cancelled=False)
[2017-06-01 19:39:40,531: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:39:40,663: INFO/job_scheduler] Job 2150 finished 1 steps successfully
[2017-06-01 19:39:40,696: INFO/job_scheduler] Job 2150: complete_job: Updating cache
[2017-06-01 19:39:40,696: INFO/job_scheduler] Job 2150 completing (errored=False, cancelled=False)
[2017-06-01 19:39:40,746: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:39:42,542: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:42,544: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:42,554: INFO/command_plan] add_jobs: created Job 2151 (Test for host connectivity)
[2017-06-01 19:39:42,571: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:39:42,608: INFO/job_scheduler] Job 2151: RunJobThread.run
[2017-06-01 19:39:42,681: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:42,688: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:42,708: INFO/command_plan] add_jobs: created Job 2152 (Test for host connectivity)
[2017-06-01 19:39:42,792: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:39:42,818: INFO/job_scheduler] Job 2152: RunJobThread.run
[2017-06-01 19:39:42,848: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:42,849: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:42,906: INFO/command_plan] add_jobs: created Job 2153 (Test for host connectivity)
[2017-06-01 19:39:42,993: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:39:43,008: INFO/job_scheduler] Job 2153: RunJobThread.run
[2017-06-01 19:39:43,060: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:43,061: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:43,091: INFO/command_plan] add_jobs: created Job 2154 (Test for host connectivity)
[2017-06-01 19:39:43,108: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:39:43,115: INFO/job_scheduler] Job 2154: RunJobThread.run
[2017-06-01 19:39:45,004: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:39:45,303: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:39:45,320: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:39:45,548: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:39:46,383: INFO/job_scheduler] Job 2151 finished 1 steps successfully
[2017-06-01 19:39:46,438: INFO/job_scheduler] Job 2151: complete_job: Updating cache
[2017-06-01 19:39:46,439: INFO/job_scheduler] Job 2151 completing (errored=False, cancelled=False)
[2017-06-01 19:39:46,509: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:39:46,645: INFO/job_scheduler] Job 2152 finished 1 steps successfully
[2017-06-01 19:39:46,678: INFO/job_scheduler] Job 2152: complete_job: Updating cache
[2017-06-01 19:39:46,679: INFO/job_scheduler] Job 2152 completing (errored=False, cancelled=False)
[2017-06-01 19:39:46,769: INFO/job_scheduler] Job 2153 finished 1 steps successfully
[2017-06-01 19:39:46,771: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:39:46,805: INFO/job_scheduler] Job 2153: complete_job: Updating cache
[2017-06-01 19:39:46,805: INFO/job_scheduler] Job 2153 completing (errored=False, cancelled=False)
[2017-06-01 19:39:46,889: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:39:47,043: INFO/job_scheduler] Job 2154 finished 1 steps successfully
[2017-06-01 19:39:47,062: INFO/job_scheduler] Job 2154: complete_job: Updating cache
[2017-06-01 19:39:47,063: INFO/job_scheduler] Job 2154 completing (errored=False, cancelled=False)
[2017-06-01 19:39:47,121: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:39:48,893: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:48,893: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:48,931: INFO/command_plan] add_jobs: created Job 2155 (Test for host connectivity)
[2017-06-01 19:39:49,056: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:49,056: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:49,085: INFO/command_plan] add_jobs: created Job 2156 (Test for host connectivity)
[2017-06-01 19:39:49,096: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:39:49,132: INFO/job_scheduler] Job 2155: RunJobThread.run
[2017-06-01 19:39:49,173: INFO/job_scheduler] Job 2156: RunJobThread.run
[2017-06-01 19:39:49,186: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:39:49,245: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:49,247: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:49,265: INFO/command_plan] add_jobs: created Job 2157 (Test for host connectivity)
[2017-06-01 19:39:49,338: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:39:49,386: INFO/job_scheduler] Job 2157: RunJobThread.run
[2017-06-01 19:39:49,414: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:49,415: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:49,436: INFO/command_plan] add_jobs: created Job 2158 (Test for host connectivity)
[2017-06-01 19:39:49,474: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:39:49,544: INFO/job_scheduler] Job 2158: RunJobThread.run
[2017-06-01 19:39:51,501: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:39:51,590: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:39:51,685: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:39:51,818: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:39:52,793: INFO/job_scheduler] Job 2155 finished 1 steps successfully
[2017-06-01 19:39:52,818: INFO/job_scheduler] Job 2155: complete_job: Updating cache
[2017-06-01 19:39:52,818: INFO/job_scheduler] Job 2155 completing (errored=False, cancelled=False)
[2017-06-01 19:39:52,885: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:39:52,949: INFO/job_scheduler] Job 2156 finished 1 steps successfully
[2017-06-01 19:39:52,997: INFO/job_scheduler] Job 2156: complete_job: Updating cache
[2017-06-01 19:39:52,997: INFO/job_scheduler] Job 2156 completing (errored=False, cancelled=False)
[2017-06-01 19:39:53,044: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:39:53,083: INFO/job_scheduler] Job 2157 finished 1 steps successfully
[2017-06-01 19:39:53,151: INFO/job_scheduler] Job 2157: complete_job: Updating cache
[2017-06-01 19:39:53,152: INFO/job_scheduler] Job 2157 completing (errored=False, cancelled=False)
[2017-06-01 19:39:53,161: INFO/job_scheduler] Job 2158 finished 1 steps successfully
[2017-06-01 19:39:53,228: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:39:53,244: INFO/job_scheduler] Job 2158: complete_job: Updating cache
[2017-06-01 19:39:53,244: INFO/job_scheduler] Job 2158 completing (errored=False, cancelled=False)
[2017-06-01 19:39:53,321: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:39:54,556: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:54,558: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:54,568: INFO/command_plan] add_jobs: created Job 2159 (Test for host connectivity)
[2017-06-01 19:39:54,694: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:54,696: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:54,710: INFO/command_plan] add_jobs: created Job 2160 (Test for host connectivity)
[2017-06-01 19:39:54,736: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:39:54,740: INFO/job_scheduler] Job 2160: RunJobThread.run
[2017-06-01 19:39:54,740: INFO/job_scheduler] Job 2159: RunJobThread.run
[2017-06-01 19:39:54,745: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:39:54,850: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:54,855: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:54,877: INFO/command_plan] add_jobs: created Job 2161 (Test for host connectivity)
[2017-06-01 19:39:54,980: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:39:55,027: INFO/job_scheduler] Job 2161: RunJobThread.run
[2017-06-01 19:39:55,102: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:39:55,103: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:39:55,133: INFO/command_plan] add_jobs: created Job 2162 (Test for host connectivity)
[2017-06-01 19:39:55,214: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:39:55,248: INFO/job_scheduler] Job 2162: RunJobThread.run
[2017-06-01 19:39:57,146: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:39:57,177: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:39:57,343: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:39:57,367: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:39:58,489: INFO/job_scheduler] Job 2160 finished 1 steps successfully
[2017-06-01 19:39:58,559: INFO/job_scheduler] Job 2160: complete_job: Updating cache
[2017-06-01 19:39:58,559: INFO/job_scheduler] Job 2160 completing (errored=False, cancelled=False)
[2017-06-01 19:39:58,619: INFO/job_scheduler] Job 2159 finished 1 steps successfully
[2017-06-01 19:39:58,619: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:39:58,635: INFO/job_scheduler] Job 2159: complete_job: Updating cache
[2017-06-01 19:39:58,635: INFO/job_scheduler] Job 2159 completing (errored=False, cancelled=False)
[2017-06-01 19:39:58,700: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:39:58,758: INFO/job_scheduler] Job 2161 finished 1 steps successfully
[2017-06-01 19:39:58,765: INFO/job_scheduler] Job 2162 finished 1 steps successfully
[2017-06-01 19:39:58,796: INFO/job_scheduler] Job 2161: complete_job: Updating cache
[2017-06-01 19:39:58,797: INFO/job_scheduler] Job 2161 completing (errored=False, cancelled=False)
[2017-06-01 19:39:58,932: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:39:58,944: INFO/job_scheduler] Job 2162: complete_job: Updating cache
[2017-06-01 19:39:58,947: INFO/job_scheduler] Job 2162 completing (errored=False, cancelled=False)
[2017-06-01 19:39:59,000: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:40:00,830: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:00,832: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:00,841: INFO/command_plan] add_jobs: created Job 2163 (Test for host connectivity)
[2017-06-01 19:40:00,944: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:40:00,949: INFO/job_scheduler] Job 2163: RunJobThread.run
[2017-06-01 19:40:01,003: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:01,004: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:01,017: INFO/command_plan] add_jobs: created Job 2164 (Test for host connectivity)
[2017-06-01 19:40:01,145: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:01,146: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:01,209: INFO/command_plan] add_jobs: created Job 2165 (Test for host connectivity)
[2017-06-01 19:40:01,233: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:40:01,241: INFO/job_scheduler] Job 2164: RunJobThread.run
[2017-06-01 19:40:01,285: INFO/job_scheduler] Job 2165: RunJobThread.run
[2017-06-01 19:40:01,294: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:40:01,391: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:01,392: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:01,428: INFO/command_plan] add_jobs: created Job 2166 (Test for host connectivity)
[2017-06-01 19:40:01,498: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:40:01,526: INFO/job_scheduler] Job 2166: RunJobThread.run
[2017-06-01 19:40:03,241: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:40:03,745: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:40:03,804: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:40:03,864: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:40:04,563: INFO/job_scheduler] Job 2163 finished 1 steps successfully
[2017-06-01 19:40:04,608: INFO/job_scheduler] Job 2163: complete_job: Updating cache
[2017-06-01 19:40:04,608: INFO/job_scheduler] Job 2163 completing (errored=False, cancelled=False)
[2017-06-01 19:40:04,666: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:40:05,185: INFO/job_scheduler] Job 2165 finished 1 steps successfully
[2017-06-01 19:40:05,232: INFO/job_scheduler] Job 2165: complete_job: Updating cache
[2017-06-01 19:40:05,233: INFO/job_scheduler] Job 2165 completing (errored=False, cancelled=False)
[2017-06-01 19:40:05,260: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:40:05,293: INFO/job_scheduler] Job 2164 finished 1 steps successfully
[2017-06-01 19:40:05,354: INFO/job_scheduler] Job 2166 finished 1 steps successfully
[2017-06-01 19:40:05,355: INFO/job_scheduler] Job 2164: complete_job: Updating cache
[2017-06-01 19:40:05,355: INFO/job_scheduler] Job 2164 completing (errored=False, cancelled=False)
[2017-06-01 19:40:05,418: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:40:05,439: INFO/job_scheduler] Job 2166: complete_job: Updating cache
[2017-06-01 19:40:05,440: INFO/job_scheduler] Job 2166 completing (errored=False, cancelled=False)
[2017-06-01 19:40:05,551: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:40:07,180: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:07,181: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:07,200: INFO/command_plan] add_jobs: created Job 2167 (Test for host connectivity)
[2017-06-01 19:40:07,220: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:40:07,281: INFO/job_scheduler] Job 2167: RunJobThread.run
[2017-06-01 19:40:07,338: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:07,340: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:07,363: INFO/command_plan] add_jobs: created Job 2168 (Test for host connectivity)
[2017-06-01 19:40:07,440: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:40:07,446: INFO/job_scheduler] Job 2168: RunJobThread.run
[2017-06-01 19:40:07,511: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:07,512: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:07,524: INFO/command_plan] add_jobs: created Job 2169 (Test for host connectivity)
[2017-06-01 19:40:07,568: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:40:07,608: INFO/job_scheduler] Job 2169: RunJobThread.run
[2017-06-01 19:40:07,680: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:07,681: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:07,707: INFO/command_plan] add_jobs: created Job 2170 (Test for host connectivity)
[2017-06-01 19:40:07,776: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:40:07,783: INFO/job_scheduler] Job 2170: RunJobThread.run
[2017-06-01 19:40:09,622: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:40:09,829: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:40:09,875: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:40:10,134: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:40:10,975: INFO/job_scheduler] Job 2167 finished 1 steps successfully
[2017-06-01 19:40:11,015: INFO/job_scheduler] Job 2167: complete_job: Updating cache
[2017-06-01 19:40:11,016: INFO/job_scheduler] Job 2167 completing (errored=False, cancelled=False)
[2017-06-01 19:40:11,086: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:40:11,220: INFO/job_scheduler] Job 2168 finished 1 steps successfully
[2017-06-01 19:40:11,290: INFO/job_scheduler] Job 2168: complete_job: Updating cache
[2017-06-01 19:40:11,291: INFO/job_scheduler] Job 2168 completing (errored=False, cancelled=False)
[2017-06-01 19:40:11,338: INFO/job_scheduler] Job 2169 finished 1 steps successfully
[2017-06-01 19:40:11,385: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:40:11,407: INFO/job_scheduler] Job 2169: complete_job: Updating cache
[2017-06-01 19:40:11,407: INFO/job_scheduler] Job 2169 completing (errored=False, cancelled=False)
[2017-06-01 19:40:11,435: INFO/job_scheduler] Job 2170 finished 1 steps successfully
[2017-06-01 19:40:11,499: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:40:11,561: INFO/job_scheduler] Job 2170: complete_job: Updating cache
[2017-06-01 19:40:11,561: INFO/job_scheduler] Job 2170 completing (errored=False, cancelled=False)
[2017-06-01 19:40:11,616: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:40:13,381: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:13,381: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:13,408: INFO/command_plan] add_jobs: created Job 2171 (Test for host connectivity)
[2017-06-01 19:40:13,439: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:40:13,528: INFO/job_scheduler] Job 2171: RunJobThread.run
[2017-06-01 19:40:13,568: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:13,568: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:13,597: INFO/command_plan] add_jobs: created Job 2172 (Test for host connectivity)
[2017-06-01 19:40:13,669: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:40:13,700: INFO/job_scheduler] Job 2172: RunJobThread.run
[2017-06-01 19:40:13,769: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:13,776: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:13,808: INFO/command_plan] add_jobs: created Job 2173 (Test for host connectivity)
[2017-06-01 19:40:13,903: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:40:13,928: INFO/job_scheduler] Job 2173: RunJobThread.run
[2017-06-01 19:40:13,985: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:13,986: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:14,007: INFO/command_plan] add_jobs: created Job 2174 (Test for host connectivity)
[2017-06-01 19:40:14,054: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:40:14,073: INFO/job_scheduler] Job 2174: RunJobThread.run
[2017-06-01 19:40:15,831: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:40:16,016: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:40:16,364: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:40:16,446: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:40:17,147: INFO/job_scheduler] Job 2171 finished 1 steps successfully
[2017-06-01 19:40:17,181: INFO/job_scheduler] Job 2171: complete_job: Updating cache
[2017-06-01 19:40:17,181: INFO/job_scheduler] Job 2171 completing (errored=False, cancelled=False)
[2017-06-01 19:40:17,207: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:40:17,392: INFO/job_scheduler] Job 2172 finished 1 steps successfully
[2017-06-01 19:40:17,473: INFO/job_scheduler] Job 2172: complete_job: Updating cache
[2017-06-01 19:40:17,473: INFO/job_scheduler] Job 2172 completing (errored=False, cancelled=False)
[2017-06-01 19:40:17,585: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:40:17,866: INFO/job_scheduler] Job 2174 finished 1 steps successfully
[2017-06-01 19:40:17,871: INFO/job_scheduler] Job 2173 finished 1 steps successfully
[2017-06-01 19:40:17,906: INFO/job_scheduler] Job 2174: complete_job: Updating cache
[2017-06-01 19:40:17,906: INFO/job_scheduler] Job 2174 completing (errored=False, cancelled=False)
[2017-06-01 19:40:17,939: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:40:17,964: INFO/job_scheduler] Job 2173: complete_job: Updating cache
[2017-06-01 19:40:17,965: INFO/job_scheduler] Job 2173 completing (errored=False, cancelled=False)
[2017-06-01 19:40:18,061: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:40:19,749: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:19,749: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:19,780: INFO/command_plan] add_jobs: created Job 2175 (Test for host connectivity)
[2017-06-01 19:40:19,897: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:40:19,905: INFO/job_scheduler] Job 2175: RunJobThread.run
[2017-06-01 19:40:19,938: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:19,939: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:19,947: INFO/command_plan] add_jobs: created Job 2176 (Test for host connectivity)
[2017-06-01 19:40:20,014: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:40:20,052: INFO/job_scheduler] Job 2176: RunJobThread.run
[2017-06-01 19:40:20,165: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:20,167: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:20,190: INFO/command_plan] add_jobs: created Job 2177 (Test for host connectivity)
[2017-06-01 19:40:20,231: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:40:20,266: INFO/job_scheduler] Job 2177: RunJobThread.run
[2017-06-01 19:40:20,348: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:20,350: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:20,377: INFO/command_plan] add_jobs: created Job 2178 (Test for host connectivity)
[2017-06-01 19:40:20,523: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:40:20,540: INFO/job_scheduler] Job 2178: RunJobThread.run
[2017-06-01 19:40:22,332: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:40:22,595: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:40:22,686: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:40:22,950: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:40:23,630: INFO/job_scheduler] Job 2175 finished 1 steps successfully
[2017-06-01 19:40:23,680: INFO/job_scheduler] Job 2175: complete_job: Updating cache
[2017-06-01 19:40:23,681: INFO/job_scheduler] Job 2175 completing (errored=False, cancelled=False)
[2017-06-01 19:40:23,737: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:40:23,970: INFO/job_scheduler] Job 2176 finished 1 steps successfully
[2017-06-01 19:40:24,034: INFO/job_scheduler] Job 2176: complete_job: Updating cache
[2017-06-01 19:40:24,034: INFO/job_scheduler] Job 2176 completing (errored=False, cancelled=False)
[2017-06-01 19:40:24,081: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:40:24,086: INFO/job_scheduler] Job 2177 finished 1 steps successfully
[2017-06-01 19:40:24,097: INFO/job_scheduler] Job 2177: complete_job: Updating cache
[2017-06-01 19:40:24,098: INFO/job_scheduler] Job 2177 completing (errored=False, cancelled=False)
[2017-06-01 19:40:24,202: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:40:24,459: INFO/job_scheduler] Job 2178 finished 1 steps successfully
[2017-06-01 19:40:24,522: INFO/job_scheduler] Job 2178: complete_job: Updating cache
[2017-06-01 19:40:24,522: INFO/job_scheduler] Job 2178 completing (errored=False, cancelled=False)
[2017-06-01 19:40:24,565: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:40:25,445: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:25,445: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:25,455: INFO/command_plan] add_jobs: created Job 2179 (Test for host connectivity)
[2017-06-01 19:40:25,505: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:40:25,527: INFO/job_scheduler] Job 2179: RunJobThread.run
[2017-06-01 19:40:25,578: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:25,585: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:25,597: INFO/command_plan] add_jobs: created Job 2180 (Test for host connectivity)
[2017-06-01 19:40:25,692: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:40:25,722: INFO/job_scheduler] Job 2180: RunJobThread.run
[2017-06-01 19:40:25,769: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:25,771: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:25,790: INFO/command_plan] add_jobs: created Job 2181 (Test for host connectivity)
[2017-06-01 19:40:25,803: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:40:25,850: INFO/job_scheduler] Job 2181: RunJobThread.run
[2017-06-01 19:40:25,971: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:25,973: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:25,989: INFO/command_plan] add_jobs: created Job 2182 (Test for host connectivity)
[2017-06-01 19:40:26,077: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:40:26,090: INFO/job_scheduler] Job 2182: RunJobThread.run
[2017-06-01 19:40:27,859: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:40:28,001: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:40:28,232: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:40:28,344: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:40:29,334: INFO/job_scheduler] Job 2180 finished 1 steps successfully
[2017-06-01 19:40:29,342: INFO/job_scheduler] Job 2179 finished 1 steps successfully
[2017-06-01 19:40:29,372: INFO/job_scheduler] Job 2180: complete_job: Updating cache
[2017-06-01 19:40:29,372: INFO/job_scheduler] Job 2180 completing (errored=False, cancelled=False)
[2017-06-01 19:40:29,478: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:40:29,511: INFO/job_scheduler] Job 2179: complete_job: Updating cache
[2017-06-01 19:40:29,512: INFO/job_scheduler] Job 2179 completing (errored=False, cancelled=False)
[2017-06-01 19:40:29,643: INFO/job_scheduler] Job 2181 finished 1 steps successfully
[2017-06-01 19:40:29,654: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:40:29,664: INFO/job_scheduler] Job 2181: complete_job: Updating cache
[2017-06-01 19:40:29,664: INFO/job_scheduler] Job 2181 completing (errored=False, cancelled=False)
[2017-06-01 19:40:29,706: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:40:29,957: INFO/job_scheduler] Job 2182 finished 1 steps successfully
[2017-06-01 19:40:30,022: INFO/job_scheduler] Job 2182: complete_job: Updating cache
[2017-06-01 19:40:30,022: INFO/job_scheduler] Job 2182 completing (errored=False, cancelled=False)
[2017-06-01 19:40:30,079: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:40:31,734: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:31,735: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:31,759: INFO/command_plan] add_jobs: created Job 2183 (Test for host connectivity)
[2017-06-01 19:40:31,821: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:40:31,830: INFO/job_scheduler] Job 2183: RunJobThread.run
[2017-06-01 19:40:31,898: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:31,898: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:31,913: INFO/command_plan] add_jobs: created Job 2184 (Test for host connectivity)
[2017-06-01 19:40:31,936: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:40:32,022: INFO/job_scheduler] Job 2184: RunJobThread.run
[2017-06-01 19:40:32,094: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:32,110: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:32,131: INFO/command_plan] add_jobs: created Job 2185 (Test for host connectivity)
[2017-06-01 19:40:32,217: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:40:32,251: INFO/job_scheduler] Job 2185: RunJobThread.run
[2017-06-01 19:40:32,312: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:32,313: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:32,333: INFO/command_plan] add_jobs: created Job 2186 (Test for host connectivity)
[2017-06-01 19:40:32,385: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:40:32,450: INFO/job_scheduler] Job 2186: RunJobThread.run
[2017-06-01 19:40:34,282: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:40:34,449: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:40:34,505: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:40:34,784: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:40:35,600: INFO/job_scheduler] Job 2183 finished 1 steps successfully
[2017-06-01 19:40:35,646: INFO/job_scheduler] Job 2183: complete_job: Updating cache
[2017-06-01 19:40:35,646: INFO/job_scheduler] Job 2183 completing (errored=False, cancelled=False)
[2017-06-01 19:40:35,673: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:40:35,811: INFO/job_scheduler] Job 2184 finished 1 steps successfully
[2017-06-01 19:40:35,846: INFO/job_scheduler] Job 2184: complete_job: Updating cache
[2017-06-01 19:40:35,846: INFO/job_scheduler] Job 2184 completing (errored=False, cancelled=False)
[2017-06-01 19:40:35,914: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:40:35,970: INFO/job_scheduler] Job 2185 finished 1 steps successfully
[2017-06-01 19:40:36,063: INFO/job_scheduler] Job 2185: complete_job: Updating cache
[2017-06-01 19:40:36,064: INFO/job_scheduler] Job 2185 completing (errored=False, cancelled=False)
[2017-06-01 19:40:36,112: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:40:36,201: INFO/job_scheduler] Job 2186 finished 1 steps successfully
[2017-06-01 19:40:36,239: INFO/job_scheduler] Job 2186: complete_job: Updating cache
[2017-06-01 19:40:36,239: INFO/job_scheduler] Job 2186 completing (errored=False, cancelled=False)
[2017-06-01 19:40:36,313: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:40:38,124: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:38,124: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:38,137: INFO/command_plan] add_jobs: created Job 2187 (Test for host connectivity)
[2017-06-01 19:40:38,205: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:40:38,217: INFO/job_scheduler] Job 2187: RunJobThread.run
[2017-06-01 19:40:38,294: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:38,295: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:38,321: INFO/command_plan] add_jobs: created Job 2188 (Test for host connectivity)
[2017-06-01 19:40:38,381: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:40:38,409: INFO/job_scheduler] Job 2188: RunJobThread.run
[2017-06-01 19:40:38,471: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:38,473: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:38,495: INFO/command_plan] add_jobs: created Job 2189 (Test for host connectivity)
[2017-06-01 19:40:38,611: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:38,611: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:38,620: INFO/command_plan] add_jobs: created Job 2190 (Test for host connectivity)
[2017-06-01 19:40:38,638: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:40:38,655: INFO/job_scheduler] Job 2189: RunJobThread.run
[2017-06-01 19:40:38,656: INFO/job_scheduler] Job 2190: RunJobThread.run
[2017-06-01 19:40:38,673: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:40:40,591: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:40:40,728: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:40:40,932: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:40:41,049: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:40:41,950: INFO/job_scheduler] Job 2187 finished 1 steps successfully
[2017-06-01 19:40:42,020: INFO/job_scheduler] Job 2187: complete_job: Updating cache
[2017-06-01 19:40:42,020: INFO/job_scheduler] Job 2187 completing (errored=False, cancelled=False)
[2017-06-01 19:40:42,055: INFO/job_scheduler] Job 2188 finished 1 steps successfully
[2017-06-01 19:40:42,063: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:40:42,085: INFO/job_scheduler] Job 2188: complete_job: Updating cache
[2017-06-01 19:40:42,087: INFO/job_scheduler] Job 2188 completing (errored=False, cancelled=False)
[2017-06-01 19:40:42,152: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:40:42,265: INFO/job_scheduler] Job 2190 finished 1 steps successfully
[2017-06-01 19:40:42,373: INFO/job_scheduler] Job 2190: complete_job: Updating cache
[2017-06-01 19:40:42,373: INFO/job_scheduler] Job 2190 completing (errored=False, cancelled=False)
[2017-06-01 19:40:42,540: INFO/job_scheduler] Job 2189 finished 1 steps successfully
[2017-06-01 19:40:42,564: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:40:42,602: INFO/job_scheduler] Job 2189: complete_job: Updating cache
[2017-06-01 19:40:42,602: INFO/job_scheduler] Job 2189 completing (errored=False, cancelled=False)
[2017-06-01 19:40:42,641: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:40:44,292: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:44,292: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:44,301: INFO/command_plan] add_jobs: created Job 2191 (Test for host connectivity)
[2017-06-01 19:40:44,440: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:44,442: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:44,456: INFO/command_plan] add_jobs: created Job 2192 (Test for host connectivity)
[2017-06-01 19:40:44,487: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:40:44,497: INFO/job_scheduler] Job 2192: RunJobThread.run
[2017-06-01 19:40:44,568: INFO/job_scheduler] Job 2191: RunJobThread.run
[2017-06-01 19:40:44,577: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:40:44,673: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:44,674: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:44,719: INFO/command_plan] add_jobs: created Job 2193 (Test for host connectivity)
[2017-06-01 19:40:44,796: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:40:44,822: INFO/job_scheduler] Job 2193: RunJobThread.run
[2017-06-01 19:40:44,944: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:44,948: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:44,962: INFO/command_plan] add_jobs: created Job 2194 (Test for host connectivity)
[2017-06-01 19:40:44,988: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:40:44,990: INFO/job_scheduler] Job 2194: RunJobThread.run
[2017-06-01 19:40:46,921: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:40:46,951: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:40:47,209: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:40:47,292: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:40:48,257: INFO/job_scheduler] Job 2191 finished 1 steps successfully
[2017-06-01 19:40:48,292: INFO/job_scheduler] Job 2191: complete_job: Updating cache
[2017-06-01 19:40:48,292: INFO/job_scheduler] Job 2191 completing (errored=False, cancelled=False)
[2017-06-01 19:40:48,340: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:40:48,367: INFO/job_scheduler] Job 2192 finished 1 steps successfully
[2017-06-01 19:40:48,391: INFO/job_scheduler] Job 2192: complete_job: Updating cache
[2017-06-01 19:40:48,394: INFO/job_scheduler] Job 2192 completing (errored=False, cancelled=False)
[2017-06-01 19:40:48,478: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:40:48,597: INFO/job_scheduler] Job 2193 finished 1 steps successfully
[2017-06-01 19:40:48,647: INFO/job_scheduler] Job 2193: complete_job: Updating cache
[2017-06-01 19:40:48,660: INFO/job_scheduler] Job 2193 completing (errored=False, cancelled=False)
[2017-06-01 19:40:48,690: INFO/job_scheduler] Job 2194 finished 1 steps successfully
[2017-06-01 19:40:48,728: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:40:48,745: INFO/job_scheduler] Job 2194: complete_job: Updating cache
[2017-06-01 19:40:48,746: INFO/job_scheduler] Job 2194 completing (errored=False, cancelled=False)
[2017-06-01 19:40:48,832: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:40:50,671: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:50,671: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:50,693: INFO/command_plan] add_jobs: created Job 2195 (Test for host connectivity)
[2017-06-01 19:40:50,801: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:40:50,808: INFO/job_scheduler] Job 2195: RunJobThread.run
[2017-06-01 19:40:50,870: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:50,870: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:50,887: INFO/command_plan] add_jobs: created Job 2196 (Test for host connectivity)
[2017-06-01 19:40:50,923: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:40:50,958: INFO/job_scheduler] Job 2196: RunJobThread.run
[2017-06-01 19:40:51,039: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:51,040: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:51,067: INFO/command_plan] add_jobs: created Job 2197 (Test for host connectivity)
[2017-06-01 19:40:51,095: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:40:51,143: INFO/job_scheduler] Job 2197: RunJobThread.run
[2017-06-01 19:40:51,291: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:51,292: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:51,335: INFO/command_plan] add_jobs: created Job 2198 (Test for host connectivity)
[2017-06-01 19:40:51,429: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:40:51,438: INFO/job_scheduler] Job 2198: RunJobThread.run
[2017-06-01 19:40:53,083: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:40:53,299: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:40:53,544: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:40:53,732: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:40:54,423: INFO/job_scheduler] Job 2195 finished 1 steps successfully
[2017-06-01 19:40:54,458: INFO/job_scheduler] Job 2195: complete_job: Updating cache
[2017-06-01 19:40:54,458: INFO/job_scheduler] Job 2195 completing (errored=False, cancelled=False)
[2017-06-01 19:40:54,494: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:40:54,645: INFO/job_scheduler] Job 2196 finished 1 steps successfully
[2017-06-01 19:40:54,732: INFO/job_scheduler] Job 2196: complete_job: Updating cache
[2017-06-01 19:40:54,733: INFO/job_scheduler] Job 2196 completing (errored=False, cancelled=False)
[2017-06-01 19:40:54,834: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:40:55,073: INFO/job_scheduler] Job 2197 finished 1 steps successfully
[2017-06-01 19:40:55,132: INFO/job_scheduler] Job 2197: complete_job: Updating cache
[2017-06-01 19:40:55,133: INFO/job_scheduler] Job 2197 completing (errored=False, cancelled=False)
[2017-06-01 19:40:55,160: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:40:55,177: INFO/job_scheduler] Job 2198 finished 1 steps successfully
[2017-06-01 19:40:55,225: INFO/job_scheduler] Job 2198: complete_job: Updating cache
[2017-06-01 19:40:55,226: INFO/job_scheduler] Job 2198 completing (errored=False, cancelled=False)
[2017-06-01 19:40:55,282: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:40:57,086: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:57,086: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:57,147: INFO/command_plan] add_jobs: created Job 2199 (Test for host connectivity)
[2017-06-01 19:40:57,232: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:40:57,236: INFO/job_scheduler] Job 2199: RunJobThread.run
[2017-06-01 19:40:57,264: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:57,265: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:57,275: INFO/command_plan] add_jobs: created Job 2200 (Test for host connectivity)
[2017-06-01 19:40:57,302: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:40:57,334: INFO/job_scheduler] Job 2200: RunJobThread.run
[2017-06-01 19:40:57,460: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:57,463: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:57,484: INFO/command_plan] add_jobs: created Job 2201 (Test for host connectivity)
[2017-06-01 19:40:57,500: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:40:57,562: INFO/job_scheduler] Job 2201: RunJobThread.run
[2017-06-01 19:40:57,638: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:40:57,639: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:40:57,651: INFO/command_plan] add_jobs: created Job 2202 (Test for host connectivity)
[2017-06-01 19:40:57,660: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:40:57,672: INFO/job_scheduler] Job 2202: RunJobThread.run
[2017-06-01 19:40:59,510: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:40:59,787: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:41:00,087: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:41:00,098: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:41:00,897: INFO/job_scheduler] Job 2199 finished 1 steps successfully
[2017-06-01 19:41:00,925: INFO/job_scheduler] Job 2199: complete_job: Updating cache
[2017-06-01 19:41:00,925: INFO/job_scheduler] Job 2199 completing (errored=False, cancelled=False)
[2017-06-01 19:41:00,960: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:41:01,135: INFO/job_scheduler] Job 2200 finished 1 steps successfully
[2017-06-01 19:41:01,184: INFO/job_scheduler] Job 2200: complete_job: Updating cache
[2017-06-01 19:41:01,184: INFO/job_scheduler] Job 2200 completing (errored=False, cancelled=False)
[2017-06-01 19:41:01,285: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:41:01,493: INFO/job_scheduler] Job 2202 finished 1 steps successfully
[2017-06-01 19:41:01,559: INFO/job_scheduler] Job 2202: complete_job: Updating cache
[2017-06-01 19:41:01,561: INFO/job_scheduler] Job 2202 completing (errored=False, cancelled=False)
[2017-06-01 19:41:01,601: INFO/job_scheduler] Job 2201 finished 1 steps successfully
[2017-06-01 19:41:01,634: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:41:01,696: INFO/job_scheduler] Job 2201: complete_job: Updating cache
[2017-06-01 19:41:01,696: INFO/job_scheduler] Job 2201 completing (errored=False, cancelled=False)
[2017-06-01 19:41:01,823: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:41:03,357: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:03,359: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:03,366: INFO/command_plan] add_jobs: created Job 2203 (Test for host connectivity)
[2017-06-01 19:41:03,392: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:41:03,396: INFO/job_scheduler] Job 2203: RunJobThread.run
[2017-06-01 19:41:03,498: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:03,498: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:03,509: INFO/command_plan] add_jobs: created Job 2204 (Test for host connectivity)
[2017-06-01 19:41:03,561: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:41:03,624: INFO/job_scheduler] Job 2204: RunJobThread.run
[2017-06-01 19:41:03,695: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:03,697: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:03,734: INFO/command_plan] add_jobs: created Job 2205 (Test for host connectivity)
[2017-06-01 19:41:03,798: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:41:03,816: INFO/job_scheduler] Job 2205: RunJobThread.run
[2017-06-01 19:41:03,888: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:03,890: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:03,925: INFO/command_plan] add_jobs: created Job 2206 (Test for host connectivity)
[2017-06-01 19:41:03,971: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:41:03,974: INFO/job_scheduler] Job 2206: RunJobThread.run
[2017-06-01 19:41:05,767: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:41:05,958: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:41:06,080: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:41:06,282: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:41:07,140: INFO/job_scheduler] Job 2203 finished 1 steps successfully
[2017-06-01 19:41:07,257: INFO/job_scheduler] Job 2203: complete_job: Updating cache
[2017-06-01 19:41:07,258: INFO/job_scheduler] Job 2203 completing (errored=False, cancelled=False)
[2017-06-01 19:41:07,325: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:41:07,341: INFO/job_scheduler] Job 2204 finished 1 steps successfully
[2017-06-01 19:41:07,356: INFO/job_scheduler] Job 2204: complete_job: Updating cache
[2017-06-01 19:41:07,356: INFO/job_scheduler] Job 2204 completing (errored=False, cancelled=False)
[2017-06-01 19:41:07,422: INFO/job_scheduler] Job 2205 finished 1 steps successfully
[2017-06-01 19:41:07,462: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:41:07,530: INFO/job_scheduler] Job 2205: complete_job: Updating cache
[2017-06-01 19:41:07,530: INFO/job_scheduler] Job 2205 completing (errored=False, cancelled=False)
[2017-06-01 19:41:07,595: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:41:07,638: INFO/job_scheduler] Job 2206 finished 1 steps successfully
[2017-06-01 19:41:07,664: INFO/job_scheduler] Job 2206: complete_job: Updating cache
[2017-06-01 19:41:07,664: INFO/job_scheduler] Job 2206 completing (errored=False, cancelled=False)
[2017-06-01 19:41:07,711: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:41:09,693: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:09,693: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:09,703: INFO/command_plan] add_jobs: created Job 2207 (Test for host connectivity)
[2017-06-01 19:41:09,718: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:41:09,727: INFO/job_scheduler] Job 2207: RunJobThread.run
[2017-06-01 19:41:09,891: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:09,892: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:09,912: INFO/command_plan] add_jobs: created Job 2208 (Test for host connectivity)
[2017-06-01 19:41:10,074: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:41:10,078: INFO/job_scheduler] Job 2208: RunJobThread.run
[2017-06-01 19:41:10,104: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:10,105: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:10,125: INFO/command_plan] add_jobs: created Job 2209 (Test for host connectivity)
[2017-06-01 19:41:10,247: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:41:10,262: INFO/job_scheduler] Job 2209: RunJobThread.run
[2017-06-01 19:41:10,345: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:10,353: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:10,391: INFO/command_plan] add_jobs: created Job 2210 (Test for host connectivity)
[2017-06-01 19:41:10,444: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:41:10,457: INFO/job_scheduler] Job 2210: RunJobThread.run
[2017-06-01 19:41:12,121: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:41:12,470: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:41:12,583: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:41:12,848: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:41:13,468: INFO/job_scheduler] Job 2207 finished 1 steps successfully
[2017-06-01 19:41:13,494: INFO/job_scheduler] Job 2207: complete_job: Updating cache
[2017-06-01 19:41:13,494: INFO/job_scheduler] Job 2207 completing (errored=False, cancelled=False)
[2017-06-01 19:41:13,558: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:41:13,866: INFO/job_scheduler] Job 2208 finished 1 steps successfully
[2017-06-01 19:41:13,913: INFO/job_scheduler] Job 2208: complete_job: Updating cache
[2017-06-01 19:41:13,931: INFO/job_scheduler] Job 2208 completing (errored=False, cancelled=False)
[2017-06-01 19:41:13,962: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:41:14,043: INFO/job_scheduler] Job 2209 finished 1 steps successfully
[2017-06-01 19:41:14,104: INFO/job_scheduler] Job 2209: complete_job: Updating cache
[2017-06-01 19:41:14,104: INFO/job_scheduler] Job 2209 completing (errored=False, cancelled=False)
[2017-06-01 19:41:14,144: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:41:14,194: INFO/job_scheduler] Job 2210 finished 1 steps successfully
[2017-06-01 19:41:14,327: INFO/job_scheduler] Job 2210: complete_job: Updating cache
[2017-06-01 19:41:14,327: INFO/job_scheduler] Job 2210 completing (errored=False, cancelled=False)
[2017-06-01 19:41:14,411: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:41:16,164: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:16,165: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:16,177: INFO/command_plan] add_jobs: created Job 2211 (Test for host connectivity)
[2017-06-01 19:41:16,264: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:16,264: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:16,287: INFO/command_plan] add_jobs: created Job 2212 (Test for host connectivity)
[2017-06-01 19:41:16,385: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:41:16,390: INFO/job_scheduler] Job 2211: RunJobThread.run
[2017-06-01 19:41:16,392: INFO/job_scheduler] Job 2212: RunJobThread.run
[2017-06-01 19:41:16,415: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:41:16,477: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:16,478: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:16,505: INFO/command_plan] add_jobs: created Job 2213 (Test for host connectivity)
[2017-06-01 19:41:16,570: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:41:16,586: INFO/job_scheduler] Job 2213: RunJobThread.run
[2017-06-01 19:41:16,679: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:16,680: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:16,694: INFO/command_plan] add_jobs: created Job 2214 (Test for host connectivity)
[2017-06-01 19:41:16,753: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:41:16,804: INFO/job_scheduler] Job 2214: RunJobThread.run
[2017-06-01 19:41:18,631: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:41:18,773: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:41:18,919: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:41:19,058: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:41:20,041: INFO/job_scheduler] Job 2211 finished 1 steps successfully
[2017-06-01 19:41:20,093: INFO/job_scheduler] Job 2211: complete_job: Updating cache
[2017-06-01 19:41:20,093: INFO/job_scheduler] Job 2211 completing (errored=False, cancelled=False)
[2017-06-01 19:41:20,150: INFO/job_scheduler] Job 2212 finished 1 steps successfully
[2017-06-01 19:41:20,314: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:41:20,335: INFO/job_scheduler] Job 2212: complete_job: Updating cache
[2017-06-01 19:41:20,335: INFO/job_scheduler] Job 2212 completing (errored=False, cancelled=False)
[2017-06-01 19:41:20,353: INFO/job_scheduler] Job 2213 finished 1 steps successfully
[2017-06-01 19:41:20,463: INFO/job_scheduler] Job 2214 finished 1 steps successfully
[2017-06-01 19:41:20,463: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:41:20,478: INFO/job_scheduler] Job 2213: complete_job: Updating cache
[2017-06-01 19:41:20,478: INFO/job_scheduler] Job 2213 completing (errored=False, cancelled=False)
[2017-06-01 19:41:20,551: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:41:20,594: INFO/job_scheduler] Job 2214: complete_job: Updating cache
[2017-06-01 19:41:20,594: INFO/job_scheduler] Job 2214 completing (errored=False, cancelled=False)
[2017-06-01 19:41:20,683: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:41:23,148: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:23,148: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:23,162: INFO/command_plan] add_jobs: created Job 2215 (Test for host connectivity)
[2017-06-01 19:41:23,207: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:41:23,222: INFO/job_scheduler] Job 2215: RunJobThread.run
[2017-06-01 19:41:23,327: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:23,328: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:23,367: INFO/command_plan] add_jobs: created Job 2216 (Test for host connectivity)
[2017-06-01 19:41:23,420: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:41:23,435: INFO/job_scheduler] Job 2216: RunJobThread.run
[2017-06-01 19:41:23,500: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:23,501: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:23,596: INFO/command_plan] add_jobs: created Job 2217 (Test for host connectivity)
[2017-06-01 19:41:23,676: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:41:23,716: INFO/job_scheduler] Job 2217: RunJobThread.run
[2017-06-01 19:41:23,770: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:23,772: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:23,802: INFO/command_plan] add_jobs: created Job 2218 (Test for host connectivity)
[2017-06-01 19:41:23,857: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:41:23,869: INFO/job_scheduler] Job 2218: RunJobThread.run
[2017-06-01 19:41:25,575: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:41:25,863: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:41:26,132: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:41:26,175: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:41:26,942: INFO/job_scheduler] Job 2215 finished 1 steps successfully
[2017-06-01 19:41:26,984: INFO/job_scheduler] Job 2215: complete_job: Updating cache
[2017-06-01 19:41:26,984: INFO/job_scheduler] Job 2215 completing (errored=False, cancelled=False)
[2017-06-01 19:41:27,010: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:41:27,245: INFO/job_scheduler] Job 2216 finished 1 steps successfully
[2017-06-01 19:41:27,317: INFO/job_scheduler] Job 2216: complete_job: Updating cache
[2017-06-01 19:41:27,319: INFO/job_scheduler] Job 2216 completing (errored=False, cancelled=False)
[2017-06-01 19:41:27,407: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:41:27,594: INFO/job_scheduler] Job 2217 finished 1 steps successfully
[2017-06-01 19:41:27,610: INFO/job_scheduler] Job 2217: complete_job: Updating cache
[2017-06-01 19:41:27,610: INFO/job_scheduler] Job 2217 completing (errored=False, cancelled=False)
[2017-06-01 19:41:27,612: INFO/job_scheduler] Job 2218 finished 1 steps successfully
[2017-06-01 19:41:27,684: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:41:27,694: INFO/job_scheduler] Job 2218: complete_job: Updating cache
[2017-06-01 19:41:27,696: INFO/job_scheduler] Job 2218 completing (errored=False, cancelled=False)
[2017-06-01 19:41:27,876: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:41:29,650: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:29,652: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:29,669: INFO/command_plan] add_jobs: created Job 2219 (Test for host connectivity)
[2017-06-01 19:41:29,758: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:41:29,775: INFO/job_scheduler] Job 2219: RunJobThread.run
[2017-06-01 19:41:29,886: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:29,886: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:29,901: INFO/command_plan] add_jobs: created Job 2220 (Test for host connectivity)
[2017-06-01 19:41:29,947: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:41:29,980: INFO/job_scheduler] Job 2220: RunJobThread.run
[2017-06-01 19:41:30,047: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:30,047: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:30,087: INFO/command_plan] add_jobs: created Job 2221 (Test for host connectivity)
[2017-06-01 19:41:30,169: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:41:30,172: INFO/job_scheduler] Job 2221: RunJobThread.run
[2017-06-01 19:41:30,220: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:30,220: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:30,246: INFO/command_plan] add_jobs: created Job 2222 (Test for host connectivity)
[2017-06-01 19:41:30,324: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:41:30,424: INFO/job_scheduler] Job 2222: RunJobThread.run
[2017-06-01 19:41:32,409: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:41:32,455: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:41:32,588: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:41:32,876: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:41:33,767: INFO/job_scheduler] Job 2220 finished 1 steps successfully
[2017-06-01 19:41:33,817: INFO/job_scheduler] Job 2220: complete_job: Updating cache
[2017-06-01 19:41:33,817: INFO/job_scheduler] Job 2220 completing (errored=False, cancelled=False)
[2017-06-01 19:41:33,842: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:41:33,954: INFO/job_scheduler] Job 2219 finished 1 steps successfully
[2017-06-01 19:41:34,017: INFO/job_scheduler] Job 2219: complete_job: Updating cache
[2017-06-01 19:41:34,018: INFO/job_scheduler] Job 2219 completing (errored=False, cancelled=False)
[2017-06-01 19:41:34,056: INFO/job_scheduler] Job 2221 finished 1 steps successfully
[2017-06-01 19:41:34,106: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:41:34,135: INFO/job_scheduler] Job 2221: complete_job: Updating cache
[2017-06-01 19:41:34,135: INFO/job_scheduler] Job 2221 completing (errored=False, cancelled=False)
[2017-06-01 19:41:34,188: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:41:34,245: INFO/job_scheduler] Job 2222 finished 1 steps successfully
[2017-06-01 19:41:34,267: INFO/job_scheduler] Job 2222: complete_job: Updating cache
[2017-06-01 19:41:34,268: INFO/job_scheduler] Job 2222 completing (errored=False, cancelled=False)
[2017-06-01 19:41:34,346: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:41:36,153: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:36,154: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:36,168: INFO/command_plan] add_jobs: created Job 2223 (Test for host connectivity)
[2017-06-01 19:41:36,225: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:41:36,228: INFO/job_scheduler] Job 2223: RunJobThread.run
[2017-06-01 19:41:36,336: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:36,338: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:36,369: INFO/command_plan] add_jobs: created Job 2224 (Test for host connectivity)
[2017-06-01 19:41:36,480: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:41:36,492: INFO/job_scheduler] Job 2224: RunJobThread.run
[2017-06-01 19:41:36,551: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:36,556: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:36,576: INFO/command_plan] add_jobs: created Job 2225 (Test for host connectivity)
[2017-06-01 19:41:36,737: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:41:36,756: INFO/job_scheduler] Job 2225: RunJobThread.run
[2017-06-01 19:41:36,912: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:36,917: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:36,951: INFO/command_plan] add_jobs: created Job 2226 (Test for host connectivity)
[2017-06-01 19:41:37,015: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:41:37,059: INFO/job_scheduler] Job 2226: RunJobThread.run
[2017-06-01 19:41:38,601: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:41:38,873: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:41:39,162: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:41:39,349: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:41:39,941: INFO/job_scheduler] Job 2223 finished 1 steps successfully
[2017-06-01 19:41:39,974: INFO/job_scheduler] Job 2223: complete_job: Updating cache
[2017-06-01 19:41:39,975: INFO/job_scheduler] Job 2223 completing (errored=False, cancelled=False)
[2017-06-01 19:41:40,033: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:41:40,206: INFO/job_scheduler] Job 2224 finished 1 steps successfully
[2017-06-01 19:41:40,244: INFO/job_scheduler] Job 2224: complete_job: Updating cache
[2017-06-01 19:41:40,245: INFO/job_scheduler] Job 2224 completing (errored=False, cancelled=False)
[2017-06-01 19:41:40,333: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:41:40,606: INFO/job_scheduler] Job 2225 finished 1 steps successfully
[2017-06-01 19:41:40,641: INFO/job_scheduler] Job 2225: complete_job: Updating cache
[2017-06-01 19:41:40,641: INFO/job_scheduler] Job 2225 completing (errored=False, cancelled=False)
[2017-06-01 19:41:40,670: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:41:40,760: INFO/job_scheduler] Job 2226 finished 1 steps successfully
[2017-06-01 19:41:40,816: INFO/job_scheduler] Job 2226: complete_job: Updating cache
[2017-06-01 19:41:40,817: INFO/job_scheduler] Job 2226 completing (errored=False, cancelled=False)
[2017-06-01 19:41:40,892: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:41:41,554: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:41,554: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:41,568: INFO/command_plan] add_jobs: created Job 2227 (Test for host connectivity)
[2017-06-01 19:41:41,638: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:41,639: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:41,727: INFO/command_plan] add_jobs: created Job 2228 (Test for host connectivity)
[2017-06-01 19:41:41,786: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:41:41,807: INFO/job_scheduler] Job 2227: RunJobThread.run
[2017-06-01 19:41:41,831: INFO/job_scheduler] Job 2228: RunJobThread.run
[2017-06-01 19:41:41,876: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:41,876: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:41,895: INFO/command_plan] add_jobs: created Job 2229 (Test for host connectivity)
[2017-06-01 19:41:41,934: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:41:41,951: INFO/job_scheduler] Job 2229: RunJobThread.run
[2017-06-01 19:41:42,066: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:42,068: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:42,100: INFO/command_plan] add_jobs: created Job 2230 (Test for host connectivity)
[2017-06-01 19:41:42,130: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:41:42,198: INFO/job_scheduler] Job 2230: RunJobThread.run
[2017-06-01 19:41:42,246: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:41:44,097: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:41:44,209: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:41:44,297: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:41:44,459: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:41:45,441: INFO/job_scheduler] Job 2227 finished 1 steps successfully
[2017-06-01 19:41:45,491: INFO/job_scheduler] Job 2227: complete_job: Updating cache
[2017-06-01 19:41:45,491: INFO/job_scheduler] Job 2227 completing (errored=False, cancelled=False)
[2017-06-01 19:41:45,527: INFO/job_scheduler] Job 2228 finished 1 steps successfully
[2017-06-01 19:41:45,559: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:41:45,574: INFO/job_scheduler] Job 2228: complete_job: Updating cache
[2017-06-01 19:41:45,576: INFO/job_scheduler] Job 2228 completing (errored=False, cancelled=False)
[2017-06-01 19:41:45,610: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:41:45,660: INFO/job_scheduler] Job 2229 finished 1 steps successfully
[2017-06-01 19:41:45,683: INFO/job_scheduler] Job 2229: complete_job: Updating cache
[2017-06-01 19:41:45,683: INFO/job_scheduler] Job 2229 completing (errored=False, cancelled=False)
[2017-06-01 19:41:45,761: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:41:45,776: INFO/job_scheduler] Job 2230 finished 1 steps successfully
[2017-06-01 19:41:45,850: INFO/job_scheduler] Job 2230: complete_job: Updating cache
[2017-06-01 19:41:45,851: INFO/job_scheduler] Job 2230 completing (errored=False, cancelled=False)
[2017-06-01 19:41:45,924: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:41:47,956: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:47,958: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:47,991: INFO/command_plan] add_jobs: created Job 2231 (Test for host connectivity)
[2017-06-01 19:41:48,082: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:48,082: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:48,120: INFO/command_plan] add_jobs: created Job 2232 (Test for host connectivity)
[2017-06-01 19:41:48,141: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:41:48,153: INFO/job_scheduler] Job 2232: RunJobThread.run
[2017-06-01 19:41:48,153: INFO/job_scheduler] Job 2231: RunJobThread.run
[2017-06-01 19:41:48,162: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:41:48,265: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:48,266: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:48,302: INFO/command_plan] add_jobs: created Job 2233 (Test for host connectivity)
[2017-06-01 19:41:48,375: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:41:48,409: INFO/job_scheduler] Job 2233: RunJobThread.run
[2017-06-01 19:41:48,471: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:48,472: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:48,490: INFO/command_plan] add_jobs: created Job 2234 (Test for host connectivity)
[2017-06-01 19:41:48,529: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:41:48,570: INFO/job_scheduler] Job 2234: RunJobThread.run
[2017-06-01 19:41:50,371: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:41:50,575: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:41:50,691: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:41:50,986: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:41:51,677: INFO/job_scheduler] Job 2231 finished 1 steps successfully
[2017-06-01 19:41:51,710: INFO/job_scheduler] Job 2231: complete_job: Updating cache
[2017-06-01 19:41:51,710: INFO/job_scheduler] Job 2231 completing (errored=False, cancelled=False)
[2017-06-01 19:41:51,782: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:41:51,907: INFO/job_scheduler] Job 2232 finished 1 steps successfully
[2017-06-01 19:41:51,960: INFO/job_scheduler] Job 2232: complete_job: Updating cache
[2017-06-01 19:41:51,960: INFO/job_scheduler] Job 2232 completing (errored=False, cancelled=False)
[2017-06-01 19:41:52,026: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:41:52,032: INFO/job_scheduler] Job 2233 finished 1 steps successfully
[2017-06-01 19:41:52,054: INFO/job_scheduler] Job 2233: complete_job: Updating cache
[2017-06-01 19:41:52,055: INFO/job_scheduler] Job 2233 completing (errored=False, cancelled=False)
[2017-06-01 19:41:52,189: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:41:52,400: INFO/job_scheduler] Job 2234 finished 1 steps successfully
[2017-06-01 19:41:52,523: INFO/job_scheduler] Job 2234: complete_job: Updating cache
[2017-06-01 19:41:52,523: INFO/job_scheduler] Job 2234 completing (errored=False, cancelled=False)
[2017-06-01 19:41:52,585: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:41:54,353: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:54,354: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:54,409: INFO/command_plan] add_jobs: created Job 2235 (Test for host connectivity)
[2017-06-01 19:41:54,504: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:54,504: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:54,516: INFO/command_plan] add_jobs: created Job 2236 (Test for host connectivity)
[2017-06-01 19:41:54,532: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:41:54,555: INFO/job_scheduler] Job 2235: RunJobThread.run
[2017-06-01 19:41:54,584: INFO/job_scheduler] Job 2236: RunJobThread.run
[2017-06-01 19:41:54,658: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:54,659: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:54,689: INFO/command_plan] add_jobs: created Job 2237 (Test for host connectivity)
[2017-06-01 19:41:54,718: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:41:54,748: INFO/job_scheduler] Job 2237: RunJobThread.run
[2017-06-01 19:41:54,924: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:41:54,932: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:41:54,966: INFO/command_plan] add_jobs: created Job 2238 (Test for host connectivity)
[2017-06-01 19:41:55,017: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:41:55,049: INFO/job_scheduler] Job 2238: RunJobThread.run
[2017-06-01 19:41:55,082: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:41:56,943: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:41:57,175: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:41:57,192: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:41:57,438: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:41:58,274: INFO/job_scheduler] Job 2235 finished 1 steps successfully
[2017-06-01 19:41:58,305: INFO/job_scheduler] Job 2235: complete_job: Updating cache
[2017-06-01 19:41:58,306: INFO/job_scheduler] Job 2235 completing (errored=False, cancelled=False)
[2017-06-01 19:41:58,386: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:41:58,558: INFO/job_scheduler] Job 2237 finished 1 steps successfully
[2017-06-01 19:41:58,597: INFO/job_scheduler] Job 2236 finished 1 steps successfully
[2017-06-01 19:41:58,643: INFO/job_scheduler] Job 2237: complete_job: Updating cache
[2017-06-01 19:41:58,646: INFO/job_scheduler] Job 2237 completing (errored=False, cancelled=False)
[2017-06-01 19:41:58,859: INFO/job_scheduler] Job 2238 finished 1 steps successfully
[2017-06-01 19:41:58,860: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:41:58,868: INFO/job_scheduler] Job 2236: complete_job: Updating cache
[2017-06-01 19:41:58,869: INFO/job_scheduler] Job 2236 completing (errored=False, cancelled=False)
[2017-06-01 19:41:58,993: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:41:59,023: INFO/job_scheduler] Job 2238: complete_job: Updating cache
[2017-06-01 19:41:59,023: INFO/job_scheduler] Job 2238 completing (errored=False, cancelled=False)
[2017-06-01 19:41:59,081: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:42:00,795: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:00,795: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:00,809: INFO/command_plan] add_jobs: created Job 2239 (Test for host connectivity)
[2017-06-01 19:42:00,884: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:42:00,934: INFO/job_scheduler] Job 2239: RunJobThread.run
[2017-06-01 19:42:00,984: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:00,985: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:01,003: INFO/command_plan] add_jobs: created Job 2240 (Test for host connectivity)
[2017-06-01 19:42:01,143: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:42:01,160: INFO/job_scheduler] Job 2240: RunJobThread.run
[2017-06-01 19:42:01,205: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:01,206: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:01,243: INFO/command_plan] add_jobs: created Job 2241 (Test for host connectivity)
[2017-06-01 19:42:01,277: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:42:01,331: INFO/job_scheduler] Job 2241: RunJobThread.run
[2017-06-01 19:42:01,399: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:01,400: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:01,507: INFO/command_plan] add_jobs: created Job 2242 (Test for host connectivity)
[2017-06-01 19:42:01,603: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:42:01,652: INFO/job_scheduler] Job 2242: RunJobThread.run
[2017-06-01 19:42:03,364: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:42:03,570: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:42:03,732: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:42:03,932: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:42:04,695: INFO/job_scheduler] Job 2239 finished 1 steps successfully
[2017-06-01 19:42:04,728: INFO/job_scheduler] Job 2239: complete_job: Updating cache
[2017-06-01 19:42:04,729: INFO/job_scheduler] Job 2239 completing (errored=False, cancelled=False)
[2017-06-01 19:42:04,802: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:42:04,890: INFO/job_scheduler] Job 2240 finished 1 steps successfully
[2017-06-01 19:42:04,970: INFO/job_scheduler] Job 2240: complete_job: Updating cache
[2017-06-01 19:42:04,970: INFO/job_scheduler] Job 2240 completing (errored=False, cancelled=False)
[2017-06-01 19:42:05,031: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:42:05,101: INFO/job_scheduler] Job 2241 finished 1 steps successfully
[2017-06-01 19:42:05,267: INFO/job_scheduler] Job 2241: complete_job: Updating cache
[2017-06-01 19:42:05,267: INFO/job_scheduler] Job 2241 completing (errored=False, cancelled=False)
[2017-06-01 19:42:05,334: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:42:05,419: INFO/job_scheduler] Job 2242 finished 1 steps successfully
[2017-06-01 19:42:05,475: INFO/job_scheduler] Job 2242: complete_job: Updating cache
[2017-06-01 19:42:05,477: INFO/job_scheduler] Job 2242 completing (errored=False, cancelled=False)
[2017-06-01 19:42:05,550: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:42:07,387: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:07,389: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:07,418: INFO/command_plan] add_jobs: created Job 2243 (Test for host connectivity)
[2017-06-01 19:42:07,523: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:42:07,575: INFO/job_scheduler] Job 2243: RunJobThread.run
[2017-06-01 19:42:07,619: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:07,620: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:07,637: INFO/command_plan] add_jobs: created Job 2244 (Test for host connectivity)
[2017-06-01 19:42:07,771: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:42:07,778: INFO/job_scheduler] Job 2244: RunJobThread.run
[2017-06-01 19:42:07,818: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:07,819: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:07,860: INFO/command_plan] add_jobs: created Job 2245 (Test for host connectivity)
[2017-06-01 19:42:07,933: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:42:07,975: INFO/job_scheduler] Job 2245: RunJobThread.run
[2017-06-01 19:42:08,082: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:08,093: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:08,149: INFO/command_plan] add_jobs: created Job 2246 (Test for host connectivity)
[2017-06-01 19:42:08,271: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:42:08,278: INFO/job_scheduler] Job 2246: RunJobThread.run
[2017-06-01 19:42:10,083: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:42:10,166: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:42:10,432: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:42:10,488: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:42:11,419: INFO/job_scheduler] Job 2243 finished 1 steps successfully
[2017-06-01 19:42:11,457: INFO/job_scheduler] Job 2243: complete_job: Updating cache
[2017-06-01 19:42:11,457: INFO/job_scheduler] Job 2243 completing (errored=False, cancelled=False)
[2017-06-01 19:42:11,507: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:42:11,511: INFO/job_scheduler] Job 2244 finished 1 steps successfully
[2017-06-01 19:42:11,537: INFO/job_scheduler] Job 2244: complete_job: Updating cache
[2017-06-01 19:42:11,556: INFO/job_scheduler] Job 2244 completing (errored=False, cancelled=False)
[2017-06-01 19:42:11,666: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:42:11,903: INFO/job_scheduler] Job 2245 finished 1 steps successfully
[2017-06-01 19:42:11,920: INFO/job_scheduler] Job 2246 finished 1 steps successfully
[2017-06-01 19:42:11,965: INFO/job_scheduler] Job 2245: complete_job: Updating cache
[2017-06-01 19:42:11,965: INFO/job_scheduler] Job 2245 completing (errored=False, cancelled=False)
[2017-06-01 19:42:12,038: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:42:12,058: INFO/job_scheduler] Job 2246: complete_job: Updating cache
[2017-06-01 19:42:12,058: INFO/job_scheduler] Job 2246 completing (errored=False, cancelled=False)
[2017-06-01 19:42:12,157: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:42:13,964: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:13,964: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:13,973: INFO/command_plan] add_jobs: created Job 2247 (Test for host connectivity)
[2017-06-01 19:42:13,994: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:42:14,020: INFO/job_scheduler] Job 2247: RunJobThread.run
[2017-06-01 19:42:14,088: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:14,088: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:14,099: INFO/command_plan] add_jobs: created Job 2248 (Test for host connectivity)
[2017-06-01 19:42:14,148: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:42:14,182: INFO/job_scheduler] Job 2248: RunJobThread.run
[2017-06-01 19:42:14,287: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:14,288: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:14,307: INFO/command_plan] add_jobs: created Job 2249 (Test for host connectivity)
[2017-06-01 19:42:14,337: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:42:14,382: INFO/job_scheduler] Job 2249: RunJobThread.run
[2017-06-01 19:42:14,540: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:14,540: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:14,598: INFO/command_plan] add_jobs: created Job 2250 (Test for host connectivity)
[2017-06-01 19:42:14,725: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:42:14,735: INFO/job_scheduler] Job 2250: RunJobThread.run
[2017-06-01 19:42:16,397: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:42:16,581: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:42:16,743: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:42:16,830: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:42:17,780: INFO/job_scheduler] Job 2247 finished 1 steps successfully
[2017-06-01 19:42:17,797: INFO/job_scheduler] Job 2247: complete_job: Updating cache
[2017-06-01 19:42:17,798: INFO/job_scheduler] Job 2247 completing (errored=False, cancelled=False)
[2017-06-01 19:42:17,879: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:42:17,969: INFO/job_scheduler] Job 2248 finished 1 steps successfully
[2017-06-01 19:42:17,997: INFO/job_scheduler] Job 2248: complete_job: Updating cache
[2017-06-01 19:42:17,997: INFO/job_scheduler] Job 2248 completing (errored=False, cancelled=False)
[2017-06-01 19:42:18,044: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:42:18,138: INFO/job_scheduler] Job 2249 finished 1 steps successfully
[2017-06-01 19:42:18,231: INFO/job_scheduler] Job 2249: complete_job: Updating cache
[2017-06-01 19:42:18,232: INFO/job_scheduler] Job 2249 completing (errored=False, cancelled=False)
[2017-06-01 19:42:18,285: INFO/job_scheduler] Job 2250 finished 1 steps successfully
[2017-06-01 19:42:18,299: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:42:18,318: INFO/job_scheduler] Job 2250: complete_job: Updating cache
[2017-06-01 19:42:18,319: INFO/job_scheduler] Job 2250 completing (errored=False, cancelled=False)
[2017-06-01 19:42:18,424: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:42:20,519: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:20,534: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:20,549: INFO/command_plan] add_jobs: created Job 2251 (Test for host connectivity)
[2017-06-01 19:42:20,652: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:42:20,657: INFO/job_scheduler] Job 2251: RunJobThread.run
[2017-06-01 19:42:20,710: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:20,716: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:20,739: INFO/command_plan] add_jobs: created Job 2252 (Test for host connectivity)
[2017-06-01 19:42:20,790: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:42:20,834: INFO/job_scheduler] Job 2252: RunJobThread.run
[2017-06-01 19:42:20,932: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:20,932: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:20,961: INFO/command_plan] add_jobs: created Job 2253 (Test for host connectivity)
[2017-06-01 19:42:21,034: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:42:21,059: INFO/job_scheduler] Job 2253: RunJobThread.run
[2017-06-01 19:42:21,135: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:21,135: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:21,175: INFO/command_plan] add_jobs: created Job 2254 (Test for host connectivity)
[2017-06-01 19:42:21,280: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:42:21,298: INFO/job_scheduler] Job 2254: RunJobThread.run
[2017-06-01 19:42:22,983: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:42:23,183: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:42:23,449: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:42:23,510: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:42:24,284: INFO/job_scheduler] Job 2251 finished 1 steps successfully
[2017-06-01 19:42:24,313: INFO/job_scheduler] Job 2251: complete_job: Updating cache
[2017-06-01 19:42:24,313: INFO/job_scheduler] Job 2251 completing (errored=False, cancelled=False)
[2017-06-01 19:42:24,346: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:42:24,565: INFO/job_scheduler] Job 2252 finished 1 steps successfully
[2017-06-01 19:42:24,604: INFO/job_scheduler] Job 2252: complete_job: Updating cache
[2017-06-01 19:42:24,605: INFO/job_scheduler] Job 2252 completing (errored=False, cancelled=False)
[2017-06-01 19:42:24,682: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:42:24,907: INFO/job_scheduler] Job 2253 finished 1 steps successfully
[2017-06-01 19:42:24,946: INFO/job_scheduler] Job 2253: complete_job: Updating cache
[2017-06-01 19:42:24,946: INFO/job_scheduler] Job 2253 completing (errored=False, cancelled=False)
[2017-06-01 19:42:24,996: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:42:25,103: INFO/job_scheduler] Job 2254 finished 1 steps successfully
[2017-06-01 19:42:25,121: INFO/job_scheduler] Job 2254: complete_job: Updating cache
[2017-06-01 19:42:25,121: INFO/job_scheduler] Job 2254 completing (errored=False, cancelled=False)
[2017-06-01 19:42:25,204: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:42:27,140: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:27,140: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:27,162: INFO/command_plan] add_jobs: created Job 2255 (Test for host connectivity)
[2017-06-01 19:42:27,198: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:42:27,219: INFO/job_scheduler] Job 2255: RunJobThread.run
[2017-06-01 19:42:27,304: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:27,309: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:27,337: INFO/command_plan] add_jobs: created Job 2256 (Test for host connectivity)
[2017-06-01 19:42:27,422: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:42:27,426: INFO/job_scheduler] Job 2256: RunJobThread.run
[2017-06-01 19:42:27,498: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:27,499: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:27,513: INFO/command_plan] add_jobs: created Job 2257 (Test for host connectivity)
[2017-06-01 19:42:27,608: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:42:27,645: INFO/job_scheduler] Job 2257: RunJobThread.run
[2017-06-01 19:42:27,714: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:27,715: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:27,735: INFO/command_plan] add_jobs: created Job 2258 (Test for host connectivity)
[2017-06-01 19:42:27,772: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:42:27,818: INFO/job_scheduler] Job 2258: RunJobThread.run
[2017-06-01 19:42:29,626: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:42:29,822: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:42:29,908: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:42:30,144: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:42:30,935: INFO/job_scheduler] Job 2255 finished 1 steps successfully
[2017-06-01 19:42:30,953: INFO/job_scheduler] Job 2255: complete_job: Updating cache
[2017-06-01 19:42:30,953: INFO/job_scheduler] Job 2255 completing (errored=False, cancelled=False)
[2017-06-01 19:42:31,015: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:42:31,247: INFO/job_scheduler] Job 2256 finished 1 steps successfully
[2017-06-01 19:42:31,305: INFO/job_scheduler] Job 2256: complete_job: Updating cache
[2017-06-01 19:42:31,306: INFO/job_scheduler] Job 2256 completing (errored=False, cancelled=False)
[2017-06-01 19:42:31,341: INFO/job_scheduler] Job 2257 finished 1 steps successfully
[2017-06-01 19:42:31,355: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:42:31,369: INFO/job_scheduler] Job 2257: complete_job: Updating cache
[2017-06-01 19:42:31,371: INFO/job_scheduler] Job 2257 completing (errored=False, cancelled=False)
[2017-06-01 19:42:31,503: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:42:31,576: INFO/job_scheduler] Job 2258 finished 1 steps successfully
[2017-06-01 19:42:31,599: INFO/job_scheduler] Job 2258: complete_job: Updating cache
[2017-06-01 19:42:31,599: INFO/job_scheduler] Job 2258 completing (errored=False, cancelled=False)
[2017-06-01 19:42:31,664: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:42:33,505: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:33,507: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:33,525: INFO/command_plan] add_jobs: created Job 2259 (Test for host connectivity)
[2017-06-01 19:42:33,639: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:33,640: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:33,721: INFO/command_plan] add_jobs: created Job 2260 (Test for host connectivity)
[2017-06-01 19:42:33,837: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:42:33,856: INFO/job_scheduler] Job 2259: RunJobThread.run
[2017-06-01 19:42:33,858: INFO/job_scheduler] Job 2260: RunJobThread.run
[2017-06-01 19:42:33,879: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:42:33,932: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:33,933: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:33,945: INFO/command_plan] add_jobs: created Job 2261 (Test for host connectivity)
[2017-06-01 19:42:33,996: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:42:34,054: INFO/job_scheduler] Job 2261: RunJobThread.run
[2017-06-01 19:42:34,116: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:34,116: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:34,136: INFO/command_plan] add_jobs: created Job 2262 (Test for host connectivity)
[2017-06-01 19:42:34,238: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:42:34,284: INFO/job_scheduler] Job 2262: RunJobThread.run
[2017-06-01 19:42:36,299: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:42:36,303: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:42:36,460: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:42:36,617: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:42:37,683: INFO/job_scheduler] Job 2260 finished 1 steps successfully
[2017-06-01 19:42:37,696: INFO/job_scheduler] Job 2259 finished 1 steps successfully
[2017-06-01 19:42:37,726: INFO/job_scheduler] Job 2260: complete_job: Updating cache
[2017-06-01 19:42:37,727: INFO/job_scheduler] Job 2260 completing (errored=False, cancelled=False)
[2017-06-01 19:42:37,803: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:42:37,827: INFO/job_scheduler] Job 2259: complete_job: Updating cache
[2017-06-01 19:42:37,827: INFO/job_scheduler] Job 2259 completing (errored=False, cancelled=False)
[2017-06-01 19:42:37,947: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:42:37,956: INFO/job_scheduler] Job 2261 finished 1 steps successfully
[2017-06-01 19:42:38,011: INFO/job_scheduler] Job 2261: complete_job: Updating cache
[2017-06-01 19:42:38,011: INFO/job_scheduler] Job 2261 completing (errored=False, cancelled=False)
[2017-06-01 19:42:38,016: INFO/job_scheduler] Job 2262 finished 1 steps successfully
[2017-06-01 19:42:38,109: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:42:38,128: INFO/job_scheduler] Job 2262: complete_job: Updating cache
[2017-06-01 19:42:38,128: INFO/job_scheduler] Job 2262 completing (errored=False, cancelled=False)
[2017-06-01 19:42:38,203: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:42:40,163: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:40,165: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:40,194: INFO/command_plan] add_jobs: created Job 2263 (Test for host connectivity)
[2017-06-01 19:42:40,214: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:42:40,274: INFO/job_scheduler] Job 2263: RunJobThread.run
[2017-06-01 19:42:40,326: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:40,327: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:40,337: INFO/command_plan] add_jobs: created Job 2264 (Test for host connectivity)
[2017-06-01 19:42:40,418: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:42:40,463: INFO/job_scheduler] Job 2264: RunJobThread.run
[2017-06-01 19:42:40,568: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:40,569: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:40,628: INFO/command_plan] add_jobs: created Job 2265 (Test for host connectivity)
[2017-06-01 19:42:40,667: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:42:40,721: INFO/job_scheduler] Job 2265: RunJobThread.run
[2017-06-01 19:42:40,814: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:40,815: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:40,851: INFO/command_plan] add_jobs: created Job 2266 (Test for host connectivity)
[2017-06-01 19:42:40,862: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:42:40,901: INFO/job_scheduler] Job 2266: RunJobThread.run
[2017-06-01 19:42:42,701: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:42:42,967: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:42:43,119: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:42:43,340: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:42:44,036: INFO/job_scheduler] Job 2263 finished 1 steps successfully
[2017-06-01 19:42:44,077: INFO/job_scheduler] Job 2263: complete_job: Updating cache
[2017-06-01 19:42:44,077: INFO/job_scheduler] Job 2263 completing (errored=False, cancelled=False)
[2017-06-01 19:42:44,174: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:42:44,339: INFO/job_scheduler] Job 2264 finished 1 steps successfully
[2017-06-01 19:42:44,391: INFO/job_scheduler] Job 2264: complete_job: Updating cache
[2017-06-01 19:42:44,392: INFO/job_scheduler] Job 2264 completing (errored=False, cancelled=False)
[2017-06-01 19:42:44,442: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:42:44,542: INFO/job_scheduler] Job 2265 finished 1 steps successfully
[2017-06-01 19:42:44,592: INFO/job_scheduler] Job 2265: complete_job: Updating cache
[2017-06-01 19:42:44,592: INFO/job_scheduler] Job 2265 completing (errored=False, cancelled=False)
[2017-06-01 19:42:44,651: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:42:44,714: INFO/job_scheduler] Job 2266 finished 1 steps successfully
[2017-06-01 19:42:44,742: INFO/job_scheduler] Job 2266: complete_job: Updating cache
[2017-06-01 19:42:44,742: INFO/job_scheduler] Job 2266 completing (errored=False, cancelled=False)
[2017-06-01 19:42:44,809: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:42:46,674: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:46,674: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:46,701: INFO/command_plan] add_jobs: created Job 2267 (Test for host connectivity)
[2017-06-01 19:42:46,800: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:42:46,820: INFO/job_scheduler] Job 2267: RunJobThread.run
[2017-06-01 19:42:46,875: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:46,877: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:46,892: INFO/command_plan] add_jobs: created Job 2268 (Test for host connectivity)
[2017-06-01 19:42:46,957: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:42:46,998: INFO/job_scheduler] Job 2268: RunJobThread.run
[2017-06-01 19:42:47,106: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:47,109: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:47,125: INFO/command_plan] add_jobs: created Job 2269 (Test for host connectivity)
[2017-06-01 19:42:47,235: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:42:47,281: INFO/job_scheduler] Job 2269: RunJobThread.run
[2017-06-01 19:42:47,350: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:47,352: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:47,382: INFO/command_plan] add_jobs: created Job 2270 (Test for host connectivity)
[2017-06-01 19:42:47,430: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:42:47,487: INFO/job_scheduler] Job 2270: RunJobThread.run
[2017-06-01 19:42:49,244: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:42:49,448: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:42:49,647: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:42:49,737: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:42:50,590: INFO/job_scheduler] Job 2267 finished 1 steps successfully
[2017-06-01 19:42:50,658: INFO/job_scheduler] Job 2267: complete_job: Updating cache
[2017-06-01 19:42:50,658: INFO/job_scheduler] Job 2267 completing (errored=False, cancelled=False)
[2017-06-01 19:42:50,701: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:42:50,950: INFO/job_scheduler] Job 2268 finished 1 steps successfully
[2017-06-01 19:42:50,985: INFO/job_scheduler] Job 2268: complete_job: Updating cache
[2017-06-01 19:42:50,986: INFO/job_scheduler] Job 2268 completing (errored=False, cancelled=False)
[2017-06-01 19:42:51,032: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:42:51,098: INFO/job_scheduler] Job 2269 finished 1 steps successfully
[2017-06-01 19:42:51,121: INFO/job_scheduler] Job 2270 finished 1 steps successfully
[2017-06-01 19:42:51,141: INFO/job_scheduler] Job 2269: complete_job: Updating cache
[2017-06-01 19:42:51,142: INFO/job_scheduler] Job 2269 completing (errored=False, cancelled=False)
[2017-06-01 19:42:51,260: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:42:51,274: INFO/job_scheduler] Job 2270: complete_job: Updating cache
[2017-06-01 19:42:51,276: INFO/job_scheduler] Job 2270 completing (errored=False, cancelled=False)
[2017-06-01 19:42:51,335: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:42:53,239: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:53,240: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:53,258: INFO/command_plan] add_jobs: created Job 2271 (Test for host connectivity)
[2017-06-01 19:42:53,371: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:42:53,378: INFO/job_scheduler] Job 2271: RunJobThread.run
[2017-06-01 19:42:53,417: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:53,418: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:53,444: INFO/command_plan] add_jobs: created Job 2272 (Test for host connectivity)
[2017-06-01 19:42:53,482: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:42:53,493: INFO/job_scheduler] Job 2272: RunJobThread.run
[2017-06-01 19:42:53,612: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:53,613: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:53,633: INFO/command_plan] add_jobs: created Job 2273 (Test for host connectivity)
[2017-06-01 19:42:53,719: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:42:53,761: INFO/job_scheduler] Job 2273: RunJobThread.run
[2017-06-01 19:42:53,861: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:53,862: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:53,884: INFO/command_plan] add_jobs: created Job 2274 (Test for host connectivity)
[2017-06-01 19:42:54,041: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:42:54,068: INFO/job_scheduler] Job 2274: RunJobThread.run
[2017-06-01 19:42:55,767: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:42:55,941: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:42:56,097: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:42:56,283: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:42:57,096: INFO/job_scheduler] Job 2271 finished 1 steps successfully
[2017-06-01 19:42:57,131: INFO/job_scheduler] Job 2271: complete_job: Updating cache
[2017-06-01 19:42:57,132: INFO/job_scheduler] Job 2271 completing (errored=False, cancelled=False)
[2017-06-01 19:42:57,230: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:42:57,304: INFO/job_scheduler] Job 2272 finished 1 steps successfully
[2017-06-01 19:42:57,367: INFO/job_scheduler] Job 2272: complete_job: Updating cache
[2017-06-01 19:42:57,370: INFO/job_scheduler] Job 2272 completing (errored=False, cancelled=False)
[2017-06-01 19:42:57,457: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:42:57,473: INFO/job_scheduler] Job 2273 finished 1 steps successfully
[2017-06-01 19:42:57,526: INFO/job_scheduler] Job 2273: complete_job: Updating cache
[2017-06-01 19:42:57,527: INFO/job_scheduler] Job 2273 completing (errored=False, cancelled=False)
[2017-06-01 19:42:57,602: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:42:57,778: INFO/job_scheduler] Job 2274 finished 1 steps successfully
[2017-06-01 19:42:57,822: INFO/job_scheduler] Job 2274: complete_job: Updating cache
[2017-06-01 19:42:57,823: INFO/job_scheduler] Job 2274 completing (errored=False, cancelled=False)
[2017-06-01 19:42:57,848: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:42:59,292: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:59,293: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:59,304: INFO/command_plan] add_jobs: created Job 2275 (Test for host connectivity)
[2017-06-01 19:42:59,318: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:42:59,322: INFO/job_scheduler] Job 2275: RunJobThread.run
[2017-06-01 19:42:59,447: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:59,449: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:59,580: INFO/command_plan] add_jobs: created Job 2276 (Test for host connectivity)
[2017-06-01 19:42:59,656: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:42:59,675: INFO/job_scheduler] Job 2276: RunJobThread.run
[2017-06-01 19:42:59,746: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:59,746: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:42:59,772: INFO/command_plan] add_jobs: created Job 2277 (Test for host connectivity)
[2017-06-01 19:42:59,897: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:42:59,897: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:00,036: INFO/command_plan] add_jobs: created Job 2278 (Test for host connectivity)
[2017-06-01 19:43:00,074: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:43:00,123: INFO/job_scheduler] Job 2277: RunJobThread.run
[2017-06-01 19:43:00,153: INFO/job_scheduler] Job 2278: RunJobThread.run
[2017-06-01 19:43:00,177: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:43:01,741: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:43:02,089: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:43:02,414: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:43:02,518: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:43:03,087: INFO/job_scheduler] Job 2275 finished 1 steps successfully
[2017-06-01 19:43:03,134: INFO/job_scheduler] Job 2275: complete_job: Updating cache
[2017-06-01 19:43:03,134: INFO/job_scheduler] Job 2275 completing (errored=False, cancelled=False)
[2017-06-01 19:43:03,174: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:43:03,472: INFO/job_scheduler] Job 2276 finished 1 steps successfully
[2017-06-01 19:43:03,536: INFO/job_scheduler] Job 2276: complete_job: Updating cache
[2017-06-01 19:43:03,536: INFO/job_scheduler] Job 2276 completing (errored=False, cancelled=False)
[2017-06-01 19:43:03,586: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:43:03,800: INFO/job_scheduler] Job 2278 finished 1 steps successfully
[2017-06-01 19:43:03,836: INFO/job_scheduler] Job 2278: complete_job: Updating cache
[2017-06-01 19:43:03,837: INFO/job_scheduler] Job 2278 completing (errored=False, cancelled=False)
[2017-06-01 19:43:03,911: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:43:04,004: INFO/job_scheduler] Job 2277 finished 1 steps successfully
[2017-06-01 19:43:04,043: INFO/job_scheduler] Job 2277: complete_job: Updating cache
[2017-06-01 19:43:04,043: INFO/job_scheduler] Job 2277 completing (errored=False, cancelled=False)
[2017-06-01 19:43:04,103: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:43:05,913: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:05,914: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:05,944: INFO/command_plan] add_jobs: created Job 2279 (Test for host connectivity)
[2017-06-01 19:43:05,982: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:43:05,989: INFO/job_scheduler] Job 2279: RunJobThread.run
[2017-06-01 19:43:06,161: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:06,161: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:06,193: INFO/command_plan] add_jobs: created Job 2280 (Test for host connectivity)
[2017-06-01 19:43:06,337: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:43:06,357: INFO/job_scheduler] Job 2280: RunJobThread.run
[2017-06-01 19:43:06,391: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:06,392: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:06,431: INFO/command_plan] add_jobs: created Job 2281 (Test for host connectivity)
[2017-06-01 19:43:06,472: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:43:06,507: INFO/job_scheduler] Job 2281: RunJobThread.run
[2017-06-01 19:43:06,621: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:06,622: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:06,666: INFO/command_plan] add_jobs: created Job 2282 (Test for host connectivity)
[2017-06-01 19:43:06,806: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:43:06,812: INFO/job_scheduler] Job 2282: RunJobThread.run
[2017-06-01 19:43:08,587: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:43:08,888: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:43:08,920: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:43:09,145: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:43:09,977: INFO/job_scheduler] Job 2279 finished 1 steps successfully
[2017-06-01 19:43:10,017: INFO/job_scheduler] Job 2279: complete_job: Updating cache
[2017-06-01 19:43:10,017: INFO/job_scheduler] Job 2279 completing (errored=False, cancelled=False)
[2017-06-01 19:43:10,066: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:43:10,224: INFO/job_scheduler] Job 2281 finished 1 steps successfully
[2017-06-01 19:43:10,279: INFO/job_scheduler] Job 2281: complete_job: Updating cache
[2017-06-01 19:43:10,279: INFO/job_scheduler] Job 2281 completing (errored=False, cancelled=False)
[2017-06-01 19:43:10,333: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:43:10,409: INFO/job_scheduler] Job 2280 finished 1 steps successfully
[2017-06-01 19:43:10,475: INFO/job_scheduler] Job 2280: complete_job: Updating cache
[2017-06-01 19:43:10,475: INFO/job_scheduler] Job 2280 completing (errored=False, cancelled=False)
[2017-06-01 19:43:10,542: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:43:10,605: INFO/job_scheduler] Job 2282 finished 1 steps successfully
[2017-06-01 19:43:10,627: INFO/job_scheduler] Job 2282: complete_job: Updating cache
[2017-06-01 19:43:10,627: INFO/job_scheduler] Job 2282 completing (errored=False, cancelled=False)
[2017-06-01 19:43:10,704: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:43:12,576: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:12,579: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:12,591: INFO/command_plan] add_jobs: created Job 2283 (Test for host connectivity)
[2017-06-01 19:43:12,625: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:43:12,636: INFO/job_scheduler] Job 2283: RunJobThread.run
[2017-06-01 19:43:12,728: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:12,728: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:12,783: INFO/command_plan] add_jobs: created Job 2284 (Test for host connectivity)
[2017-06-01 19:43:12,911: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:43:12,920: INFO/job_scheduler] Job 2284: RunJobThread.run
[2017-06-01 19:43:12,982: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:12,983: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:13,005: INFO/command_plan] add_jobs: created Job 2285 (Test for host connectivity)
[2017-06-01 19:43:13,045: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:43:13,094: INFO/job_scheduler] Job 2285: RunJobThread.run
[2017-06-01 19:43:13,143: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:13,144: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:13,176: INFO/command_plan] add_jobs: created Job 2286 (Test for host connectivity)
[2017-06-01 19:43:13,208: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:43:13,253: INFO/job_scheduler] Job 2286: RunJobThread.run
[2017-06-01 19:43:15,045: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:43:15,203: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:43:15,374: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:43:15,564: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:43:16,372: INFO/job_scheduler] Job 2283 finished 1 steps successfully
[2017-06-01 19:43:16,432: INFO/job_scheduler] Job 2283: complete_job: Updating cache
[2017-06-01 19:43:16,432: INFO/job_scheduler] Job 2283 completing (errored=False, cancelled=False)
[2017-06-01 19:43:16,501: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:43:16,583: INFO/job_scheduler] Job 2284 finished 1 steps successfully
[2017-06-01 19:43:16,644: INFO/job_scheduler] Job 2284: complete_job: Updating cache
[2017-06-01 19:43:16,650: INFO/job_scheduler] Job 2284 completing (errored=False, cancelled=False)
[2017-06-01 19:43:16,765: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:43:16,830: INFO/job_scheduler] Job 2285 finished 1 steps successfully
[2017-06-01 19:43:16,914: INFO/job_scheduler] Job 2285: complete_job: Updating cache
[2017-06-01 19:43:16,914: INFO/job_scheduler] Job 2285 completing (errored=False, cancelled=False)
[2017-06-01 19:43:16,937: INFO/job_scheduler] Job 2286 finished 1 steps successfully
[2017-06-01 19:43:16,966: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:43:16,999: INFO/job_scheduler] Job 2286: complete_job: Updating cache
[2017-06-01 19:43:16,999: INFO/job_scheduler] Job 2286 completing (errored=False, cancelled=False)
[2017-06-01 19:43:17,084: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:43:19,085: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:19,085: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:19,106: INFO/command_plan] add_jobs: created Job 2287 (Test for host connectivity)
[2017-06-01 19:43:19,139: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:43:19,162: INFO/job_scheduler] Job 2287: RunJobThread.run
[2017-06-01 19:43:19,224: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:19,224: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:19,242: INFO/command_plan] add_jobs: created Job 2288 (Test for host connectivity)
[2017-06-01 19:43:19,307: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:43:19,315: INFO/job_scheduler] Job 2288: RunJobThread.run
[2017-06-01 19:43:19,435: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:19,436: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:19,458: INFO/command_plan] add_jobs: created Job 2289 (Test for host connectivity)
[2017-06-01 19:43:19,542: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:43:19,565: INFO/job_scheduler] Job 2289: RunJobThread.run
[2017-06-01 19:43:19,630: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:19,631: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:19,658: INFO/command_plan] add_jobs: created Job 2290 (Test for host connectivity)
[2017-06-01 19:43:19,769: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:43:19,782: INFO/job_scheduler] Job 2290: RunJobThread.run
[2017-06-01 19:43:21,625: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:43:21,740: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:43:21,846: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:43:22,002: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:43:22,950: INFO/job_scheduler] Job 2287 finished 1 steps successfully
[2017-06-01 19:43:22,981: INFO/job_scheduler] Job 2287: complete_job: Updating cache
[2017-06-01 19:43:22,981: INFO/job_scheduler] Job 2287 completing (errored=False, cancelled=False)
[2017-06-01 19:43:23,039: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:43:23,072: INFO/job_scheduler] Job 2288 finished 1 steps successfully
[2017-06-01 19:43:23,105: INFO/job_scheduler] Job 2288: complete_job: Updating cache
[2017-06-01 19:43:23,105: INFO/job_scheduler] Job 2288 completing (errored=False, cancelled=False)
[2017-06-01 19:43:23,148: INFO/job_scheduler] Job 2289 finished 1 steps successfully
[2017-06-01 19:43:23,149: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:43:23,164: INFO/job_scheduler] Job 2289: complete_job: Updating cache
[2017-06-01 19:43:23,164: INFO/job_scheduler] Job 2289 completing (errored=False, cancelled=False)
[2017-06-01 19:43:23,271: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:43:23,322: INFO/job_scheduler] Job 2290 finished 1 steps successfully
[2017-06-01 19:43:23,368: INFO/job_scheduler] Job 2290: complete_job: Updating cache
[2017-06-01 19:43:23,368: INFO/job_scheduler] Job 2290 completing (errored=False, cancelled=False)
[2017-06-01 19:43:23,431: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:43:25,899: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:25,900: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:25,921: INFO/command_plan] add_jobs: created Job 2291 (Test for host connectivity)
[2017-06-01 19:43:25,951: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:43:25,971: INFO/job_scheduler] Job 2291: RunJobThread.run
[2017-06-01 19:43:26,035: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:26,036: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:26,048: INFO/command_plan] add_jobs: created Job 2292 (Test for host connectivity)
[2017-06-01 19:43:26,096: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:43:26,112: INFO/job_scheduler] Job 2292: RunJobThread.run
[2017-06-01 19:43:26,221: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:26,221: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:26,255: INFO/command_plan] add_jobs: created Job 2293 (Test for host connectivity)
[2017-06-01 19:43:26,322: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:43:26,340: INFO/job_scheduler] Job 2293: RunJobThread.run
[2017-06-01 19:43:26,415: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:26,419: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:26,431: INFO/command_plan] add_jobs: created Job 2294 (Test for host connectivity)
[2017-06-01 19:43:26,476: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:43:26,489: INFO/job_scheduler] Job 2294: RunJobThread.run
[2017-06-01 19:43:28,382: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:43:28,566: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:43:28,696: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:43:28,799: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:43:29,672: INFO/job_scheduler] Job 2291 finished 1 steps successfully
[2017-06-01 19:43:29,746: INFO/job_scheduler] Job 2291: complete_job: Updating cache
[2017-06-01 19:43:29,746: INFO/job_scheduler] Job 2291 completing (errored=False, cancelled=False)
[2017-06-01 19:43:29,797: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:43:29,988: INFO/job_scheduler] Job 2292 finished 1 steps successfully
[2017-06-01 19:43:30,042: INFO/job_scheduler] Job 2292: complete_job: Updating cache
[2017-06-01 19:43:30,042: INFO/job_scheduler] Job 2292 completing (errored=False, cancelled=False)
[2017-06-01 19:43:30,053: INFO/job_scheduler] Job 2293 finished 1 steps successfully
[2017-06-01 19:43:30,246: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:43:30,287: INFO/job_scheduler] Job 2294 finished 1 steps successfully
[2017-06-01 19:43:30,297: INFO/job_scheduler] Job 2293: complete_job: Updating cache
[2017-06-01 19:43:30,297: INFO/job_scheduler] Job 2293 completing (errored=False, cancelled=False)
[2017-06-01 19:43:30,382: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:43:30,415: INFO/job_scheduler] Job 2294: complete_job: Updating cache
[2017-06-01 19:43:30,415: INFO/job_scheduler] Job 2294 completing (errored=False, cancelled=False)
[2017-06-01 19:43:30,497: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:43:32,458: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:32,459: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:32,473: INFO/command_plan] add_jobs: created Job 2295 (Test for host connectivity)
[2017-06-01 19:43:32,504: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:43:32,506: INFO/job_scheduler] Job 2295: RunJobThread.run
[2017-06-01 19:43:32,611: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:32,611: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:32,641: INFO/command_plan] add_jobs: created Job 2296 (Test for host connectivity)
[2017-06-01 19:43:32,715: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:43:32,727: INFO/job_scheduler] Job 2296: RunJobThread.run
[2017-06-01 19:43:32,779: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:32,780: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:32,795: INFO/command_plan] add_jobs: created Job 2297 (Test for host connectivity)
[2017-06-01 19:43:32,879: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:43:32,900: INFO/job_scheduler] Job 2297: RunJobThread.run
[2017-06-01 19:43:32,973: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:32,974: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:33,012: INFO/command_plan] add_jobs: created Job 2298 (Test for host connectivity)
[2017-06-01 19:43:33,031: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:43:33,095: INFO/job_scheduler] Job 2298: RunJobThread.run
[2017-06-01 19:43:34,927: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:43:35,278: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:43:35,375: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:43:35,391: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:43:36,271: INFO/job_scheduler] Job 2295 finished 1 steps successfully
[2017-06-01 19:43:36,303: INFO/job_scheduler] Job 2295: complete_job: Updating cache
[2017-06-01 19:43:36,303: INFO/job_scheduler] Job 2295 completing (errored=False, cancelled=False)
[2017-06-01 19:43:36,372: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:43:36,696: INFO/job_scheduler] Job 2296 finished 1 steps successfully
[2017-06-01 19:43:36,769: INFO/job_scheduler] Job 2296: complete_job: Updating cache
[2017-06-01 19:43:36,770: INFO/job_scheduler] Job 2296 completing (errored=False, cancelled=False)
[2017-06-01 19:43:36,867: INFO/job_scheduler] Job 2297 finished 1 steps successfully
[2017-06-01 19:43:36,870: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:43:36,877: INFO/job_scheduler] Job 2298 finished 1 steps successfully
[2017-06-01 19:43:36,888: INFO/job_scheduler] Job 2297: complete_job: Updating cache
[2017-06-01 19:43:36,888: INFO/job_scheduler] Job 2297 completing (errored=False, cancelled=False)
[2017-06-01 19:43:36,993: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:43:37,044: INFO/job_scheduler] Job 2298: complete_job: Updating cache
[2017-06-01 19:43:37,044: INFO/job_scheduler] Job 2298 completing (errored=False, cancelled=False)
[2017-06-01 19:43:37,087: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:43:38,919: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:38,919: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:38,927: INFO/command_plan] add_jobs: created Job 2299 (Test for host connectivity)
[2017-06-01 19:43:38,960: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:43:39,013: INFO/job_scheduler] Job 2299: RunJobThread.run
[2017-06-01 19:43:39,170: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:39,172: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:39,187: INFO/command_plan] add_jobs: created Job 2300 (Test for host connectivity)
[2017-06-01 19:43:39,232: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:43:39,274: INFO/job_scheduler] Job 2300: RunJobThread.run
[2017-06-01 19:43:39,301: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:39,301: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:39,345: INFO/command_plan] add_jobs: created Job 2301 (Test for host connectivity)
[2017-06-01 19:43:39,398: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:43:39,408: INFO/job_scheduler] Job 2301: RunJobThread.run
[2017-06-01 19:43:39,494: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:39,494: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:39,527: INFO/command_plan] add_jobs: created Job 2302 (Test for host connectivity)
[2017-06-01 19:43:39,588: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:43:39,668: INFO/job_scheduler] Job 2302: RunJobThread.run
[2017-06-01 19:43:41,394: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:43:41,588: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:43:41,776: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:43:41,915: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:43:42,696: INFO/job_scheduler] Job 2299 finished 1 steps successfully
[2017-06-01 19:43:42,726: INFO/job_scheduler] Job 2299: complete_job: Updating cache
[2017-06-01 19:43:42,726: INFO/job_scheduler] Job 2299 completing (errored=False, cancelled=False)
[2017-06-01 19:43:42,795: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:43:42,930: INFO/job_scheduler] Job 2300 finished 1 steps successfully
[2017-06-01 19:43:42,994: INFO/job_scheduler] Job 2300: complete_job: Updating cache
[2017-06-01 19:43:42,994: INFO/job_scheduler] Job 2300 completing (errored=False, cancelled=False)
[2017-06-01 19:43:43,118: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:43:43,219: INFO/job_scheduler] Job 2301 finished 1 steps successfully
[2017-06-01 19:43:43,252: INFO/job_scheduler] Job 2301: complete_job: Updating cache
[2017-06-01 19:43:43,255: INFO/job_scheduler] Job 2301 completing (errored=False, cancelled=False)
[2017-06-01 19:43:43,335: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:43:43,346: INFO/job_scheduler] Job 2302 finished 1 steps successfully
[2017-06-01 19:43:43,388: INFO/job_scheduler] Job 2302: complete_job: Updating cache
[2017-06-01 19:43:43,388: INFO/job_scheduler] Job 2302 completing (errored=False, cancelled=False)
[2017-06-01 19:43:43,437: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:43:45,500: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:45,501: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:45,526: INFO/command_plan] add_jobs: created Job 2303 (Test for host connectivity)
[2017-06-01 19:43:45,584: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:43:45,590: INFO/job_scheduler] Job 2303: RunJobThread.run
[2017-06-01 19:43:45,723: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:45,724: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:45,743: INFO/command_plan] add_jobs: created Job 2304 (Test for host connectivity)
[2017-06-01 19:43:45,837: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:43:45,850: INFO/job_scheduler] Job 2304: RunJobThread.run
[2017-06-01 19:43:45,927: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:45,927: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:45,969: INFO/command_plan] add_jobs: created Job 2305 (Test for host connectivity)
[2017-06-01 19:43:46,084: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:43:46,100: INFO/job_scheduler] Job 2305: RunJobThread.run
[2017-06-01 19:43:46,206: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:46,207: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:46,234: INFO/command_plan] add_jobs: created Job 2306 (Test for host connectivity)
[2017-06-01 19:43:46,275: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:43:46,337: INFO/job_scheduler] Job 2306: RunJobThread.run
[2017-06-01 19:43:48,001: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:43:48,200: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:43:48,403: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:43:48,645: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:43:49,290: INFO/job_scheduler] Job 2303 finished 1 steps successfully
[2017-06-01 19:43:49,334: INFO/job_scheduler] Job 2303: complete_job: Updating cache
[2017-06-01 19:43:49,334: INFO/job_scheduler] Job 2303 completing (errored=False, cancelled=False)
[2017-06-01 19:43:49,441: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:43:49,566: INFO/job_scheduler] Job 2304 finished 1 steps successfully
[2017-06-01 19:43:49,620: INFO/job_scheduler] Job 2304: complete_job: Updating cache
[2017-06-01 19:43:49,620: INFO/job_scheduler] Job 2304 completing (errored=False, cancelled=False)
[2017-06-01 19:43:49,681: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:43:49,804: INFO/job_scheduler] Job 2305 finished 1 steps successfully
[2017-06-01 19:43:49,858: INFO/job_scheduler] Job 2305: complete_job: Updating cache
[2017-06-01 19:43:49,858: INFO/job_scheduler] Job 2305 completing (errored=False, cancelled=False)
[2017-06-01 19:43:49,942: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:43:50,074: INFO/job_scheduler] Job 2306 finished 1 steps successfully
[2017-06-01 19:43:50,100: INFO/job_scheduler] Job 2306: complete_job: Updating cache
[2017-06-01 19:43:50,100: INFO/job_scheduler] Job 2306 completing (errored=False, cancelled=False)
[2017-06-01 19:43:50,125: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:43:52,101: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:52,104: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:52,116: INFO/command_plan] add_jobs: created Job 2307 (Test for host connectivity)
[2017-06-01 19:43:52,136: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:43:52,201: INFO/job_scheduler] Job 2307: RunJobThread.run
[2017-06-01 19:43:52,261: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:52,262: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:52,284: INFO/command_plan] add_jobs: created Job 2308 (Test for host connectivity)
[2017-06-01 19:43:52,303: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:43:52,350: INFO/job_scheduler] Job 2308: RunJobThread.run
[2017-06-01 19:43:52,413: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:52,414: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:52,442: INFO/command_plan] add_jobs: created Job 2309 (Test for host connectivity)
[2017-06-01 19:43:52,532: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:43:52,594: INFO/job_scheduler] Job 2309: RunJobThread.run
[2017-06-01 19:43:52,633: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:52,634: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:52,675: INFO/command_plan] add_jobs: created Job 2310 (Test for host connectivity)
[2017-06-01 19:43:52,705: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:43:52,770: INFO/job_scheduler] Job 2310: RunJobThread.run
[2017-06-01 19:43:54,535: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:43:54,701: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:43:54,842: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:43:55,042: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:43:55,875: INFO/job_scheduler] Job 2307 finished 1 steps successfully
[2017-06-01 19:43:55,924: INFO/job_scheduler] Job 2307: complete_job: Updating cache
[2017-06-01 19:43:55,924: INFO/job_scheduler] Job 2307 completing (errored=False, cancelled=False)
[2017-06-01 19:43:55,969: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:43:56,058: INFO/job_scheduler] Job 2308 finished 1 steps successfully
[2017-06-01 19:43:56,101: INFO/job_scheduler] Job 2308: complete_job: Updating cache
[2017-06-01 19:43:56,109: INFO/job_scheduler] Job 2308 completing (errored=False, cancelled=False)
[2017-06-01 19:43:56,209: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:43:56,270: INFO/job_scheduler] Job 2309 finished 1 steps successfully
[2017-06-01 19:43:56,333: INFO/job_scheduler] Job 2309: complete_job: Updating cache
[2017-06-01 19:43:56,334: INFO/job_scheduler] Job 2309 completing (errored=False, cancelled=False)
[2017-06-01 19:43:56,400: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:43:56,535: INFO/job_scheduler] Job 2310 finished 1 steps successfully
[2017-06-01 19:43:56,657: INFO/job_scheduler] Job 2310: complete_job: Updating cache
[2017-06-01 19:43:56,657: INFO/job_scheduler] Job 2310 completing (errored=False, cancelled=False)
[2017-06-01 19:43:56,708: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:43:58,577: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:58,577: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:58,654: INFO/command_plan] add_jobs: created Job 2311 (Test for host connectivity)
[2017-06-01 19:43:58,724: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:43:58,768: INFO/job_scheduler] Job 2311: RunJobThread.run
[2017-06-01 19:43:58,821: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:58,822: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:58,868: INFO/command_plan] add_jobs: created Job 2312 (Test for host connectivity)
[2017-06-01 19:43:58,953: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:43:58,997: INFO/job_scheduler] Job 2312: RunJobThread.run
[2017-06-01 19:43:59,188: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:59,188: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:59,241: INFO/command_plan] add_jobs: created Job 2313 (Test for host connectivity)
[2017-06-01 19:43:59,290: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:43:59,351: INFO/job_scheduler] Job 2313: RunJobThread.run
[2017-06-01 19:43:59,422: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:43:59,422: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:43:59,436: INFO/command_plan] add_jobs: created Job 2314 (Test for host connectivity)
[2017-06-01 19:43:59,468: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:43:59,525: INFO/job_scheduler] Job 2314: RunJobThread.run
[2017-06-01 19:44:01,217: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:44:01,576: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:44:01,763: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:44:01,819: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:44:02,569: INFO/job_scheduler] Job 2311 finished 1 steps successfully
[2017-06-01 19:44:02,604: INFO/job_scheduler] Job 2311: complete_job: Updating cache
[2017-06-01 19:44:02,605: INFO/job_scheduler] Job 2311 completing (errored=False, cancelled=False)
[2017-06-01 19:44:02,637: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:44:02,955: INFO/job_scheduler] Job 2312 finished 1 steps successfully
[2017-06-01 19:44:03,002: INFO/job_scheduler] Job 2312: complete_job: Updating cache
[2017-06-01 19:44:03,002: INFO/job_scheduler] Job 2312 completing (errored=False, cancelled=False)
[2017-06-01 19:44:03,079: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:44:03,200: INFO/job_scheduler] Job 2314 finished 1 steps successfully
[2017-06-01 19:44:03,214: INFO/job_scheduler] Job 2314: complete_job: Updating cache
[2017-06-01 19:44:03,214: INFO/job_scheduler] Job 2314 completing (errored=False, cancelled=False)
[2017-06-01 19:44:03,248: INFO/job_scheduler] Job 2313 finished 1 steps successfully
[2017-06-01 19:44:03,270: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:44:03,288: INFO/job_scheduler] Job 2313: complete_job: Updating cache
[2017-06-01 19:44:03,288: INFO/job_scheduler] Job 2313 completing (errored=False, cancelled=False)
[2017-06-01 19:44:03,372: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:44:05,299: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:05,299: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:05,318: INFO/command_plan] add_jobs: created Job 2315 (Test for host connectivity)
[2017-06-01 19:44:05,355: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:44:05,373: INFO/job_scheduler] Job 2315: RunJobThread.run
[2017-06-01 19:44:05,459: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:05,459: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:05,507: INFO/command_plan] add_jobs: created Job 2316 (Test for host connectivity)
[2017-06-01 19:44:05,540: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:44:05,552: INFO/job_scheduler] Job 2316: RunJobThread.run
[2017-06-01 19:44:05,673: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:05,675: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:05,694: INFO/command_plan] add_jobs: created Job 2317 (Test for host connectivity)
[2017-06-01 19:44:05,826: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:44:05,832: INFO/job_scheduler] Job 2317: RunJobThread.run
[2017-06-01 19:44:05,888: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:05,889: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:05,903: INFO/command_plan] add_jobs: created Job 2318 (Test for host connectivity)
[2017-06-01 19:44:05,956: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:44:05,976: INFO/job_scheduler] Job 2318: RunJobThread.run
[2017-06-01 19:44:07,876: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:44:07,965: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:44:08,049: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:44:08,282: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:44:09,182: INFO/job_scheduler] Job 2315 finished 1 steps successfully
[2017-06-01 19:44:09,233: INFO/job_scheduler] Job 2315: complete_job: Updating cache
[2017-06-01 19:44:09,234: INFO/job_scheduler] Job 2315 completing (errored=False, cancelled=False)
[2017-06-01 19:44:09,282: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:44:09,423: INFO/job_scheduler] Job 2316 finished 1 steps successfully
[2017-06-01 19:44:09,450: INFO/job_scheduler] Job 2316: complete_job: Updating cache
[2017-06-01 19:44:09,452: INFO/job_scheduler] Job 2316 completing (errored=False, cancelled=False)
[2017-06-01 19:44:09,467: INFO/job_scheduler] Job 2317 finished 1 steps successfully
[2017-06-01 19:44:09,514: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:44:09,595: INFO/job_scheduler] Job 2317: complete_job: Updating cache
[2017-06-01 19:44:09,595: INFO/job_scheduler] Job 2317 completing (errored=False, cancelled=False)
[2017-06-01 19:44:09,642: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:44:09,664: INFO/job_scheduler] Job 2318 finished 1 steps successfully
[2017-06-01 19:44:09,709: INFO/job_scheduler] Job 2318: complete_job: Updating cache
[2017-06-01 19:44:09,710: INFO/job_scheduler] Job 2318 completing (errored=False, cancelled=False)
[2017-06-01 19:44:09,758: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:44:11,796: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:11,798: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:11,811: INFO/command_plan] add_jobs: created Job 2319 (Test for host connectivity)
[2017-06-01 19:44:11,900: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:44:11,912: INFO/job_scheduler] Job 2319: RunJobThread.run
[2017-06-01 19:44:11,980: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:11,985: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:12,000: INFO/command_plan] add_jobs: created Job 2320 (Test for host connectivity)
[2017-06-01 19:44:12,110: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:44:12,125: INFO/job_scheduler] Job 2320: RunJobThread.run
[2017-06-01 19:44:12,198: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:12,199: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:12,233: INFO/command_plan] add_jobs: created Job 2321 (Test for host connectivity)
[2017-06-01 19:44:12,362: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:44:12,368: INFO/job_scheduler] Job 2321: RunJobThread.run
[2017-06-01 19:44:12,428: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:12,429: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:12,459: INFO/command_plan] add_jobs: created Job 2322 (Test for host connectivity)
[2017-06-01 19:44:12,521: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:44:12,576: INFO/job_scheduler] Job 2322: RunJobThread.run
[2017-06-01 19:44:14,272: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:44:14,476: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:44:14,689: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:44:14,773: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:44:15,594: INFO/job_scheduler] Job 2319 finished 1 steps successfully
[2017-06-01 19:44:15,607: INFO/job_scheduler] Job 2319: complete_job: Updating cache
[2017-06-01 19:44:15,608: INFO/job_scheduler] Job 2319 completing (errored=False, cancelled=False)
[2017-06-01 19:44:15,659: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:44:15,790: INFO/job_scheduler] Job 2320 finished 1 steps successfully
[2017-06-01 19:44:15,857: INFO/job_scheduler] Job 2320: complete_job: Updating cache
[2017-06-01 19:44:15,858: INFO/job_scheduler] Job 2320 completing (errored=False, cancelled=False)
[2017-06-01 19:44:15,992: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:44:16,165: INFO/job_scheduler] Job 2322 finished 1 steps successfully
[2017-06-01 19:44:16,229: INFO/job_scheduler] Job 2321 finished 1 steps successfully
[2017-06-01 19:44:16,240: INFO/job_scheduler] Job 2322: complete_job: Updating cache
[2017-06-01 19:44:16,241: INFO/job_scheduler] Job 2322 completing (errored=False, cancelled=False)
[2017-06-01 19:44:16,275: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:44:16,292: INFO/job_scheduler] Job 2321: complete_job: Updating cache
[2017-06-01 19:44:16,292: INFO/job_scheduler] Job 2321 completing (errored=False, cancelled=False)
[2017-06-01 19:44:16,367: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:44:18,382: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:18,383: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:18,400: INFO/command_plan] add_jobs: created Job 2323 (Test for host connectivity)
[2017-06-01 19:44:18,455: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:44:18,496: INFO/job_scheduler] Job 2323: RunJobThread.run
[2017-06-01 19:44:18,536: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:18,543: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:18,559: INFO/command_plan] add_jobs: created Job 2324 (Test for host connectivity)
[2017-06-01 19:44:18,664: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:44:18,707: INFO/job_scheduler] Job 2324: RunJobThread.run
[2017-06-01 19:44:18,794: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:18,796: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:18,857: INFO/command_plan] add_jobs: created Job 2325 (Test for host connectivity)
[2017-06-01 19:44:18,909: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:44:18,963: INFO/job_scheduler] Job 2325: RunJobThread.run
[2017-06-01 19:44:19,003: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:19,008: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:19,041: INFO/command_plan] add_jobs: created Job 2326 (Test for host connectivity)
[2017-06-01 19:44:19,098: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:44:19,131: INFO/job_scheduler] Job 2326: RunJobThread.run
[2017-06-01 19:44:20,891: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:44:21,147: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:44:21,365: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:44:21,374: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:44:22,215: INFO/job_scheduler] Job 2323 finished 1 steps successfully
[2017-06-01 19:44:22,256: INFO/job_scheduler] Job 2323: complete_job: Updating cache
[2017-06-01 19:44:22,256: INFO/job_scheduler] Job 2323 completing (errored=False, cancelled=False)
[2017-06-01 19:44:22,290: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:44:22,467: INFO/job_scheduler] Job 2324 finished 1 steps successfully
[2017-06-01 19:44:22,525: INFO/job_scheduler] Job 2324: complete_job: Updating cache
[2017-06-01 19:44:22,527: INFO/job_scheduler] Job 2324 completing (errored=False, cancelled=False)
[2017-06-01 19:44:22,634: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:44:22,776: INFO/job_scheduler] Job 2325 finished 1 steps successfully
[2017-06-01 19:44:22,840: INFO/job_scheduler] Job 2325: complete_job: Updating cache
[2017-06-01 19:44:22,840: INFO/job_scheduler] Job 2325 completing (errored=False, cancelled=False)
[2017-06-01 19:44:22,904: INFO/job_scheduler] Job 2326 finished 1 steps successfully
[2017-06-01 19:44:22,905: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:44:22,922: INFO/job_scheduler] Job 2326: complete_job: Updating cache
[2017-06-01 19:44:22,923: INFO/job_scheduler] Job 2326 completing (errored=False, cancelled=False)
[2017-06-01 19:44:22,998: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:44:24,934: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:24,936: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:24,956: INFO/command_plan] add_jobs: created Job 2327 (Test for host connectivity)
[2017-06-01 19:44:25,010: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:44:25,017: INFO/job_scheduler] Job 2327: RunJobThread.run
[2017-06-01 19:44:25,150: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:25,158: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:25,174: INFO/command_plan] add_jobs: created Job 2328 (Test for host connectivity)
[2017-06-01 19:44:25,265: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:44:25,291: INFO/job_scheduler] Job 2328: RunJobThread.run
[2017-06-01 19:44:25,377: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:25,378: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:25,398: INFO/command_plan] add_jobs: created Job 2329 (Test for host connectivity)
[2017-06-01 19:44:25,516: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:44:25,530: INFO/job_scheduler] Job 2329: RunJobThread.run
[2017-06-01 19:44:25,576: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:25,577: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:25,590: INFO/command_plan] add_jobs: created Job 2330 (Test for host connectivity)
[2017-06-01 19:44:25,652: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:44:25,673: INFO/job_scheduler] Job 2330: RunJobThread.run
[2017-06-01 19:44:27,399: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:44:27,652: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:44:27,943: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:44:27,947: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:44:28,736: INFO/job_scheduler] Job 2327 finished 1 steps successfully
[2017-06-01 19:44:28,796: INFO/job_scheduler] Job 2327: complete_job: Updating cache
[2017-06-01 19:44:28,797: INFO/job_scheduler] Job 2327 completing (errored=False, cancelled=False)
[2017-06-01 19:44:28,838: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:44:29,040: INFO/job_scheduler] Job 2328 finished 1 steps successfully
[2017-06-01 19:44:29,107: INFO/job_scheduler] Job 2328: complete_job: Updating cache
[2017-06-01 19:44:29,107: INFO/job_scheduler] Job 2328 completing (errored=False, cancelled=False)
[2017-06-01 19:44:29,163: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:44:29,322: INFO/job_scheduler] Job 2330 finished 1 steps successfully
[2017-06-01 19:44:29,363: INFO/job_scheduler] Job 2330: complete_job: Updating cache
[2017-06-01 19:44:29,364: INFO/job_scheduler] Job 2330 completing (errored=False, cancelled=False)
[2017-06-01 19:44:29,392: INFO/job_scheduler] Job 2329 finished 1 steps successfully
[2017-06-01 19:44:29,415: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:44:29,430: INFO/job_scheduler] Job 2329: complete_job: Updating cache
[2017-06-01 19:44:29,431: INFO/job_scheduler] Job 2329 completing (errored=False, cancelled=False)
[2017-06-01 19:44:29,498: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:44:32,304: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:32,305: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:32,312: INFO/command_plan] add_jobs: created Job 2331 (Test for host connectivity)
[2017-06-01 19:44:32,377: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:44:32,407: INFO/job_scheduler] Job 2331: RunJobThread.run
[2017-06-01 19:44:32,467: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:32,468: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:32,488: INFO/command_plan] add_jobs: created Job 2332 (Test for host connectivity)
[2017-06-01 19:44:32,577: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:44:32,607: INFO/job_scheduler] Job 2332: RunJobThread.run
[2017-06-01 19:44:32,653: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:32,654: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:32,663: INFO/command_plan] add_jobs: created Job 2333 (Test for host connectivity)
[2017-06-01 19:44:32,786: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:32,787: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:32,809: INFO/command_plan] add_jobs: created Job 2334 (Test for host connectivity)
[2017-06-01 19:44:32,835: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:44:32,853: INFO/job_scheduler] Job 2333: RunJobThread.run
[2017-06-01 19:44:32,857: INFO/job_scheduler] Job 2334: RunJobThread.run
[2017-06-01 19:44:32,873: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:44:34,881: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:44:34,971: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:44:35,280: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:44:35,281: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:44:36,296: INFO/job_scheduler] Job 2331 finished 1 steps successfully
[2017-06-01 19:44:36,311: INFO/job_scheduler] Job 2331: complete_job: Updating cache
[2017-06-01 19:44:36,312: INFO/job_scheduler] Job 2331 completing (errored=False, cancelled=False)
[2017-06-01 19:44:36,378: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:44:36,384: INFO/job_scheduler] Job 2332 finished 1 steps successfully
[2017-06-01 19:44:36,434: INFO/job_scheduler] Job 2332: complete_job: Updating cache
[2017-06-01 19:44:36,434: INFO/job_scheduler] Job 2332 completing (errored=False, cancelled=False)
[2017-06-01 19:44:36,586: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:44:36,638: INFO/job_scheduler] Job 2334 finished 1 steps successfully
[2017-06-01 19:44:36,653: INFO/job_scheduler] Job 2334: complete_job: Updating cache
[2017-06-01 19:44:36,654: INFO/job_scheduler] Job 2334 completing (errored=False, cancelled=False)
[2017-06-01 19:44:36,687: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:44:36,853: INFO/job_scheduler] Job 2333 finished 1 steps successfully
[2017-06-01 19:44:36,895: INFO/job_scheduler] Job 2333: complete_job: Updating cache
[2017-06-01 19:44:36,895: INFO/job_scheduler] Job 2333 completing (errored=False, cancelled=False)
[2017-06-01 19:44:36,920: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:44:38,905: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:38,905: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:38,920: INFO/command_plan] add_jobs: created Job 2335 (Test for host connectivity)
[2017-06-01 19:44:38,987: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:44:39,016: INFO/job_scheduler] Job 2335: RunJobThread.run
[2017-06-01 19:44:39,082: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:39,083: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:39,122: INFO/command_plan] add_jobs: created Job 2336 (Test for host connectivity)
[2017-06-01 19:44:39,238: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:39,239: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:39,294: INFO/command_plan] add_jobs: created Job 2337 (Test for host connectivity)
[2017-06-01 19:44:39,331: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:44:39,356: INFO/job_scheduler] Job 2336: RunJobThread.run
[2017-06-01 19:44:39,378: INFO/job_scheduler] Job 2337: RunJobThread.run
[2017-06-01 19:44:39,439: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:39,440: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:39,463: INFO/command_plan] add_jobs: created Job 2338 (Test for host connectivity)
[2017-06-01 19:44:39,514: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:44:39,567: INFO/job_scheduler] Job 2338: RunJobThread.run
[2017-06-01 19:44:39,661: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:44:41,387: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:44:41,804: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:44:41,834: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:44:41,886: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:44:42,719: INFO/job_scheduler] Job 2335 finished 1 steps successfully
[2017-06-01 19:44:42,777: INFO/job_scheduler] Job 2335: complete_job: Updating cache
[2017-06-01 19:44:42,778: INFO/job_scheduler] Job 2335 completing (errored=False, cancelled=False)
[2017-06-01 19:44:42,805: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:44:43,124: INFO/job_scheduler] Job 2336 finished 1 steps successfully
[2017-06-01 19:44:43,177: INFO/job_scheduler] Job 2336: complete_job: Updating cache
[2017-06-01 19:44:43,178: INFO/job_scheduler] Job 2336 completing (errored=False, cancelled=False)
[2017-06-01 19:44:43,253: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:44:43,268: INFO/job_scheduler] Job 2337 finished 1 steps successfully
[2017-06-01 19:44:43,317: INFO/job_scheduler] Job 2337: complete_job: Updating cache
[2017-06-01 19:44:43,337: INFO/job_scheduler] Job 2337 completing (errored=False, cancelled=False)
[2017-06-01 19:44:43,363: INFO/job_scheduler] Job 2338 finished 1 steps successfully
[2017-06-01 19:44:43,422: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:44:43,449: INFO/job_scheduler] Job 2338: complete_job: Updating cache
[2017-06-01 19:44:43,450: INFO/job_scheduler] Job 2338 completing (errored=False, cancelled=False)
[2017-06-01 19:44:43,561: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:44:45,479: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:45,481: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:45,501: INFO/command_plan] add_jobs: created Job 2339 (Test for host connectivity)
[2017-06-01 19:44:45,545: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:44:45,564: INFO/job_scheduler] Job 2339: RunJobThread.run
[2017-06-01 19:44:45,645: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:45,645: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:45,662: INFO/command_plan] add_jobs: created Job 2340 (Test for host connectivity)
[2017-06-01 19:44:45,699: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:44:45,724: INFO/job_scheduler] Job 2340: RunJobThread.run
[2017-06-01 19:44:45,875: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:45,876: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:45,885: INFO/command_plan] add_jobs: created Job 2341 (Test for host connectivity)
[2017-06-01 19:44:45,953: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:44:45,969: INFO/job_scheduler] Job 2341: RunJobThread.run
[2017-06-01 19:44:46,058: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:46,065: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:46,118: INFO/command_plan] add_jobs: created Job 2342 (Test for host connectivity)
[2017-06-01 19:44:46,154: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:44:46,228: INFO/job_scheduler] Job 2342: RunJobThread.run
[2017-06-01 19:44:47,885: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:44:48,072: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:44:48,398: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:44:48,485: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:44:49,188: INFO/job_scheduler] Job 2339 finished 1 steps successfully
[2017-06-01 19:44:49,243: INFO/job_scheduler] Job 2339: complete_job: Updating cache
[2017-06-01 19:44:49,244: INFO/job_scheduler] Job 2339 completing (errored=False, cancelled=False)
[2017-06-01 19:44:49,326: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:44:49,453: INFO/job_scheduler] Job 2340 finished 1 steps successfully
[2017-06-01 19:44:49,526: INFO/job_scheduler] Job 2340: complete_job: Updating cache
[2017-06-01 19:44:49,526: INFO/job_scheduler] Job 2340 completing (errored=False, cancelled=False)
[2017-06-01 19:44:49,602: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:44:49,821: INFO/job_scheduler] Job 2341 finished 1 steps successfully
[2017-06-01 19:44:49,869: INFO/job_scheduler] Job 2341: complete_job: Updating cache
[2017-06-01 19:44:49,869: INFO/job_scheduler] Job 2341 completing (errored=False, cancelled=False)
[2017-06-01 19:44:49,892: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:44:49,965: INFO/job_scheduler] Job 2342 finished 1 steps successfully
[2017-06-01 19:44:49,994: INFO/job_scheduler] Job 2342: complete_job: Updating cache
[2017-06-01 19:44:49,996: INFO/job_scheduler] Job 2342 completing (errored=False, cancelled=False)
[2017-06-01 19:44:50,052: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:44:52,102: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:52,104: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:52,117: INFO/command_plan] add_jobs: created Job 2343 (Test for host connectivity)
[2017-06-01 19:44:52,181: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:52,182: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:52,213: INFO/command_plan] add_jobs: created Job 2344 (Test for host connectivity)
[2017-06-01 19:44:52,230: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:44:52,267: INFO/job_scheduler] Job 2344: RunJobThread.run
[2017-06-01 19:44:52,292: INFO/job_scheduler] Job 2343: RunJobThread.run
[2017-06-01 19:44:52,301: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:44:52,379: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:52,380: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:52,401: INFO/command_plan] add_jobs: created Job 2345 (Test for host connectivity)
[2017-06-01 19:44:52,502: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:44:52,553: INFO/job_scheduler] Job 2345: RunJobThread.run
[2017-06-01 19:44:52,630: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:52,631: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:52,651: INFO/command_plan] add_jobs: created Job 2346 (Test for host connectivity)
[2017-06-01 19:44:52,767: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:44:52,772: INFO/job_scheduler] Job 2346: RunJobThread.run
[2017-06-01 19:44:54,620: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:44:54,676: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:44:54,919: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:44:54,999: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:44:55,951: INFO/job_scheduler] Job 2344 finished 1 steps successfully
[2017-06-01 19:44:55,993: INFO/job_scheduler] Job 2343 finished 1 steps successfully
[2017-06-01 19:44:55,999: INFO/job_scheduler] Job 2344: complete_job: Updating cache
[2017-06-01 19:44:55,999: INFO/job_scheduler] Job 2344 completing (errored=False, cancelled=False)
[2017-06-01 19:44:56,077: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:44:56,092: INFO/job_scheduler] Job 2343: complete_job: Updating cache
[2017-06-01 19:44:56,092: INFO/job_scheduler] Job 2343 completing (errored=False, cancelled=False)
[2017-06-01 19:44:56,203: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:44:56,268: INFO/job_scheduler] Job 2345 finished 1 steps successfully
[2017-06-01 19:44:56,283: INFO/job_scheduler] Job 2345: complete_job: Updating cache
[2017-06-01 19:44:56,283: INFO/job_scheduler] Job 2345 completing (errored=False, cancelled=False)
[2017-06-01 19:44:56,349: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:44:56,376: INFO/job_scheduler] Job 2346 finished 1 steps successfully
[2017-06-01 19:44:56,415: INFO/job_scheduler] Job 2346: complete_job: Updating cache
[2017-06-01 19:44:56,415: INFO/job_scheduler] Job 2346 completing (errored=False, cancelled=False)
[2017-06-01 19:44:56,516: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:44:58,668: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:58,668: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:58,824: INFO/command_plan] add_jobs: created Job 2347 (Test for host connectivity)
[2017-06-01 19:44:58,876: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:58,876: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:59,087: INFO/command_plan] add_jobs: created Job 2348 (Test for host connectivity)
[2017-06-01 19:44:59,111: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:44:59,165: INFO/job_scheduler] Job 2347: RunJobThread.run
[2017-06-01 19:44:59,180: INFO/job_scheduler] Job 2348: RunJobThread.run
[2017-06-01 19:44:59,211: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:44:59,295: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:59,295: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:59,350: INFO/command_plan] add_jobs: created Job 2349 (Test for host connectivity)
[2017-06-01 19:44:59,520: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:44:59,550: INFO/job_scheduler] Job 2349: RunJobThread.run
[2017-06-01 19:44:59,666: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:44:59,666: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:44:59,674: INFO/command_plan] add_jobs: created Job 2350 (Test for host connectivity)
[2017-06-01 19:44:59,691: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:44:59,735: INFO/job_scheduler] Job 2350: RunJobThread.run
[2017-06-01 19:45:01,769: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:45:01,822: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:45:01,994: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:45:02,177: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:45:03,172: INFO/job_scheduler] Job 2348 finished 1 steps successfully
[2017-06-01 19:45:03,220: INFO/job_scheduler] Job 2348: complete_job: Updating cache
[2017-06-01 19:45:03,220: INFO/job_scheduler] Job 2348 completing (errored=False, cancelled=False)
[2017-06-01 19:45:03,229: INFO/job_scheduler] Job 2347 finished 1 steps successfully
[2017-06-01 19:45:03,261: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:45:03,278: INFO/job_scheduler] Job 2347: complete_job: Updating cache
[2017-06-01 19:45:03,279: INFO/job_scheduler] Job 2347 completing (errored=False, cancelled=False)
[2017-06-01 19:45:03,378: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:45:03,380: INFO/job_scheduler] Job 2349 finished 1 steps successfully
[2017-06-01 19:45:03,424: INFO/job_scheduler] Job 2349: complete_job: Updating cache
[2017-06-01 19:45:03,425: INFO/job_scheduler] Job 2349 completing (errored=False, cancelled=False)
[2017-06-01 19:45:03,463: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:45:03,647: INFO/job_scheduler] Job 2350 finished 1 steps successfully
[2017-06-01 19:45:03,678: INFO/job_scheduler] Job 2350: complete_job: Updating cache
[2017-06-01 19:45:03,678: INFO/job_scheduler] Job 2350 completing (errored=False, cancelled=False)
[2017-06-01 19:45:03,747: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:45:05,672: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:05,673: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:05,687: INFO/command_plan] add_jobs: created Job 2351 (Test for host connectivity)
[2017-06-01 19:45:05,728: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:45:05,762: INFO/job_scheduler] Job 2351: RunJobThread.run
[2017-06-01 19:45:05,884: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:05,885: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:05,894: INFO/command_plan] add_jobs: created Job 2352 (Test for host connectivity)
[2017-06-01 19:45:05,948: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:45:05,987: INFO/job_scheduler] Job 2352: RunJobThread.run
[2017-06-01 19:45:06,040: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:06,041: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:06,061: INFO/command_plan] add_jobs: created Job 2353 (Test for host connectivity)
[2017-06-01 19:45:06,183: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:45:06,208: INFO/job_scheduler] Job 2353: RunJobThread.run
[2017-06-01 19:45:06,237: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:06,238: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:06,252: INFO/command_plan] add_jobs: created Job 2354 (Test for host connectivity)
[2017-06-01 19:45:06,332: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:45:06,365: INFO/job_scheduler] Job 2354: RunJobThread.run
[2017-06-01 19:45:08,167: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:45:08,286: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:45:08,483: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:45:08,564: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:45:09,494: INFO/job_scheduler] Job 2351 finished 1 steps successfully
[2017-06-01 19:45:09,545: INFO/job_scheduler] Job 2351: complete_job: Updating cache
[2017-06-01 19:45:09,545: INFO/job_scheduler] Job 2351 completing (errored=False, cancelled=False)
[2017-06-01 19:45:09,616: INFO/job_scheduler] Job 2352 finished 1 steps successfully
[2017-06-01 19:45:09,652: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:45:09,694: INFO/job_scheduler] Job 2352: complete_job: Updating cache
[2017-06-01 19:45:09,694: INFO/job_scheduler] Job 2352 completing (errored=False, cancelled=False)
[2017-06-01 19:45:09,794: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:45:09,931: INFO/job_scheduler] Job 2353 finished 1 steps successfully
[2017-06-01 19:45:09,969: INFO/job_scheduler] Job 2353: complete_job: Updating cache
[2017-06-01 19:45:09,969: INFO/job_scheduler] Job 2353 completing (errored=False, cancelled=False)
[2017-06-01 19:45:10,038: INFO/job_scheduler] Job 2354 finished 1 steps successfully
[2017-06-01 19:45:10,057: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:45:10,072: INFO/job_scheduler] Job 2354: complete_job: Updating cache
[2017-06-01 19:45:10,072: INFO/job_scheduler] Job 2354 completing (errored=False, cancelled=False)
[2017-06-01 19:45:10,131: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:45:12,306: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:12,307: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:12,318: INFO/command_plan] add_jobs: created Job 2355 (Test for host connectivity)
[2017-06-01 19:45:12,362: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:45:12,383: INFO/job_scheduler] Job 2355: RunJobThread.run
[2017-06-01 19:45:12,473: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:12,479: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:12,494: INFO/command_plan] add_jobs: created Job 2356 (Test for host connectivity)
[2017-06-01 19:45:12,570: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:45:12,595: INFO/job_scheduler] Job 2356: RunJobThread.run
[2017-06-01 19:45:12,672: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:12,673: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:12,694: INFO/command_plan] add_jobs: created Job 2357 (Test for host connectivity)
[2017-06-01 19:45:12,810: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:45:12,835: INFO/job_scheduler] Job 2357: RunJobThread.run
[2017-06-01 19:45:12,902: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:12,903: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:12,933: INFO/command_plan] add_jobs: created Job 2358 (Test for host connectivity)
[2017-06-01 19:45:13,000: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:45:13,013: INFO/job_scheduler] Job 2358: RunJobThread.run
[2017-06-01 19:45:14,877: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:45:15,126: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:45:15,266: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:45:15,299: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:45:16,196: INFO/job_scheduler] Job 2355 finished 1 steps successfully
[2017-06-01 19:45:16,350: INFO/job_scheduler] Job 2355: complete_job: Updating cache
[2017-06-01 19:45:16,351: INFO/job_scheduler] Job 2355 completing (errored=False, cancelled=False)
[2017-06-01 19:45:16,470: INFO/job_scheduler] Job 2356 finished 1 steps successfully
[2017-06-01 19:45:16,476: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:45:16,507: INFO/job_scheduler] Job 2356: complete_job: Updating cache
[2017-06-01 19:45:16,507: INFO/job_scheduler] Job 2356 completing (errored=False, cancelled=False)
[2017-06-01 19:45:16,650: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:45:16,651: INFO/job_scheduler] Job 2358 finished 1 steps successfully
[2017-06-01 19:45:16,674: INFO/job_scheduler] Job 2358: complete_job: Updating cache
[2017-06-01 19:45:16,675: INFO/job_scheduler] Job 2358 completing (errored=False, cancelled=False)
[2017-06-01 19:45:16,749: INFO/job_scheduler] Job 2357 finished 1 steps successfully
[2017-06-01 19:45:16,767: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:45:16,775: INFO/job_scheduler] Job 2357: complete_job: Updating cache
[2017-06-01 19:45:16,775: INFO/job_scheduler] Job 2357 completing (errored=False, cancelled=False)
[2017-06-01 19:45:16,849: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:45:18,951: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:18,953: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:18,967: INFO/command_plan] add_jobs: created Job 2359 (Test for host connectivity)
[2017-06-01 19:45:19,092: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:19,094: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:19,109: INFO/command_plan] add_jobs: created Job 2360 (Test for host connectivity)
[2017-06-01 19:45:19,116: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:45:19,167: INFO/job_scheduler] Job 2360: RunJobThread.run
[2017-06-01 19:45:19,196: INFO/job_scheduler] Job 2359: RunJobThread.run
[2017-06-01 19:45:19,294: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:19,294: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:19,352: INFO/command_plan] add_jobs: created Job 2361 (Test for host connectivity)
[2017-06-01 19:45:19,362: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:45:19,368: INFO/job_scheduler] Job 2361: RunJobThread.run
[2017-06-01 19:45:19,468: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:45:19,547: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:19,548: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:19,584: INFO/command_plan] add_jobs: created Job 2362 (Test for host connectivity)
[2017-06-01 19:45:19,639: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:45:19,711: INFO/job_scheduler] Job 2362: RunJobThread.run
[2017-06-01 19:45:21,587: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:45:21,688: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:45:21,837: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:45:21,931: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:45:22,964: INFO/job_scheduler] Job 2360 finished 1 steps successfully
[2017-06-01 19:45:23,030: INFO/job_scheduler] Job 2360: complete_job: Updating cache
[2017-06-01 19:45:23,030: INFO/job_scheduler] Job 2360 completing (errored=False, cancelled=False)
[2017-06-01 19:45:23,036: INFO/job_scheduler] Job 2359 finished 1 steps successfully
[2017-06-01 19:45:23,082: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:45:23,096: INFO/job_scheduler] Job 2359: complete_job: Updating cache
[2017-06-01 19:45:23,096: INFO/job_scheduler] Job 2359 completing (errored=False, cancelled=False)
[2017-06-01 19:45:23,206: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:45:23,299: INFO/job_scheduler] Job 2361 finished 1 steps successfully
[2017-06-01 19:45:23,309: INFO/job_scheduler] Job 2362 finished 1 steps successfully
[2017-06-01 19:45:23,358: INFO/job_scheduler] Job 2361: complete_job: Updating cache
[2017-06-01 19:45:23,359: INFO/job_scheduler] Job 2361 completing (errored=False, cancelled=False)
[2017-06-01 19:45:23,413: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:45:23,438: INFO/job_scheduler] Job 2362: complete_job: Updating cache
[2017-06-01 19:45:23,445: INFO/job_scheduler] Job 2362 completing (errored=False, cancelled=False)
[2017-06-01 19:45:23,565: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:45:25,587: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:25,589: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:25,623: INFO/command_plan] add_jobs: created Job 2363 (Test for host connectivity)
[2017-06-01 19:45:25,779: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:25,780: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:25,796: INFO/command_plan] add_jobs: created Job 2364 (Test for host connectivity)
[2017-06-01 19:45:25,821: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:45:25,837: INFO/job_scheduler] Job 2363: RunJobThread.run
[2017-06-01 19:45:25,864: INFO/job_scheduler] Job 2364: RunJobThread.run
[2017-06-01 19:45:25,880: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:45:25,988: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:25,990: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:26,030: INFO/command_plan] add_jobs: created Job 2365 (Test for host connectivity)
[2017-06-01 19:45:26,108: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:45:26,130: INFO/job_scheduler] Job 2365: RunJobThread.run
[2017-06-01 19:45:26,262: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:26,263: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:26,284: INFO/command_plan] add_jobs: created Job 2366 (Test for host connectivity)
[2017-06-01 19:45:26,364: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:45:26,368: INFO/job_scheduler] Job 2366: RunJobThread.run
[2017-06-01 19:45:28,257: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:45:28,300: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:45:28,555: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:45:28,796: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:45:29,675: INFO/job_scheduler] Job 2363 finished 1 steps successfully
[2017-06-01 19:45:29,751: INFO/job_scheduler] Job 2364 finished 1 steps successfully
[2017-06-01 19:45:29,837: INFO/job_scheduler] Job 2363: complete_job: Updating cache
[2017-06-01 19:45:29,837: INFO/job_scheduler] Job 2363 completing (errored=False, cancelled=False)
[2017-06-01 19:45:29,849: INFO/job_scheduler] Job 2365 finished 1 steps successfully
[2017-06-01 19:45:29,888: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:45:29,898: INFO/job_scheduler] Job 2364: complete_job: Updating cache
[2017-06-01 19:45:29,899: INFO/job_scheduler] Job 2364 completing (errored=False, cancelled=False)
[2017-06-01 19:45:29,998: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:45:30,011: INFO/job_scheduler] Job 2365: complete_job: Updating cache
[2017-06-01 19:45:30,011: INFO/job_scheduler] Job 2365 completing (errored=False, cancelled=False)
[2017-06-01 19:45:30,168: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:45:30,199: INFO/job_scheduler] Job 2366 finished 1 steps successfully
[2017-06-01 19:45:30,304: INFO/job_scheduler] Job 2366: complete_job: Updating cache
[2017-06-01 19:45:30,304: INFO/job_scheduler] Job 2366 completing (errored=False, cancelled=False)
[2017-06-01 19:45:30,365: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:45:32,301: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:32,302: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:32,311: INFO/command_plan] add_jobs: created Job 2367 (Test for host connectivity)
[2017-06-01 19:45:32,373: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:32,373: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:32,412: INFO/command_plan] add_jobs: created Job 2368 (Test for host connectivity)
[2017-06-01 19:45:32,433: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:45:32,462: INFO/job_scheduler] Job 2368: RunJobThread.run
[2017-06-01 19:45:32,463: INFO/job_scheduler] Job 2367: RunJobThread.run
[2017-06-01 19:45:32,470: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:45:32,538: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:32,538: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:32,573: INFO/command_plan] add_jobs: created Job 2369 (Test for host connectivity)
[2017-06-01 19:45:32,639: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:45:32,713: INFO/job_scheduler] Job 2369: RunJobThread.run
[2017-06-01 19:45:32,790: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:32,792: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:32,806: INFO/command_plan] add_jobs: created Job 2370 (Test for host connectivity)
[2017-06-01 19:45:32,883: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:45:32,910: INFO/job_scheduler] Job 2370: RunJobThread.run
[2017-06-01 19:45:34,872: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:45:34,888: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:45:35,009: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:45:35,110: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:45:36,248: INFO/job_scheduler] Job 2367 finished 1 steps successfully
[2017-06-01 19:45:36,282: INFO/job_scheduler] Job 2368 finished 1 steps successfully
[2017-06-01 19:45:36,285: INFO/job_scheduler] Job 2367: complete_job: Updating cache
[2017-06-01 19:45:36,285: INFO/job_scheduler] Job 2367 completing (errored=False, cancelled=False)
[2017-06-01 19:45:36,310: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:45:36,333: INFO/job_scheduler] Job 2368: complete_job: Updating cache
[2017-06-01 19:45:36,333: INFO/job_scheduler] Job 2368 completing (errored=False, cancelled=False)
[2017-06-01 19:45:36,396: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:45:36,432: INFO/job_scheduler] Job 2369 finished 1 steps successfully
[2017-06-01 19:45:36,487: INFO/job_scheduler] Job 2369: complete_job: Updating cache
[2017-06-01 19:45:36,487: INFO/job_scheduler] Job 2369 completing (errored=False, cancelled=False)
[2017-06-01 19:45:36,504: INFO/job_scheduler] Job 2370 finished 1 steps successfully
[2017-06-01 19:45:36,593: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:45:36,624: INFO/job_scheduler] Job 2370: complete_job: Updating cache
[2017-06-01 19:45:36,625: INFO/job_scheduler] Job 2370 completing (errored=False, cancelled=False)
[2017-06-01 19:45:36,711: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:45:38,870: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:38,872: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:38,898: INFO/command_plan] add_jobs: created Job 2371 (Test for host connectivity)
[2017-06-01 19:45:38,929: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:45:38,951: INFO/job_scheduler] Job 2371: RunJobThread.run
[2017-06-01 19:45:39,047: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:39,047: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:39,077: INFO/command_plan] add_jobs: created Job 2372 (Test for host connectivity)
[2017-06-01 19:45:39,165: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:45:39,170: INFO/job_scheduler] Job 2372: RunJobThread.run
[2017-06-01 19:45:39,226: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:39,245: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:39,270: INFO/command_plan] add_jobs: created Job 2373 (Test for host connectivity)
[2017-06-01 19:45:39,395: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:39,395: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:39,405: INFO/command_plan] add_jobs: created Job 2374 (Test for host connectivity)
[2017-06-01 19:45:39,440: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:45:39,478: INFO/job_scheduler] Job 2373: RunJobThread.run
[2017-06-01 19:45:39,508: INFO/job_scheduler] Job 2374: RunJobThread.run
[2017-06-01 19:45:39,529: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:45:41,380: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:45:41,523: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:45:41,773: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:45:41,928: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:45:42,723: INFO/job_scheduler] Job 2371 finished 1 steps successfully
[2017-06-01 19:45:42,795: INFO/job_scheduler] Job 2371: complete_job: Updating cache
[2017-06-01 19:45:42,796: INFO/job_scheduler] Job 2371 completing (errored=False, cancelled=False)
[2017-06-01 19:45:42,844: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:45:42,928: INFO/job_scheduler] Job 2372 finished 1 steps successfully
[2017-06-01 19:45:42,978: INFO/job_scheduler] Job 2372: complete_job: Updating cache
[2017-06-01 19:45:42,978: INFO/job_scheduler] Job 2372 completing (errored=False, cancelled=False)
[2017-06-01 19:45:43,037: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:45:43,135: INFO/job_scheduler] Job 2374 finished 1 steps successfully
[2017-06-01 19:45:43,184: INFO/job_scheduler] Job 2374: complete_job: Updating cache
[2017-06-01 19:45:43,185: INFO/job_scheduler] Job 2374 completing (errored=False, cancelled=False)
[2017-06-01 19:45:43,243: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:45:43,449: INFO/job_scheduler] Job 2373 finished 1 steps successfully
[2017-06-01 19:45:43,459: INFO/job_scheduler] Job 2373: complete_job: Updating cache
[2017-06-01 19:45:43,459: INFO/job_scheduler] Job 2373 completing (errored=False, cancelled=False)
[2017-06-01 19:45:43,510: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:45:45,409: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:45,411: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:45,430: INFO/command_plan] add_jobs: created Job 2375 (Test for host connectivity)
[2017-06-01 19:45:45,509: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:45,509: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:45,536: INFO/command_plan] add_jobs: created Job 2376 (Test for host connectivity)
[2017-06-01 19:45:45,581: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:45:45,629: INFO/job_scheduler] Job 2376: RunJobThread.run
[2017-06-01 19:45:45,658: INFO/job_scheduler] Job 2375: RunJobThread.run
[2017-06-01 19:45:45,730: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:45,731: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:45,750: INFO/command_plan] add_jobs: created Job 2377 (Test for host connectivity)
[2017-06-01 19:45:45,854: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:45:45,900: INFO/job_scheduler] Job 2377: RunJobThread.run
[2017-06-01 19:45:46,059: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:46,059: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:46,075: INFO/command_plan] add_jobs: created Job 2378 (Test for host connectivity)
[2017-06-01 19:45:46,101: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:45:46,158: INFO/job_scheduler] Job 2378: RunJobThread.run
[2017-06-01 19:45:46,196: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:45:48,174: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:45:48,229: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:45:48,455: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:45:48,540: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:45:49,489: INFO/job_scheduler] Job 2375 finished 1 steps successfully
[2017-06-01 19:45:49,541: INFO/job_scheduler] Job 2375: complete_job: Updating cache
[2017-06-01 19:45:49,541: INFO/job_scheduler] Job 2375 completing (errored=False, cancelled=False)
[2017-06-01 19:45:49,614: INFO/job_scheduler] Job 2376 finished 1 steps successfully
[2017-06-01 19:45:49,659: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:45:49,690: INFO/job_scheduler] Job 2376: complete_job: Updating cache
[2017-06-01 19:45:49,690: INFO/job_scheduler] Job 2376 completing (errored=False, cancelled=False)
[2017-06-01 19:45:49,783: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:45:49,954: INFO/job_scheduler] Job 2377 finished 1 steps successfully
[2017-06-01 19:45:50,016: INFO/job_scheduler] Job 2377: complete_job: Updating cache
[2017-06-01 19:45:50,016: INFO/job_scheduler] Job 2377 completing (errored=False, cancelled=False)
[2017-06-01 19:45:50,018: INFO/job_scheduler] Job 2378 finished 1 steps successfully
[2017-06-01 19:45:50,042: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:45:50,075: INFO/job_scheduler] Job 2378: complete_job: Updating cache
[2017-06-01 19:45:50,075: INFO/job_scheduler] Job 2378 completing (errored=False, cancelled=False)
[2017-06-01 19:45:50,241: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:45:52,123: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:52,126: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:52,149: INFO/command_plan] add_jobs: created Job 2379 (Test for host connectivity)
[2017-06-01 19:45:52,257: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:45:52,276: INFO/job_scheduler] Job 2379: RunJobThread.run
[2017-06-01 19:45:52,431: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:52,432: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:52,449: INFO/command_plan] add_jobs: created Job 2380 (Test for host connectivity)
[2017-06-01 19:45:52,512: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:45:52,566: INFO/job_scheduler] Job 2380: RunJobThread.run
[2017-06-01 19:45:52,617: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:52,618: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:52,649: INFO/command_plan] add_jobs: created Job 2381 (Test for host connectivity)
[2017-06-01 19:45:52,668: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:45:52,713: INFO/job_scheduler] Job 2381: RunJobThread.run
[2017-06-01 19:45:52,917: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:52,918: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:52,942: INFO/command_plan] add_jobs: created Job 2382 (Test for host connectivity)
[2017-06-01 19:45:52,965: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:45:53,001: INFO/job_scheduler] Job 2382: RunJobThread.run
[2017-06-01 19:45:54,670: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:45:54,906: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:45:55,269: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:45:55,382: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:45:56,033: INFO/job_scheduler] Job 2379 finished 1 steps successfully
[2017-06-01 19:45:56,056: INFO/job_scheduler] Job 2379: complete_job: Updating cache
[2017-06-01 19:45:56,057: INFO/job_scheduler] Job 2379 completing (errored=False, cancelled=False)
[2017-06-01 19:45:56,124: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:45:56,290: INFO/job_scheduler] Job 2380 finished 1 steps successfully
[2017-06-01 19:45:56,352: INFO/job_scheduler] Job 2380: complete_job: Updating cache
[2017-06-01 19:45:56,369: INFO/job_scheduler] Job 2380 completing (errored=False, cancelled=False)
[2017-06-01 19:45:56,470: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:45:56,736: INFO/job_scheduler] Job 2381 finished 1 steps successfully
[2017-06-01 19:45:56,765: INFO/job_scheduler] Job 2381: complete_job: Updating cache
[2017-06-01 19:45:56,765: INFO/job_scheduler] Job 2381 completing (errored=False, cancelled=False)
[2017-06-01 19:45:56,767: INFO/job_scheduler] Job 2382 finished 1 steps successfully
[2017-06-01 19:45:56,914: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:45:56,928: INFO/job_scheduler] Job 2382: complete_job: Updating cache
[2017-06-01 19:45:56,928: INFO/job_scheduler] Job 2382 completing (errored=False, cancelled=False)
[2017-06-01 19:45:57,108: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:45:57,803: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:57,805: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:57,814: INFO/command_plan] add_jobs: created Job 2383 (Test for host connectivity)
[2017-06-01 19:45:57,842: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:45:57,882: INFO/job_scheduler] Job 2383: RunJobThread.run
[2017-06-01 19:45:57,943: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:57,947: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:57,974: INFO/command_plan] add_jobs: created Job 2384 (Test for host connectivity)
[2017-06-01 19:45:58,008: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:45:58,058: INFO/job_scheduler] Job 2384: RunJobThread.run
[2017-06-01 19:45:58,132: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:58,132: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:58,140: INFO/command_plan] add_jobs: created Job 2385 (Test for host connectivity)
[2017-06-01 19:45:58,230: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:45:58,243: INFO/job_scheduler] Job 2385: RunJobThread.run
[2017-06-01 19:45:58,322: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:45:58,323: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:45:58,348: INFO/command_plan] add_jobs: created Job 2386 (Test for host connectivity)
[2017-06-01 19:45:58,403: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:45:58,439: INFO/job_scheduler] Job 2386: RunJobThread.run
[2017-06-01 19:46:00,191: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:46:00,376: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:46:00,569: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:46:00,650: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:46:01,515: INFO/job_scheduler] Job 2383 finished 1 steps successfully
[2017-06-01 19:46:01,522: INFO/job_scheduler] Job 2383: complete_job: Updating cache
[2017-06-01 19:46:01,522: INFO/job_scheduler] Job 2383 completing (errored=False, cancelled=False)
[2017-06-01 19:46:01,547: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:46:01,710: INFO/job_scheduler] Job 2384 finished 1 steps successfully
[2017-06-01 19:46:01,773: INFO/job_scheduler] Job 2384: complete_job: Updating cache
[2017-06-01 19:46:01,774: INFO/job_scheduler] Job 2384 completing (errored=False, cancelled=False)
[2017-06-01 19:46:01,841: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:46:01,904: INFO/job_scheduler] Job 2385 finished 1 steps successfully
[2017-06-01 19:46:01,990: INFO/job_scheduler] Job 2385: complete_job: Updating cache
[2017-06-01 19:46:01,996: INFO/job_scheduler] Job 2385 completing (errored=False, cancelled=False)
[2017-06-01 19:46:02,097: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:46:02,229: INFO/job_scheduler] Job 2386 finished 1 steps successfully
[2017-06-01 19:46:02,255: INFO/job_scheduler] Job 2386: complete_job: Updating cache
[2017-06-01 19:46:02,265: INFO/job_scheduler] Job 2386 completing (errored=False, cancelled=False)
[2017-06-01 19:46:02,444: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:46:04,383: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:04,385: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:04,397: INFO/command_plan] add_jobs: created Job 2387 (Test for host connectivity)
[2017-06-01 19:46:04,446: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:46:04,482: INFO/job_scheduler] Job 2387: RunJobThread.run
[2017-06-01 19:46:04,550: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:04,551: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:04,565: INFO/command_plan] add_jobs: created Job 2388 (Test for host connectivity)
[2017-06-01 19:46:04,698: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:46:04,708: INFO/job_scheduler] Job 2388: RunJobThread.run
[2017-06-01 19:46:04,774: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:04,774: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:04,804: INFO/command_plan] add_jobs: created Job 2389 (Test for host connectivity)
[2017-06-01 19:46:04,844: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:46:04,884: INFO/job_scheduler] Job 2389: RunJobThread.run
[2017-06-01 19:46:04,959: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:04,959: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:04,972: INFO/command_plan] add_jobs: created Job 2390 (Test for host connectivity)
[2017-06-01 19:46:05,050: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:46:05,053: INFO/job_scheduler] Job 2390: RunJobThread.run
[2017-06-01 19:46:06,925: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:46:07,085: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:46:07,172: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:46:07,392: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:46:08,280: INFO/job_scheduler] Job 2387 finished 1 steps successfully
[2017-06-01 19:46:08,321: INFO/job_scheduler] Job 2387: complete_job: Updating cache
[2017-06-01 19:46:08,321: INFO/job_scheduler] Job 2387 completing (errored=False, cancelled=False)
[2017-06-01 19:46:08,413: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:46:08,461: INFO/job_scheduler] Job 2388 finished 1 steps successfully
[2017-06-01 19:46:08,507: INFO/job_scheduler] Job 2388: complete_job: Updating cache
[2017-06-01 19:46:08,507: INFO/job_scheduler] Job 2388 completing (errored=False, cancelled=False)
[2017-06-01 19:46:08,518: INFO/job_scheduler] Job 2389 finished 1 steps successfully
[2017-06-01 19:46:08,557: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:46:08,621: INFO/job_scheduler] Job 2389: complete_job: Updating cache
[2017-06-01 19:46:08,622: INFO/job_scheduler] Job 2389 completing (errored=False, cancelled=False)
[2017-06-01 19:46:08,656: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:46:08,894: INFO/job_scheduler] Job 2390 finished 1 steps successfully
[2017-06-01 19:46:08,912: INFO/job_scheduler] Job 2390: complete_job: Updating cache
[2017-06-01 19:46:08,913: INFO/job_scheduler] Job 2390 completing (errored=False, cancelled=False)
[2017-06-01 19:46:08,958: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:46:10,373: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:10,375: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:10,396: INFO/command_plan] add_jobs: created Job 2391 (Test for host connectivity)
[2017-06-01 19:46:10,431: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:46:10,483: INFO/job_scheduler] Job 2391: RunJobThread.run
[2017-06-01 19:46:10,546: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:10,547: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:10,588: INFO/command_plan] add_jobs: created Job 2392 (Test for host connectivity)
[2017-06-01 19:46:10,684: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:46:10,726: INFO/job_scheduler] Job 2392: RunJobThread.run
[2017-06-01 19:46:10,773: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:10,774: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:10,788: INFO/command_plan] add_jobs: created Job 2393 (Test for host connectivity)
[2017-06-01 19:46:10,798: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:46:10,817: INFO/job_scheduler] Job 2393: RunJobThread.run
[2017-06-01 19:46:11,016: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:11,019: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:11,071: INFO/command_plan] add_jobs: created Job 2394 (Test for host connectivity)
[2017-06-01 19:46:11,145: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:46:11,181: INFO/job_scheduler] Job 2394: RunJobThread.run
[2017-06-01 19:46:12,939: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:46:13,117: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:46:13,314: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:46:13,412: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:46:14,296: INFO/job_scheduler] Job 2391 finished 1 steps successfully
[2017-06-01 19:46:14,323: INFO/job_scheduler] Job 2391: complete_job: Updating cache
[2017-06-01 19:46:14,323: INFO/job_scheduler] Job 2391 completing (errored=False, cancelled=False)
[2017-06-01 19:46:14,424: INFO/job_scheduler] Job 2392 finished 1 steps successfully
[2017-06-01 19:46:14,466: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:46:14,507: INFO/job_scheduler] Job 2392: complete_job: Updating cache
[2017-06-01 19:46:14,507: INFO/job_scheduler] Job 2392 completing (errored=False, cancelled=False)
[2017-06-01 19:46:14,567: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:46:14,676: INFO/job_scheduler] Job 2393 finished 1 steps successfully
[2017-06-01 19:46:14,750: INFO/job_scheduler] Job 2393: complete_job: Updating cache
[2017-06-01 19:46:14,750: INFO/job_scheduler] Job 2393 completing (errored=False, cancelled=False)
[2017-06-01 19:46:14,782: INFO/job_scheduler] Job 2394 finished 1 steps successfully
[2017-06-01 19:46:14,807: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:46:14,845: INFO/job_scheduler] Job 2394: complete_job: Updating cache
[2017-06-01 19:46:14,845: INFO/job_scheduler] Job 2394 completing (errored=False, cancelled=False)
[2017-06-01 19:46:14,890: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:46:16,973: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:16,973: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:16,990: INFO/command_plan] add_jobs: created Job 2395 (Test for host connectivity)
[2017-06-01 19:46:17,071: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:46:17,079: INFO/job_scheduler] Job 2395: RunJobThread.run
[2017-06-01 19:46:17,143: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:17,151: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:17,166: INFO/command_plan] add_jobs: created Job 2396 (Test for host connectivity)
[2017-06-01 19:46:17,219: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:46:17,228: INFO/job_scheduler] Job 2396: RunJobThread.run
[2017-06-01 19:46:17,312: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:17,313: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:17,340: INFO/command_plan] add_jobs: created Job 2397 (Test for host connectivity)
[2017-06-01 19:46:17,433: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:46:17,474: INFO/job_scheduler] Job 2397: RunJobThread.run
[2017-06-01 19:46:17,557: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:17,566: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:17,581: INFO/command_plan] add_jobs: created Job 2398 (Test for host connectivity)
[2017-06-01 19:46:17,634: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:46:17,668: INFO/job_scheduler] Job 2398: RunJobThread.run
[2017-06-01 19:46:19,428: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:46:19,715: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:46:19,830: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:46:19,980: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:46:20,786: INFO/job_scheduler] Job 2395 finished 1 steps successfully
[2017-06-01 19:46:20,797: INFO/job_scheduler] Job 2395: complete_job: Updating cache
[2017-06-01 19:46:20,797: INFO/job_scheduler] Job 2395 completing (errored=False, cancelled=False)
[2017-06-01 19:46:20,923: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:46:21,125: INFO/job_scheduler] Job 2396 finished 1 steps successfully
[2017-06-01 19:46:21,177: INFO/job_scheduler] Job 2396: complete_job: Updating cache
[2017-06-01 19:46:21,178: INFO/job_scheduler] Job 2396 completing (errored=False, cancelled=False)
[2017-06-01 19:46:21,243: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:46:21,443: INFO/job_scheduler] Job 2397 finished 1 steps successfully
[2017-06-01 19:46:21,476: INFO/job_scheduler] Job 2398 finished 1 steps successfully
[2017-06-01 19:46:21,489: INFO/job_scheduler] Job 2397: complete_job: Updating cache
[2017-06-01 19:46:21,489: INFO/job_scheduler] Job 2397 completing (errored=False, cancelled=False)
[2017-06-01 19:46:21,556: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:46:21,573: INFO/job_scheduler] Job 2398: complete_job: Updating cache
[2017-06-01 19:46:21,573: INFO/job_scheduler] Job 2398 completing (errored=False, cancelled=False)
[2017-06-01 19:46:21,630: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:46:23,458: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:23,459: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:23,471: INFO/command_plan] add_jobs: created Job 2399 (Test for host connectivity)
[2017-06-01 19:46:23,632: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:46:23,634: INFO/job_scheduler] Job 2399: RunJobThread.run
[2017-06-01 19:46:23,656: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:23,657: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:23,680: INFO/command_plan] add_jobs: created Job 2400 (Test for host connectivity)
[2017-06-01 19:46:23,715: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:46:23,744: INFO/job_scheduler] Job 2400: RunJobThread.run
[2017-06-01 19:46:23,853: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:23,854: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:23,874: INFO/command_plan] add_jobs: created Job 2401 (Test for host connectivity)
[2017-06-01 19:46:23,958: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:46:23,989: INFO/job_scheduler] Job 2401: RunJobThread.run
[2017-06-01 19:46:24,040: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:24,041: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:24,061: INFO/command_plan] add_jobs: created Job 2402 (Test for host connectivity)
[2017-06-01 19:46:24,121: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:46:24,150: INFO/job_scheduler] Job 2402: RunJobThread.run
[2017-06-01 19:46:25,966: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:46:26,168: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:46:26,224: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:46:26,500: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:46:27,298: INFO/job_scheduler] Job 2399 finished 1 steps successfully
[2017-06-01 19:46:27,344: INFO/job_scheduler] Job 2399: complete_job: Updating cache
[2017-06-01 19:46:27,344: INFO/job_scheduler] Job 2399 completing (errored=False, cancelled=False)
[2017-06-01 19:46:27,393: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:46:27,586: INFO/job_scheduler] Job 2400 finished 1 steps successfully
[2017-06-01 19:46:27,663: INFO/job_scheduler] Job 2400: complete_job: Updating cache
[2017-06-01 19:46:27,663: INFO/job_scheduler] Job 2400 completing (errored=False, cancelled=False)
[2017-06-01 19:46:27,697: INFO/job_scheduler] Job 2401 finished 1 steps successfully
[2017-06-01 19:46:27,770: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:46:27,799: INFO/job_scheduler] Job 2401: complete_job: Updating cache
[2017-06-01 19:46:27,800: INFO/job_scheduler] Job 2401 completing (errored=False, cancelled=False)
[2017-06-01 19:46:27,874: INFO/job_scheduler] Job 2402 finished 1 steps successfully
[2017-06-01 19:46:27,908: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:46:27,930: INFO/job_scheduler] Job 2402: complete_job: Updating cache
[2017-06-01 19:46:27,931: INFO/job_scheduler] Job 2402 completing (errored=False, cancelled=False)
[2017-06-01 19:46:27,990: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:46:29,994: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:29,996: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:30,003: INFO/command_plan] add_jobs: created Job 2403 (Test for host connectivity)
[2017-06-01 19:46:30,085: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:46:30,099: INFO/job_scheduler] Job 2403: RunJobThread.run
[2017-06-01 19:46:30,143: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:30,144: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:30,155: INFO/command_plan] add_jobs: created Job 2404 (Test for host connectivity)
[2017-06-01 19:46:30,192: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:46:30,274: INFO/job_scheduler] Job 2404: RunJobThread.run
[2017-06-01 19:46:30,324: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:30,326: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:30,363: INFO/command_plan] add_jobs: created Job 2405 (Test for host connectivity)
[2017-06-01 19:46:30,461: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:30,461: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:30,499: INFO/command_plan] add_jobs: created Job 2406 (Test for host connectivity)
[2017-06-01 19:46:30,535: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:46:30,576: INFO/job_scheduler] Job 2405: RunJobThread.run
[2017-06-01 19:46:30,634: INFO/job_scheduler] Job 2406: RunJobThread.run
[2017-06-01 19:46:30,646: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:46:32,434: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:46:32,774: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:46:32,983: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:46:33,042: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:46:33,767: INFO/job_scheduler] Job 2403 finished 1 steps successfully
[2017-06-01 19:46:33,811: INFO/job_scheduler] Job 2403: complete_job: Updating cache
[2017-06-01 19:46:33,811: INFO/job_scheduler] Job 2403 completing (errored=False, cancelled=False)
[2017-06-01 19:46:33,862: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:46:34,334: INFO/job_scheduler] Job 2404 finished 1 steps successfully
[2017-06-01 19:46:34,419: INFO/job_scheduler] Job 2404: complete_job: Updating cache
[2017-06-01 19:46:34,420: INFO/job_scheduler] Job 2404 completing (errored=False, cancelled=False)
[2017-06-01 19:46:34,463: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:46:34,475: INFO/job_scheduler] Job 2406 finished 1 steps successfully
[2017-06-01 19:46:34,511: INFO/job_scheduler] Job 2406: complete_job: Updating cache
[2017-06-01 19:46:34,512: INFO/job_scheduler] Job 2406 completing (errored=False, cancelled=False)
[2017-06-01 19:46:34,515: INFO/job_scheduler] Job 2405 finished 1 steps successfully
[2017-06-01 19:46:34,628: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:46:34,637: INFO/job_scheduler] Job 2405: complete_job: Updating cache
[2017-06-01 19:46:34,637: INFO/job_scheduler] Job 2405 completing (errored=False, cancelled=False)
[2017-06-01 19:46:34,713: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:46:36,342: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:36,343: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:36,352: INFO/command_plan] add_jobs: created Job 2407 (Test for host connectivity)
[2017-06-01 19:46:36,527: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:36,527: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:36,555: INFO/command_plan] add_jobs: created Job 2408 (Test for host connectivity)
[2017-06-01 19:46:36,604: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:46:36,639: INFO/job_scheduler] Job 2408: RunJobThread.run
[2017-06-01 19:46:36,659: INFO/job_scheduler] Job 2407: RunJobThread.run
[2017-06-01 19:46:36,674: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:46:36,750: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:36,752: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:36,769: INFO/command_plan] add_jobs: created Job 2409 (Test for host connectivity)
[2017-06-01 19:46:36,878: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:46:36,897: INFO/job_scheduler] Job 2409: RunJobThread.run
[2017-06-01 19:46:36,951: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:36,952: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:36,978: INFO/command_plan] add_jobs: created Job 2410 (Test for host connectivity)
[2017-06-01 19:46:37,001: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:46:37,033: INFO/job_scheduler] Job 2410: RunJobThread.run
[2017-06-01 19:46:39,024: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:46:39,113: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:46:39,177: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:46:39,378: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:46:40,403: INFO/job_scheduler] Job 2408 finished 1 steps successfully
[2017-06-01 19:46:40,435: INFO/job_scheduler] Job 2408: complete_job: Updating cache
[2017-06-01 19:46:40,436: INFO/job_scheduler] Job 2408 completing (errored=False, cancelled=False)
[2017-06-01 19:46:40,506: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:46:40,508: INFO/job_scheduler] Job 2407 finished 1 steps successfully
[2017-06-01 19:46:40,535: INFO/job_scheduler] Job 2407: complete_job: Updating cache
[2017-06-01 19:46:40,535: INFO/job_scheduler] Job 2407 completing (errored=False, cancelled=False)
[2017-06-01 19:46:40,557: INFO/job_scheduler] Job 2409 finished 1 steps successfully
[2017-06-01 19:46:40,606: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:46:40,636: INFO/job_scheduler] Job 2409: complete_job: Updating cache
[2017-06-01 19:46:40,637: INFO/job_scheduler] Job 2409 completing (errored=False, cancelled=False)
[2017-06-01 19:46:40,729: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:46:40,746: INFO/job_scheduler] Job 2410 finished 1 steps successfully
[2017-06-01 19:46:40,786: INFO/job_scheduler] Job 2410: complete_job: Updating cache
[2017-06-01 19:46:40,786: INFO/job_scheduler] Job 2410 completing (errored=False, cancelled=False)
[2017-06-01 19:46:40,861: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:46:42,920: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:42,921: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:42,936: INFO/command_plan] add_jobs: created Job 2411 (Test for host connectivity)
[2017-06-01 19:46:43,009: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:46:43,026: INFO/job_scheduler] Job 2411: RunJobThread.run
[2017-06-01 19:46:43,087: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:43,093: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:43,120: INFO/command_plan] add_jobs: created Job 2412 (Test for host connectivity)
[2017-06-01 19:46:43,212: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:46:43,258: INFO/job_scheduler] Job 2412: RunJobThread.run
[2017-06-01 19:46:43,335: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:43,342: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:43,363: INFO/command_plan] add_jobs: created Job 2413 (Test for host connectivity)
[2017-06-01 19:46:43,395: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:46:43,424: INFO/job_scheduler] Job 2413: RunJobThread.run
[2017-06-01 19:46:43,557: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:43,558: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:43,585: INFO/command_plan] add_jobs: created Job 2414 (Test for host connectivity)
[2017-06-01 19:46:43,638: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:46:43,722: INFO/job_scheduler] Job 2414: RunJobThread.run
[2017-06-01 19:46:45,527: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:46:45,881: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:46:46,002: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:46:46,208: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:46:46,872: INFO/job_scheduler] Job 2411 finished 1 steps successfully
[2017-06-01 19:46:46,901: INFO/job_scheduler] Job 2411: complete_job: Updating cache
[2017-06-01 19:46:46,901: INFO/job_scheduler] Job 2411 completing (errored=False, cancelled=False)
[2017-06-01 19:46:46,934: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:46:47,297: INFO/job_scheduler] Job 2412 finished 1 steps successfully
[2017-06-01 19:46:47,319: INFO/job_scheduler] Job 2412: complete_job: Updating cache
[2017-06-01 19:46:47,320: INFO/job_scheduler] Job 2412 completing (errored=False, cancelled=False)
[2017-06-01 19:46:47,384: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:46:47,531: INFO/job_scheduler] Job 2413 finished 1 steps successfully
[2017-06-01 19:46:47,576: INFO/job_scheduler] Job 2414 finished 1 steps successfully
[2017-06-01 19:46:47,583: INFO/job_scheduler] Job 2413: complete_job: Updating cache
[2017-06-01 19:46:47,585: INFO/job_scheduler] Job 2413 completing (errored=False, cancelled=False)
[2017-06-01 19:46:47,650: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:46:47,667: INFO/job_scheduler] Job 2414: complete_job: Updating cache
[2017-06-01 19:46:47,667: INFO/job_scheduler] Job 2414 completing (errored=False, cancelled=False)
[2017-06-01 19:46:47,717: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:46:49,457: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:49,458: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:49,466: INFO/command_plan] add_jobs: created Job 2415 (Test for host connectivity)
[2017-06-01 19:46:49,505: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:46:49,522: INFO/job_scheduler] Job 2415: RunJobThread.run
[2017-06-01 19:46:49,550: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:49,550: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:49,569: INFO/command_plan] add_jobs: created Job 2416 (Test for host connectivity)
[2017-06-01 19:46:49,621: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:46:49,630: INFO/job_scheduler] Job 2416: RunJobThread.run
[2017-06-01 19:46:49,747: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:49,749: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:49,779: INFO/command_plan] add_jobs: created Job 2417 (Test for host connectivity)
[2017-06-01 19:46:49,892: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:46:49,939: INFO/job_scheduler] Job 2417: RunJobThread.run
[2017-06-01 19:46:50,011: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:50,012: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:50,030: INFO/command_plan] add_jobs: created Job 2418 (Test for host connectivity)
[2017-06-01 19:46:50,118: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:46:50,137: INFO/job_scheduler] Job 2418: RunJobThread.run
[2017-06-01 19:46:51,813: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:46:52,046: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:46:52,330: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:46:52,432: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:46:53,160: INFO/job_scheduler] Job 2415 finished 1 steps successfully
[2017-06-01 19:46:53,182: INFO/job_scheduler] Job 2415: complete_job: Updating cache
[2017-06-01 19:46:53,183: INFO/job_scheduler] Job 2415 completing (errored=False, cancelled=False)
[2017-06-01 19:46:53,251: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:46:53,593: INFO/job_scheduler] Job 2416 finished 1 steps successfully
[2017-06-01 19:46:53,707: INFO/job_scheduler] Job 2416: complete_job: Updating cache
[2017-06-01 19:46:53,707: INFO/job_scheduler] Job 2416 completing (errored=False, cancelled=False)
[2017-06-01 19:46:53,762: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:46:53,958: INFO/job_scheduler] Job 2417 finished 1 steps successfully
[2017-06-01 19:46:53,982: INFO/job_scheduler] Job 2417: complete_job: Updating cache
[2017-06-01 19:46:53,983: INFO/job_scheduler] Job 2417 completing (errored=False, cancelled=False)
[2017-06-01 19:46:54,028: INFO/job_scheduler] Job 2418 finished 1 steps successfully
[2017-06-01 19:46:54,044: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:46:54,067: INFO/job_scheduler] Job 2418: complete_job: Updating cache
[2017-06-01 19:46:54,067: INFO/job_scheduler] Job 2418 completing (errored=False, cancelled=False)
[2017-06-01 19:46:54,143: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:46:55,894: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:55,896: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:55,908: INFO/command_plan] add_jobs: created Job 2419 (Test for host connectivity)
[2017-06-01 19:46:56,052: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:46:56,057: INFO/job_scheduler] Job 2419: RunJobThread.run
[2017-06-01 19:46:56,107: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:56,107: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:56,132: INFO/command_plan] add_jobs: created Job 2420 (Test for host connectivity)
[2017-06-01 19:46:56,161: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:46:56,204: INFO/job_scheduler] Job 2420: RunJobThread.run
[2017-06-01 19:46:56,289: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:56,289: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:56,315: INFO/command_plan] add_jobs: created Job 2421 (Test for host connectivity)
[2017-06-01 19:46:56,340: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:46:56,364: INFO/job_scheduler] Job 2421: RunJobThread.run
[2017-06-01 19:46:56,535: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:46:56,536: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:46:56,567: INFO/command_plan] add_jobs: created Job 2422 (Test for host connectivity)
[2017-06-01 19:46:56,638: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:46:56,665: INFO/job_scheduler] Job 2422: RunJobThread.run
[2017-06-01 19:46:58,523: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:46:58,676: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:46:58,823: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:46:58,911: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:46:59,883: INFO/job_scheduler] Job 2419 finished 1 steps successfully
[2017-06-01 19:47:00,002: INFO/job_scheduler] Job 2420 finished 1 steps successfully
[2017-06-01 19:47:00,031: INFO/job_scheduler] Job 2419: complete_job: Updating cache
[2017-06-01 19:47:00,031: INFO/job_scheduler] Job 2419 completing (errored=False, cancelled=False)
[2017-06-01 19:47:00,094: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:47:00,115: INFO/job_scheduler] Job 2420: complete_job: Updating cache
[2017-06-01 19:47:00,115: INFO/job_scheduler] Job 2420 completing (errored=False, cancelled=False)
[2017-06-01 19:47:00,160: INFO/job_scheduler] Job 2421 finished 1 steps successfully
[2017-06-01 19:47:00,195: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:47:00,258: INFO/job_scheduler] Job 2421: complete_job: Updating cache
[2017-06-01 19:47:00,258: INFO/job_scheduler] Job 2421 completing (errored=False, cancelled=False)
[2017-06-01 19:47:00,282: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:47:00,287: INFO/job_scheduler] Job 2422 finished 1 steps successfully
[2017-06-01 19:47:00,348: INFO/job_scheduler] Job 2422: complete_job: Updating cache
[2017-06-01 19:47:00,349: INFO/job_scheduler] Job 2422 completing (errored=False, cancelled=False)
[2017-06-01 19:47:00,428: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:47:02,407: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:02,409: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:02,426: INFO/command_plan] add_jobs: created Job 2423 (Test for host connectivity)
[2017-06-01 19:47:02,487: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:47:02,503: INFO/job_scheduler] Job 2423: RunJobThread.run
[2017-06-01 19:47:02,564: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:02,564: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:02,585: INFO/command_plan] add_jobs: created Job 2424 (Test for host connectivity)
[2017-06-01 19:47:02,621: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:47:02,624: INFO/job_scheduler] Job 2424: RunJobThread.run
[2017-06-01 19:47:02,761: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:02,763: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:02,797: INFO/command_plan] add_jobs: created Job 2425 (Test for host connectivity)
[2017-06-01 19:47:02,876: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:47:02,906: INFO/job_scheduler] Job 2425: RunJobThread.run
[2017-06-01 19:47:02,984: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:02,984: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:02,998: INFO/command_plan] add_jobs: created Job 2426 (Test for host connectivity)
[2017-06-01 19:47:03,043: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:47:03,076: INFO/job_scheduler] Job 2426: RunJobThread.run
[2017-06-01 19:47:04,893: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:47:05,070: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:47:05,392: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:47:05,503: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:47:06,212: INFO/job_scheduler] Job 2423 finished 1 steps successfully
[2017-06-01 19:47:06,241: INFO/job_scheduler] Job 2423: complete_job: Updating cache
[2017-06-01 19:47:06,241: INFO/job_scheduler] Job 2423 completing (errored=False, cancelled=False)
[2017-06-01 19:47:06,326: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:47:06,437: INFO/job_scheduler] Job 2424 finished 1 steps successfully
[2017-06-01 19:47:06,497: INFO/job_scheduler] Job 2424: complete_job: Updating cache
[2017-06-01 19:47:06,501: INFO/job_scheduler] Job 2424 completing (errored=False, cancelled=False)
[2017-06-01 19:47:06,566: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:47:06,778: INFO/job_scheduler] Job 2425 finished 1 steps successfully
[2017-06-01 19:47:06,822: INFO/job_scheduler] Job 2425: complete_job: Updating cache
[2017-06-01 19:47:06,822: INFO/job_scheduler] Job 2425 completing (errored=False, cancelled=False)
[2017-06-01 19:47:06,877: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:47:06,972: INFO/job_scheduler] Job 2426 finished 1 steps successfully
[2017-06-01 19:47:07,034: INFO/job_scheduler] Job 2426: complete_job: Updating cache
[2017-06-01 19:47:07,036: INFO/job_scheduler] Job 2426 completing (errored=False, cancelled=False)
[2017-06-01 19:47:07,112: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:47:08,885: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:08,887: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:08,901: INFO/command_plan] add_jobs: created Job 2427 (Test for host connectivity)
[2017-06-01 19:47:08,934: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:47:08,945: INFO/job_scheduler] Job 2427: RunJobThread.run
[2017-06-01 19:47:09,000: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:09,001: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:09,025: INFO/command_plan] add_jobs: created Job 2428 (Test for host connectivity)
[2017-06-01 19:47:09,076: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:47:09,106: INFO/job_scheduler] Job 2428: RunJobThread.run
[2017-06-01 19:47:09,214: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:09,214: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:09,253: INFO/command_plan] add_jobs: created Job 2429 (Test for host connectivity)
[2017-06-01 19:47:09,353: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:47:09,384: INFO/job_scheduler] Job 2429: RunJobThread.run
[2017-06-01 19:47:09,463: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:09,463: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:09,477: INFO/command_plan] add_jobs: created Job 2430 (Test for host connectivity)
[2017-06-01 19:47:09,514: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:47:09,575: INFO/job_scheduler] Job 2430: RunJobThread.run
[2017-06-01 19:47:11,353: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:47:11,587: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:47:11,797: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:47:11,805: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:47:12,707: INFO/job_scheduler] Job 2427 finished 1 steps successfully
[2017-06-01 19:47:12,750: INFO/job_scheduler] Job 2427: complete_job: Updating cache
[2017-06-01 19:47:12,750: INFO/job_scheduler] Job 2427 completing (errored=False, cancelled=False)
[2017-06-01 19:47:12,790: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:47:12,890: INFO/job_scheduler] Job 2428 finished 1 steps successfully
[2017-06-01 19:47:12,991: INFO/job_scheduler] Job 2428: complete_job: Updating cache
[2017-06-01 19:47:12,991: INFO/job_scheduler] Job 2428 completing (errored=False, cancelled=False)
[2017-06-01 19:47:13,040: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:47:13,235: INFO/job_scheduler] Job 2430 finished 1 steps successfully
[2017-06-01 19:47:13,242: INFO/job_scheduler] Job 2429 finished 1 steps successfully
[2017-06-01 19:47:13,290: INFO/job_scheduler] Job 2430: complete_job: Updating cache
[2017-06-01 19:47:13,290: INFO/job_scheduler] Job 2430 completing (errored=False, cancelled=False)
[2017-06-01 19:47:13,323: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:47:13,360: INFO/job_scheduler] Job 2429: complete_job: Updating cache
[2017-06-01 19:47:13,360: INFO/job_scheduler] Job 2429 completing (errored=False, cancelled=False)
[2017-06-01 19:47:13,424: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:47:15,304: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:15,305: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:15,328: INFO/command_plan] add_jobs: created Job 2431 (Test for host connectivity)
[2017-06-01 19:47:15,352: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:47:15,376: INFO/job_scheduler] Job 2431: RunJobThread.run
[2017-06-01 19:47:15,510: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:15,510: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:15,525: INFO/command_plan] add_jobs: created Job 2432 (Test for host connectivity)
[2017-06-01 19:47:15,558: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:47:15,560: INFO/job_scheduler] Job 2432: RunJobThread.run
[2017-06-01 19:47:15,717: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:15,722: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:15,732: INFO/command_plan] add_jobs: created Job 2433 (Test for host connectivity)
[2017-06-01 19:47:15,856: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:15,856: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:15,873: INFO/command_plan] add_jobs: created Job 2434 (Test for host connectivity)
[2017-06-01 19:47:15,887: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:47:15,920: INFO/job_scheduler] Job 2433: RunJobThread.run
[2017-06-01 19:47:15,951: INFO/job_scheduler] Job 2434: RunJobThread.run
[2017-06-01 19:47:15,976: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:47:17,878: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:47:17,977: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:47:18,235: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:47:18,412: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:47:19,216: INFO/job_scheduler] Job 2431 finished 1 steps successfully
[2017-06-01 19:47:19,257: INFO/job_scheduler] Job 2431: complete_job: Updating cache
[2017-06-01 19:47:19,257: INFO/job_scheduler] Job 2431 completing (errored=False, cancelled=False)
[2017-06-01 19:47:19,316: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:47:19,322: INFO/job_scheduler] Job 2432 finished 1 steps successfully
[2017-06-01 19:47:19,354: INFO/job_scheduler] Job 2432: complete_job: Updating cache
[2017-06-01 19:47:19,354: INFO/job_scheduler] Job 2432 completing (errored=False, cancelled=False)
[2017-06-01 19:47:19,481: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:47:19,842: INFO/job_scheduler] Job 2434 finished 1 steps successfully
[2017-06-01 19:47:19,864: INFO/job_scheduler] Job 2434: complete_job: Updating cache
[2017-06-01 19:47:19,864: INFO/job_scheduler] Job 2434 completing (errored=False, cancelled=False)
[2017-06-01 19:47:19,890: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:47:19,962: INFO/job_scheduler] Job 2433 finished 1 steps successfully
[2017-06-01 19:47:19,998: INFO/job_scheduler] Job 2433: complete_job: Updating cache
[2017-06-01 19:47:20,000: INFO/job_scheduler] Job 2433 completing (errored=False, cancelled=False)
[2017-06-01 19:47:20,190: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:47:21,827: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:21,827: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:21,840: INFO/command_plan] add_jobs: created Job 2435 (Test for host connectivity)
[2017-06-01 19:47:21,904: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:47:21,911: INFO/job_scheduler] Job 2435: RunJobThread.run
[2017-06-01 19:47:21,961: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:21,962: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:21,981: INFO/command_plan] add_jobs: created Job 2436 (Test for host connectivity)
[2017-06-01 19:47:22,111: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:22,112: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:22,148: INFO/command_plan] add_jobs: created Job 2437 (Test for host connectivity)
[2017-06-01 19:47:22,203: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:47:22,211: INFO/job_scheduler] Job 2436: RunJobThread.run
[2017-06-01 19:47:22,247: INFO/job_scheduler] Job 2437: RunJobThread.run
[2017-06-01 19:47:22,276: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:47:22,348: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:22,349: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:22,417: INFO/command_plan] add_jobs: created Job 2438 (Test for host connectivity)
[2017-06-01 19:47:22,496: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:47:22,545: INFO/job_scheduler] Job 2438: RunJobThread.run
[2017-06-01 19:47:24,244: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:47:24,485: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:47:24,566: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:47:24,772: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:47:25,582: INFO/job_scheduler] Job 2435 finished 1 steps successfully
[2017-06-01 19:47:25,655: INFO/job_scheduler] Job 2435: complete_job: Updating cache
[2017-06-01 19:47:25,656: INFO/job_scheduler] Job 2435 completing (errored=False, cancelled=False)
[2017-06-01 19:47:25,681: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:47:25,855: INFO/job_scheduler] Job 2436 finished 1 steps successfully
[2017-06-01 19:47:25,923: INFO/job_scheduler] Job 2436: complete_job: Updating cache
[2017-06-01 19:47:25,923: INFO/job_scheduler] Job 2436 completing (errored=False, cancelled=False)
[2017-06-01 19:47:25,947: INFO/job_scheduler] Job 2437 finished 1 steps successfully
[2017-06-01 19:47:26,017: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:47:26,048: INFO/job_scheduler] Job 2437: complete_job: Updating cache
[2017-06-01 19:47:26,053: INFO/job_scheduler] Job 2437 completing (errored=False, cancelled=False)
[2017-06-01 19:47:26,150: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:47:26,218: INFO/job_scheduler] Job 2438 finished 1 steps successfully
[2017-06-01 19:47:26,257: INFO/job_scheduler] Job 2438: complete_job: Updating cache
[2017-06-01 19:47:26,257: INFO/job_scheduler] Job 2438 completing (errored=False, cancelled=False)
[2017-06-01 19:47:26,341: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:47:28,288: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:28,288: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:28,297: INFO/command_plan] add_jobs: created Job 2439 (Test for host connectivity)
[2017-06-01 19:47:28,310: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:47:28,334: INFO/job_scheduler] Job 2439: RunJobThread.run
[2017-06-01 19:47:28,365: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:28,409: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:28,439: INFO/command_plan] add_jobs: created Job 2440 (Test for host connectivity)
[2017-06-01 19:47:28,480: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:47:28,498: INFO/job_scheduler] Job 2440: RunJobThread.run
[2017-06-01 19:47:28,609: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:28,611: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:28,630: INFO/command_plan] add_jobs: created Job 2441 (Test for host connectivity)
[2017-06-01 19:47:28,679: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:47:28,698: INFO/job_scheduler] Job 2441: RunJobThread.run
[2017-06-01 19:47:28,795: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:28,795: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:28,818: INFO/command_plan] add_jobs: created Job 2442 (Test for host connectivity)
[2017-06-01 19:47:28,877: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:47:28,932: INFO/job_scheduler] Job 2442: RunJobThread.run
[2017-06-01 19:47:30,746: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:47:31,002: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:47:31,085: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:47:31,219: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:47:32,117: INFO/job_scheduler] Job 2439 finished 1 steps successfully
[2017-06-01 19:47:32,159: INFO/job_scheduler] Job 2439: complete_job: Updating cache
[2017-06-01 19:47:32,160: INFO/job_scheduler] Job 2439 completing (errored=False, cancelled=False)
[2017-06-01 19:47:32,197: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:47:32,381: INFO/job_scheduler] Job 2440 finished 1 steps successfully
[2017-06-01 19:47:32,431: INFO/job_scheduler] Job 2440: complete_job: Updating cache
[2017-06-01 19:47:32,432: INFO/job_scheduler] Job 2440 completing (errored=False, cancelled=False)
[2017-06-01 19:47:32,488: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:47:32,593: INFO/job_scheduler] Job 2441 finished 1 steps successfully
[2017-06-01 19:47:32,633: INFO/job_scheduler] Job 2442 finished 1 steps successfully
[2017-06-01 19:47:32,639: INFO/job_scheduler] Job 2441: complete_job: Updating cache
[2017-06-01 19:47:32,639: INFO/job_scheduler] Job 2441 completing (errored=False, cancelled=False)
[2017-06-01 19:47:32,697: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:47:32,713: INFO/job_scheduler] Job 2442: complete_job: Updating cache
[2017-06-01 19:47:32,713: INFO/job_scheduler] Job 2442 completing (errored=False, cancelled=False)
[2017-06-01 19:47:32,848: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:47:34,694: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:34,695: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:34,704: INFO/command_plan] add_jobs: created Job 2443 (Test for host connectivity)
[2017-06-01 19:47:34,733: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:47:34,737: INFO/job_scheduler] Job 2443: RunJobThread.run
[2017-06-01 19:47:34,814: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:34,817: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:34,863: INFO/command_plan] add_jobs: created Job 2444 (Test for host connectivity)
[2017-06-01 19:47:34,954: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:47:34,973: INFO/job_scheduler] Job 2444: RunJobThread.run
[2017-06-01 19:47:35,070: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:35,070: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:35,084: INFO/command_plan] add_jobs: created Job 2445 (Test for host connectivity)
[2017-06-01 19:47:35,171: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:35,172: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:35,212: INFO/command_plan] add_jobs: created Job 2446 (Test for host connectivity)
[2017-06-01 19:47:35,235: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:47:35,258: INFO/job_scheduler] Job 2445: RunJobThread.run
[2017-06-01 19:47:35,286: INFO/job_scheduler] Job 2446: RunJobThread.run
[2017-06-01 19:47:35,299: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:47:37,202: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:47:37,364: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:47:37,496: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:47:37,617: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:47:38,489: INFO/job_scheduler] Job 2443 finished 1 steps successfully
[2017-06-01 19:47:38,521: INFO/job_scheduler] Job 2443: complete_job: Updating cache
[2017-06-01 19:47:38,521: INFO/job_scheduler] Job 2443 completing (errored=False, cancelled=False)
[2017-06-01 19:47:38,555: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:47:38,695: INFO/job_scheduler] Job 2444 finished 1 steps successfully
[2017-06-01 19:47:38,782: INFO/job_scheduler] Job 2444: complete_job: Updating cache
[2017-06-01 19:47:38,782: INFO/job_scheduler] Job 2444 completing (errored=False, cancelled=False)
[2017-06-01 19:47:38,816: INFO/job_scheduler] Job 2446 finished 1 steps successfully
[2017-06-01 19:47:38,859: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:47:38,887: INFO/job_scheduler] Job 2446: complete_job: Updating cache
[2017-06-01 19:47:38,887: INFO/job_scheduler] Job 2446 completing (errored=False, cancelled=False)
[2017-06-01 19:47:39,005: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:47:39,058: INFO/job_scheduler] Job 2445 finished 1 steps successfully
[2017-06-01 19:47:39,097: INFO/job_scheduler] Job 2445: complete_job: Updating cache
[2017-06-01 19:47:39,098: INFO/job_scheduler] Job 2445 completing (errored=False, cancelled=False)
[2017-06-01 19:47:39,155: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:47:40,737: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:40,739: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:40,754: INFO/command_plan] add_jobs: created Job 2447 (Test for host connectivity)
[2017-06-01 19:47:40,867: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:47:40,876: INFO/job_scheduler] Job 2447: RunJobThread.run
[2017-06-01 19:47:40,921: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:40,921: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:40,939: INFO/command_plan] add_jobs: created Job 2448 (Test for host connectivity)
[2017-06-01 19:47:41,156: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:41,156: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:41,170: INFO/command_plan] add_jobs: created Job 2449 (Test for host connectivity)
[2017-06-01 19:47:41,207: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:47:41,219: INFO/job_scheduler] Job 2448: RunJobThread.run
[2017-06-01 19:47:41,220: INFO/job_scheduler] Job 2449: RunJobThread.run
[2017-06-01 19:47:41,242: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:47:41,261: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:41,261: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:41,291: INFO/command_plan] add_jobs: created Job 2450 (Test for host connectivity)
[2017-06-01 19:47:41,324: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:47:41,351: INFO/job_scheduler] Job 2450: RunJobThread.run
[2017-06-01 19:47:43,270: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:47:43,424: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:47:43,583: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:47:43,618: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:47:44,716: INFO/job_scheduler] Job 2447 finished 1 steps successfully
[2017-06-01 19:47:44,774: INFO/job_scheduler] Job 2448 finished 1 steps successfully
[2017-06-01 19:47:44,787: INFO/job_scheduler] Job 2447: complete_job: Updating cache
[2017-06-01 19:47:44,787: INFO/job_scheduler] Job 2447 completing (errored=False, cancelled=False)
[2017-06-01 19:47:44,837: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:47:44,853: INFO/job_scheduler] Job 2448: complete_job: Updating cache
[2017-06-01 19:47:44,854: INFO/job_scheduler] Job 2448 completing (errored=False, cancelled=False)
[2017-06-01 19:47:44,922: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:47:44,930: INFO/job_scheduler] Job 2450 finished 1 steps successfully
[2017-06-01 19:47:44,965: INFO/job_scheduler] Job 2450: complete_job: Updating cache
[2017-06-01 19:47:44,965: INFO/job_scheduler] Job 2450 completing (errored=False, cancelled=False)
[2017-06-01 19:47:45,013: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:47:45,016: INFO/job_scheduler] Job 2449 finished 1 steps successfully
[2017-06-01 19:47:45,029: INFO/job_scheduler] Job 2449: complete_job: Updating cache
[2017-06-01 19:47:45,029: INFO/job_scheduler] Job 2449 completing (errored=False, cancelled=False)
[2017-06-01 19:47:45,091: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:47:47,164: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:47,166: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:47,183: INFO/command_plan] add_jobs: created Job 2451 (Test for host connectivity)
[2017-06-01 19:47:47,210: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:47:47,227: INFO/job_scheduler] Job 2451: RunJobThread.run
[2017-06-01 19:47:47,396: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:47,398: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:47,416: INFO/command_plan] add_jobs: created Job 2452 (Test for host connectivity)
[2017-06-01 19:47:47,537: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:47:47,540: INFO/job_scheduler] Job 2452: RunJobThread.run
[2017-06-01 19:47:47,636: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:47,637: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:47,662: INFO/command_plan] add_jobs: created Job 2453 (Test for host connectivity)
[2017-06-01 19:47:47,722: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:47:47,757: INFO/job_scheduler] Job 2453: RunJobThread.run
[2017-06-01 19:47:47,832: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:47,833: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:47,894: INFO/command_plan] add_jobs: created Job 2454 (Test for host connectivity)
[2017-06-01 19:47:47,952: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:47:47,973: INFO/job_scheduler] Job 2454: RunJobThread.run
[2017-06-01 19:47:49,711: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:47:49,960: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:47:50,135: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:47:50,355: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:47:51,034: INFO/job_scheduler] Job 2451 finished 1 steps successfully
[2017-06-01 19:47:51,094: INFO/job_scheduler] Job 2451: complete_job: Updating cache
[2017-06-01 19:47:51,094: INFO/job_scheduler] Job 2451 completing (errored=False, cancelled=False)
[2017-06-01 19:47:51,162: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:47:51,370: INFO/job_scheduler] Job 2452 finished 1 steps successfully
[2017-06-01 19:47:51,403: INFO/job_scheduler] Job 2452: complete_job: Updating cache
[2017-06-01 19:47:51,404: INFO/job_scheduler] Job 2452 completing (errored=False, cancelled=False)
[2017-06-01 19:47:51,519: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:47:51,565: INFO/job_scheduler] Job 2453 finished 1 steps successfully
[2017-06-01 19:47:51,612: INFO/job_scheduler] Job 2453: complete_job: Updating cache
[2017-06-01 19:47:51,612: INFO/job_scheduler] Job 2453 completing (errored=False, cancelled=False)
[2017-06-01 19:47:51,665: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:47:51,726: INFO/job_scheduler] Job 2454 finished 1 steps successfully
[2017-06-01 19:47:51,812: INFO/job_scheduler] Job 2454: complete_job: Updating cache
[2017-06-01 19:47:51,813: INFO/job_scheduler] Job 2454 completing (errored=False, cancelled=False)
[2017-06-01 19:47:51,870: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:47:53,726: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:53,726: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:53,735: INFO/command_plan] add_jobs: created Job 2455 (Test for host connectivity)
[2017-06-01 19:47:53,796: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:47:53,800: INFO/job_scheduler] Job 2455: RunJobThread.run
[2017-06-01 19:47:53,861: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:53,861: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:53,882: INFO/command_plan] add_jobs: created Job 2456 (Test for host connectivity)
[2017-06-01 19:47:53,930: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:47:53,967: INFO/job_scheduler] Job 2456: RunJobThread.run
[2017-06-01 19:47:54,059: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:54,059: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:54,069: INFO/command_plan] add_jobs: created Job 2457 (Test for host connectivity)
[2017-06-01 19:47:54,209: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:47:54,211: INFO/job_scheduler] Job 2457: RunJobThread.run
[2017-06-01 19:47:54,257: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:47:54,260: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:47:54,270: INFO/command_plan] add_jobs: created Job 2458 (Test for host connectivity)
[2017-06-01 19:47:54,295: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:47:54,354: INFO/job_scheduler] Job 2458: RunJobThread.run
[2017-06-01 19:47:56,248: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:47:56,420: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:47:56,591: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:47:56,760: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:47:57,717: INFO/job_scheduler] Job 2455 finished 1 steps successfully
[2017-06-01 19:47:57,784: INFO/job_scheduler] Job 2455: complete_job: Updating cache
[2017-06-01 19:47:57,784: INFO/job_scheduler] Job 2455 completing (errored=False, cancelled=False)
[2017-06-01 19:47:57,819: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:47:57,868: INFO/job_scheduler] Job 2456 finished 1 steps successfully
[2017-06-01 19:47:57,937: INFO/job_scheduler] Job 2456: complete_job: Updating cache
[2017-06-01 19:47:57,937: INFO/job_scheduler] Job 2456 completing (errored=False, cancelled=False)
[2017-06-01 19:47:57,987: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:47:58,025: INFO/job_scheduler] Job 2457 finished 1 steps successfully
[2017-06-01 19:47:58,091: INFO/job_scheduler] Job 2457: complete_job: Updating cache
[2017-06-01 19:47:58,091: INFO/job_scheduler] Job 2457 completing (errored=False, cancelled=False)
[2017-06-01 19:47:58,104: INFO/job_scheduler] Job 2458 finished 1 steps successfully
[2017-06-01 19:47:58,162: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:47:58,177: INFO/job_scheduler] Job 2458: complete_job: Updating cache
[2017-06-01 19:47:58,177: INFO/job_scheduler] Job 2458 completing (errored=False, cancelled=False)
[2017-06-01 19:47:58,261: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:48:00,101: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:00,103: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:00,113: INFO/command_plan] add_jobs: created Job 2459 (Test for host connectivity)
[2017-06-01 19:48:00,131: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:48:00,154: INFO/job_scheduler] Job 2459: RunJobThread.run
[2017-06-01 19:48:00,219: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:00,219: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:00,235: INFO/command_plan] add_jobs: created Job 2460 (Test for host connectivity)
[2017-06-01 19:48:00,249: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:48:00,251: INFO/job_scheduler] Job 2460: RunJobThread.run
[2017-06-01 19:48:00,405: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:00,405: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:00,447: INFO/command_plan] add_jobs: created Job 2461 (Test for host connectivity)
[2017-06-01 19:48:00,517: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:48:00,557: INFO/job_scheduler] Job 2461: RunJobThread.run
[2017-06-01 19:48:00,633: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:00,634: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:00,651: INFO/command_plan] add_jobs: created Job 2462 (Test for host connectivity)
[2017-06-01 19:48:00,727: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:48:00,764: INFO/job_scheduler] Job 2462: RunJobThread.run
[2017-06-01 19:48:02,600: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:48:02,647: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:48:02,759: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:48:03,051: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:48:03,954: INFO/job_scheduler] Job 2460 finished 1 steps successfully
[2017-06-01 19:48:03,971: INFO/job_scheduler] Job 2460: complete_job: Updating cache
[2017-06-01 19:48:03,971: INFO/job_scheduler] Job 2460 completing (errored=False, cancelled=False)
[2017-06-01 19:48:04,005: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:48:04,037: INFO/job_scheduler] Job 2459 finished 1 steps successfully
[2017-06-01 19:48:04,146: INFO/job_scheduler] Job 2461 finished 1 steps successfully
[2017-06-01 19:48:04,154: INFO/job_scheduler] Job 2459: complete_job: Updating cache
[2017-06-01 19:48:04,155: INFO/job_scheduler] Job 2459 completing (errored=False, cancelled=False)
[2017-06-01 19:48:04,212: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:48:04,230: INFO/job_scheduler] Job 2461: complete_job: Updating cache
[2017-06-01 19:48:04,230: INFO/job_scheduler] Job 2461 completing (errored=False, cancelled=False)
[2017-06-01 19:48:04,319: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:48:04,469: INFO/job_scheduler] Job 2462 finished 1 steps successfully
[2017-06-01 19:48:04,495: INFO/job_scheduler] Job 2462: complete_job: Updating cache
[2017-06-01 19:48:04,496: INFO/job_scheduler] Job 2462 completing (errored=False, cancelled=False)
[2017-06-01 19:48:04,546: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:48:06,496: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:06,496: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:06,503: INFO/command_plan] add_jobs: created Job 2463 (Test for host connectivity)
[2017-06-01 19:48:06,517: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:48:06,531: INFO/job_scheduler] Job 2463: RunJobThread.run
[2017-06-01 19:48:06,601: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:06,602: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:06,621: INFO/command_plan] add_jobs: created Job 2464 (Test for host connectivity)
[2017-06-01 19:48:06,688: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:48:06,707: INFO/job_scheduler] Job 2464: RunJobThread.run
[2017-06-01 19:48:06,761: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:06,761: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:06,796: INFO/command_plan] add_jobs: created Job 2465 (Test for host connectivity)
[2017-06-01 19:48:06,884: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:48:06,893: INFO/job_scheduler] Job 2465: RunJobThread.run
[2017-06-01 19:48:07,000: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:07,002: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:07,029: INFO/command_plan] add_jobs: created Job 2466 (Test for host connectivity)
[2017-06-01 19:48:07,062: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:48:07,152: INFO/job_scheduler] Job 2466: RunJobThread.run
[2017-06-01 19:48:08,956: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:48:09,039: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:48:09,254: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:48:09,452: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:48:10,281: INFO/job_scheduler] Job 2464 finished 1 steps successfully
[2017-06-01 19:48:10,328: INFO/job_scheduler] Job 2464: complete_job: Updating cache
[2017-06-01 19:48:10,328: INFO/job_scheduler] Job 2464 completing (errored=False, cancelled=False)
[2017-06-01 19:48:10,371: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:48:10,378: INFO/job_scheduler] Job 2463 finished 1 steps successfully
[2017-06-01 19:48:10,429: INFO/job_scheduler] Job 2463: complete_job: Updating cache
[2017-06-01 19:48:10,429: INFO/job_scheduler] Job 2463 completing (errored=False, cancelled=False)
[2017-06-01 19:48:10,530: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:48:10,674: INFO/job_scheduler] Job 2465 finished 1 steps successfully
[2017-06-01 19:48:10,728: INFO/job_scheduler] Job 2465: complete_job: Updating cache
[2017-06-01 19:48:10,729: INFO/job_scheduler] Job 2465 completing (errored=False, cancelled=False)
[2017-06-01 19:48:10,781: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:48:10,830: INFO/job_scheduler] Job 2466 finished 1 steps successfully
[2017-06-01 19:48:10,895: INFO/job_scheduler] Job 2466: complete_job: Updating cache
[2017-06-01 19:48:10,895: INFO/job_scheduler] Job 2466 completing (errored=False, cancelled=False)
[2017-06-01 19:48:10,970: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:48:12,867: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:12,869: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:12,877: INFO/command_plan] add_jobs: created Job 2467 (Test for host connectivity)
[2017-06-01 19:48:12,893: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:48:12,900: INFO/job_scheduler] Job 2467: RunJobThread.run
[2017-06-01 19:48:13,055: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:13,056: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:13,133: INFO/command_plan] add_jobs: created Job 2468 (Test for host connectivity)
[2017-06-01 19:48:13,167: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:48:13,179: INFO/job_scheduler] Job 2468: RunJobThread.run
[2017-06-01 19:48:13,245: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:13,245: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:13,277: INFO/command_plan] add_jobs: created Job 2469 (Test for host connectivity)
[2017-06-01 19:48:13,346: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:48:13,386: INFO/job_scheduler] Job 2469: RunJobThread.run
[2017-06-01 19:48:13,456: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:13,457: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:13,616: INFO/command_plan] add_jobs: created Job 2470 (Test for host connectivity)
[2017-06-01 19:48:13,684: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:48:13,734: INFO/job_scheduler] Job 2470: RunJobThread.run
[2017-06-01 19:48:15,370: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:48:15,706: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:48:15,822: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:48:16,170: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:48:16,698: INFO/job_scheduler] Job 2467 finished 1 steps successfully
[2017-06-01 19:48:16,719: INFO/job_scheduler] Job 2467: complete_job: Updating cache
[2017-06-01 19:48:16,719: INFO/job_scheduler] Job 2467 completing (errored=False, cancelled=False)
[2017-06-01 19:48:16,763: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:48:17,117: INFO/job_scheduler] Job 2468 finished 1 steps successfully
[2017-06-01 19:48:17,152: INFO/job_scheduler] Job 2468: complete_job: Updating cache
[2017-06-01 19:48:17,152: INFO/job_scheduler] Job 2468 completing (errored=False, cancelled=False)
[2017-06-01 19:48:17,233: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:48:17,308: INFO/job_scheduler] Job 2469 finished 1 steps successfully
[2017-06-01 19:48:17,368: INFO/job_scheduler] Job 2469: complete_job: Updating cache
[2017-06-01 19:48:17,368: INFO/job_scheduler] Job 2469 completing (errored=False, cancelled=False)
[2017-06-01 19:48:17,419: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:48:17,578: INFO/job_scheduler] Job 2470 finished 1 steps successfully
[2017-06-01 19:48:17,627: INFO/job_scheduler] Job 2470: complete_job: Updating cache
[2017-06-01 19:48:17,628: INFO/job_scheduler] Job 2470 completing (errored=False, cancelled=False)
[2017-06-01 19:48:17,651: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:48:19,500: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:19,500: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:19,509: INFO/command_plan] add_jobs: created Job 2471 (Test for host connectivity)
[2017-06-01 19:48:19,580: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:48:19,582: INFO/job_scheduler] Job 2471: RunJobThread.run
[2017-06-01 19:48:19,635: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:19,636: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:19,652: INFO/command_plan] add_jobs: created Job 2472 (Test for host connectivity)
[2017-06-01 19:48:19,756: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:48:19,762: INFO/job_scheduler] Job 2472: RunJobThread.run
[2017-06-01 19:48:19,823: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:19,827: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:19,846: INFO/command_plan] add_jobs: created Job 2473 (Test for host connectivity)
[2017-06-01 19:48:19,883: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:48:19,910: INFO/job_scheduler] Job 2473: RunJobThread.run
[2017-06-01 19:48:20,136: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:20,138: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:20,167: INFO/command_plan] add_jobs: created Job 2474 (Test for host connectivity)
[2017-06-01 19:48:20,203: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:48:20,243: INFO/job_scheduler] Job 2474: RunJobThread.run
[2017-06-01 19:48:22,005: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:48:22,278: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:48:22,409: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:48:22,472: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:48:23,316: INFO/job_scheduler] Job 2471 finished 1 steps successfully
[2017-06-01 19:48:23,342: INFO/job_scheduler] Job 2471: complete_job: Updating cache
[2017-06-01 19:48:23,342: INFO/job_scheduler] Job 2471 completing (errored=False, cancelled=False)
[2017-06-01 19:48:23,376: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:48:23,652: INFO/job_scheduler] Job 2472 finished 1 steps successfully
[2017-06-01 19:48:23,700: INFO/job_scheduler] Job 2472: complete_job: Updating cache
[2017-06-01 19:48:23,700: INFO/job_scheduler] Job 2472 completing (errored=False, cancelled=False)
[2017-06-01 19:48:23,767: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:48:23,861: INFO/job_scheduler] Job 2473 finished 1 steps successfully
[2017-06-01 19:48:23,900: INFO/job_scheduler] Job 2473: complete_job: Updating cache
[2017-06-01 19:48:23,900: INFO/job_scheduler] Job 2473 completing (errored=False, cancelled=False)
[2017-06-01 19:48:23,942: INFO/job_scheduler] Job 2474 finished 1 steps successfully
[2017-06-01 19:48:23,971: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:48:24,083: INFO/job_scheduler] Job 2474: complete_job: Updating cache
[2017-06-01 19:48:24,084: INFO/job_scheduler] Job 2474 completing (errored=False, cancelled=False)
[2017-06-01 19:48:24,125: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:48:25,997: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:25,999: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:26,008: INFO/command_plan] add_jobs: created Job 2475 (Test for host connectivity)
[2017-06-01 19:48:26,053: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:26,053: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:26,090: INFO/command_plan] add_jobs: created Job 2476 (Test for host connectivity)
[2017-06-01 19:48:26,156: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:48:26,163: INFO/job_scheduler] Job 2475: RunJobThread.run
[2017-06-01 19:48:26,169: INFO/job_scheduler] Job 2476: RunJobThread.run
[2017-06-01 19:48:26,176: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:48:26,296: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:26,298: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:26,325: INFO/command_plan] add_jobs: created Job 2477 (Test for host connectivity)
[2017-06-01 19:48:26,433: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:48:26,437: INFO/job_scheduler] Job 2477: RunJobThread.run
[2017-06-01 19:48:26,501: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:26,502: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:26,519: INFO/command_plan] add_jobs: created Job 2478 (Test for host connectivity)
[2017-06-01 19:48:26,554: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:48:26,609: INFO/job_scheduler] Job 2478: RunJobThread.run
[2017-06-01 19:48:28,554: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:48:28,621: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:48:28,806: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:48:28,833: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:48:29,898: INFO/job_scheduler] Job 2475 finished 1 steps successfully
[2017-06-01 19:48:29,954: INFO/job_scheduler] Job 2476 finished 1 steps successfully
[2017-06-01 19:48:29,959: INFO/job_scheduler] Job 2475: complete_job: Updating cache
[2017-06-01 19:48:29,959: INFO/job_scheduler] Job 2475 completing (errored=False, cancelled=False)
[2017-06-01 19:48:29,992: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:48:30,007: INFO/job_scheduler] Job 2476: complete_job: Updating cache
[2017-06-01 19:48:30,007: INFO/job_scheduler] Job 2476 completing (errored=False, cancelled=False)
[2017-06-01 19:48:30,134: INFO/job_scheduler] Job 2478 finished 1 steps successfully
[2017-06-01 19:48:30,149: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:48:30,166: INFO/job_scheduler] Job 2478: complete_job: Updating cache
[2017-06-01 19:48:30,167: INFO/job_scheduler] Job 2478 completing (errored=False, cancelled=False)
[2017-06-01 19:48:30,242: INFO/job_scheduler] Job 2477 finished 1 steps successfully
[2017-06-01 19:48:30,298: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:48:30,316: INFO/job_scheduler] Job 2477: complete_job: Updating cache
[2017-06-01 19:48:30,316: INFO/job_scheduler] Job 2477 completing (errored=False, cancelled=False)
[2017-06-01 19:48:30,441: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:48:32,389: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:32,396: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:32,409: INFO/command_plan] add_jobs: created Job 2479 (Test for host connectivity)
[2017-06-01 19:48:32,457: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:48:32,460: INFO/job_scheduler] Job 2479: RunJobThread.run
[2017-06-01 19:48:32,523: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:32,523: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:32,532: INFO/command_plan] add_jobs: created Job 2480 (Test for host connectivity)
[2017-06-01 19:48:32,574: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:48:32,616: INFO/job_scheduler] Job 2480: RunJobThread.run
[2017-06-01 19:48:32,715: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:32,716: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:32,761: INFO/command_plan] add_jobs: created Job 2481 (Test for host connectivity)
[2017-06-01 19:48:32,931: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:32,932: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:32,972: INFO/command_plan] add_jobs: created Job 2482 (Test for host connectivity)
[2017-06-01 19:48:33,008: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:48:33,029: INFO/job_scheduler] Job 2481: RunJobThread.run
[2017-06-01 19:48:33,051: INFO/job_scheduler] Job 2482: RunJobThread.run
[2017-06-01 19:48:33,069: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:48:34,822: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:48:35,005: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:48:35,369: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:48:35,419: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:48:36,189: INFO/job_scheduler] Job 2479 finished 1 steps successfully
[2017-06-01 19:48:36,214: INFO/job_scheduler] Job 2479: complete_job: Updating cache
[2017-06-01 19:48:36,214: INFO/job_scheduler] Job 2479 completing (errored=False, cancelled=False)
[2017-06-01 19:48:36,264: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:48:36,391: INFO/job_scheduler] Job 2480 finished 1 steps successfully
[2017-06-01 19:48:36,443: INFO/job_scheduler] Job 2480: complete_job: Updating cache
[2017-06-01 19:48:36,445: INFO/job_scheduler] Job 2480 completing (errored=False, cancelled=False)
[2017-06-01 19:48:36,550: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:48:36,597: INFO/job_scheduler] Job 2482 finished 1 steps successfully
[2017-06-01 19:48:36,607: INFO/job_scheduler] Job 2482: complete_job: Updating cache
[2017-06-01 19:48:36,608: INFO/job_scheduler] Job 2482 completing (errored=False, cancelled=False)
[2017-06-01 19:48:36,664: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:48:36,883: INFO/job_scheduler] Job 2481 finished 1 steps successfully
[2017-06-01 19:48:36,947: INFO/job_scheduler] Job 2481: complete_job: Updating cache
[2017-06-01 19:48:36,947: INFO/job_scheduler] Job 2481 completing (errored=False, cancelled=False)
[2017-06-01 19:48:36,988: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:48:38,948: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:38,949: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:38,972: INFO/command_plan] add_jobs: created Job 2483 (Test for host connectivity)
[2017-06-01 19:48:38,984: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:48:39,082: INFO/job_scheduler] Job 2483: RunJobThread.run
[2017-06-01 19:48:39,169: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:39,170: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:39,257: INFO/command_plan] add_jobs: created Job 2484 (Test for host connectivity)
[2017-06-01 19:48:39,309: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:48:39,316: INFO/job_scheduler] Job 2484: RunJobThread.run
[2017-06-01 19:48:39,436: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:39,438: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:39,464: INFO/command_plan] add_jobs: created Job 2485 (Test for host connectivity)
[2017-06-01 19:48:39,632: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:39,632: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:39,666: INFO/command_plan] add_jobs: created Job 2486 (Test for host connectivity)
[2017-06-01 19:48:39,705: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:48:39,744: INFO/job_scheduler] Job 2485: RunJobThread.run
[2017-06-01 19:48:39,784: INFO/job_scheduler] Job 2486: RunJobThread.run
[2017-06-01 19:48:39,801: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:48:41,511: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:48:41,728: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:48:42,084: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:48:42,166: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:48:42,846: INFO/job_scheduler] Job 2483 finished 1 steps successfully
[2017-06-01 19:48:42,863: INFO/job_scheduler] Job 2483: complete_job: Updating cache
[2017-06-01 19:48:42,863: INFO/job_scheduler] Job 2483 completing (errored=False, cancelled=False)
[2017-06-01 19:48:42,923: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:48:43,109: INFO/job_scheduler] Job 2484 finished 1 steps successfully
[2017-06-01 19:48:43,146: INFO/job_scheduler] Job 2484: complete_job: Updating cache
[2017-06-01 19:48:43,147: INFO/job_scheduler] Job 2484 completing (errored=False, cancelled=False)
[2017-06-01 19:48:43,231: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:48:43,537: INFO/job_scheduler] Job 2486 finished 1 steps successfully
[2017-06-01 19:48:43,579: INFO/job_scheduler] Job 2486: complete_job: Updating cache
[2017-06-01 19:48:43,579: INFO/job_scheduler] Job 2486 completing (errored=False, cancelled=False)
[2017-06-01 19:48:43,626: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:48:43,676: INFO/job_scheduler] Job 2485 finished 1 steps successfully
[2017-06-01 19:48:43,725: INFO/job_scheduler] Job 2485: complete_job: Updating cache
[2017-06-01 19:48:43,725: INFO/job_scheduler] Job 2485 completing (errored=False, cancelled=False)
[2017-06-01 19:48:43,788: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:48:45,619: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:45,619: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:45,629: INFO/command_plan] add_jobs: created Job 2487 (Test for host connectivity)
[2017-06-01 19:48:45,659: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:48:45,709: INFO/job_scheduler] Job 2487: RunJobThread.run
[2017-06-01 19:48:45,770: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:45,771: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:45,792: INFO/command_plan] add_jobs: created Job 2488 (Test for host connectivity)
[2017-06-01 19:48:45,831: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:48:45,869: INFO/job_scheduler] Job 2488: RunJobThread.run
[2017-06-01 19:48:45,939: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:45,939: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:45,955: INFO/command_plan] add_jobs: created Job 2489 (Test for host connectivity)
[2017-06-01 19:48:45,971: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:48:46,010: INFO/job_scheduler] Job 2489: RunJobThread.run
[2017-06-01 19:48:46,138: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:46,138: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:46,155: INFO/command_plan] add_jobs: created Job 2490 (Test for host connectivity)
[2017-06-01 19:48:46,252: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:48:46,303: INFO/job_scheduler] Job 2490: RunJobThread.run
[2017-06-01 19:48:47,978: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:48:48,289: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:48:48,374: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:48:48,535: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:48:49,306: INFO/job_scheduler] Job 2487 finished 1 steps successfully
[2017-06-01 19:48:49,345: INFO/job_scheduler] Job 2487: complete_job: Updating cache
[2017-06-01 19:48:49,345: INFO/job_scheduler] Job 2487 completing (errored=False, cancelled=False)
[2017-06-01 19:48:49,406: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:48:49,688: INFO/job_scheduler] Job 2488 finished 1 steps successfully
[2017-06-01 19:48:49,753: INFO/job_scheduler] Job 2488: complete_job: Updating cache
[2017-06-01 19:48:49,754: INFO/job_scheduler] Job 2488 completing (errored=False, cancelled=False)
[2017-06-01 19:48:49,812: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:48:49,826: INFO/job_scheduler] Job 2489 finished 1 steps successfully
[2017-06-01 19:48:49,859: INFO/job_scheduler] Job 2489: complete_job: Updating cache
[2017-06-01 19:48:49,859: INFO/job_scheduler] Job 2489 completing (errored=False, cancelled=False)
[2017-06-01 19:48:49,866: INFO/job_scheduler] Job 2490 finished 1 steps successfully
[2017-06-01 19:48:49,903: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:48:49,914: INFO/job_scheduler] Job 2490: complete_job: Updating cache
[2017-06-01 19:48:49,915: INFO/job_scheduler] Job 2490 completing (errored=False, cancelled=False)
[2017-06-01 19:48:49,989: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:48:52,063: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:52,065: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:52,078: INFO/command_plan] add_jobs: created Job 2491 (Test for host connectivity)
[2017-06-01 19:48:52,108: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:48:52,110: INFO/job_scheduler] Job 2491: RunJobThread.run
[2017-06-01 19:48:52,209: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:52,210: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:52,228: INFO/command_plan] add_jobs: created Job 2492 (Test for host connectivity)
[2017-06-01 19:48:52,325: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:48:52,344: INFO/job_scheduler] Job 2492: RunJobThread.run
[2017-06-01 19:48:52,450: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:52,450: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:52,463: INFO/command_plan] add_jobs: created Job 2493 (Test for host connectivity)
[2017-06-01 19:48:52,480: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:48:52,503: INFO/job_scheduler] Job 2493: RunJobThread.run
[2017-06-01 19:48:52,606: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:52,607: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:52,634: INFO/command_plan] add_jobs: created Job 2494 (Test for host connectivity)
[2017-06-01 19:48:52,662: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:48:52,738: INFO/job_scheduler] Job 2494: RunJobThread.run
[2017-06-01 19:48:54,577: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:48:54,837: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:48:54,932: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:48:55,093: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:48:55,926: INFO/job_scheduler] Job 2491 finished 1 steps successfully
[2017-06-01 19:48:55,968: INFO/job_scheduler] Job 2491: complete_job: Updating cache
[2017-06-01 19:48:55,969: INFO/job_scheduler] Job 2491 completing (errored=False, cancelled=False)
[2017-06-01 19:48:56,020: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:48:56,239: INFO/job_scheduler] Job 2492 finished 1 steps successfully
[2017-06-01 19:48:56,310: INFO/job_scheduler] Job 2492: complete_job: Updating cache
[2017-06-01 19:48:56,314: INFO/job_scheduler] Job 2492 completing (errored=False, cancelled=False)
[2017-06-01 19:48:56,378: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:48:56,386: INFO/job_scheduler] Job 2493 finished 1 steps successfully
[2017-06-01 19:48:56,409: INFO/job_scheduler] Job 2493: complete_job: Updating cache
[2017-06-01 19:48:56,409: INFO/job_scheduler] Job 2493 completing (errored=False, cancelled=False)
[2017-06-01 19:48:56,503: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:48:56,554: INFO/job_scheduler] Job 2494 finished 1 steps successfully
[2017-06-01 19:48:56,605: INFO/job_scheduler] Job 2494: complete_job: Updating cache
[2017-06-01 19:48:56,605: INFO/job_scheduler] Job 2494 completing (errored=False, cancelled=False)
[2017-06-01 19:48:56,668: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:48:58,667: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:58,668: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:58,693: INFO/command_plan] add_jobs: created Job 2495 (Test for host connectivity)
[2017-06-01 19:48:58,816: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:58,816: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:58,841: INFO/command_plan] add_jobs: created Job 2496 (Test for host connectivity)
[2017-06-01 19:48:58,859: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:48:58,911: INFO/job_scheduler] Job 2496: RunJobThread.run
[2017-06-01 19:48:58,918: INFO/job_scheduler] Job 2495: RunJobThread.run
[2017-06-01 19:48:58,959: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:58,959: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:58,985: INFO/command_plan] add_jobs: created Job 2497 (Test for host connectivity)
[2017-06-01 19:48:59,006: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:48:59,020: INFO/job_scheduler] Job 2497: RunJobThread.run
[2017-06-01 19:48:59,075: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:48:59,256: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:48:59,256: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:48:59,325: INFO/command_plan] add_jobs: created Job 2498 (Test for host connectivity)
[2017-06-01 19:48:59,363: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:48:59,393: INFO/job_scheduler] Job 2498: RunJobThread.run
[2017-06-01 19:49:01,353: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:49:01,411: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:49:01,439: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:49:01,664: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:49:02,702: INFO/job_scheduler] Job 2495 finished 1 steps successfully
[2017-06-01 19:49:02,759: INFO/job_scheduler] Job 2495: complete_job: Updating cache
[2017-06-01 19:49:02,759: INFO/job_scheduler] Job 2495 completing (errored=False, cancelled=False)
[2017-06-01 19:49:02,792: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:49:02,810: INFO/job_scheduler] Job 2497 finished 1 steps successfully
[2017-06-01 19:49:02,851: INFO/job_scheduler] Job 2496 finished 1 steps successfully
[2017-06-01 19:49:02,851: INFO/job_scheduler] Job 2497: complete_job: Updating cache
[2017-06-01 19:49:02,852: INFO/job_scheduler] Job 2497 completing (errored=False, cancelled=False)
[2017-06-01 19:49:02,913: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:49:02,924: INFO/job_scheduler] Job 2496: complete_job: Updating cache
[2017-06-01 19:49:02,924: INFO/job_scheduler] Job 2496 completing (errored=False, cancelled=False)
[2017-06-01 19:49:02,944: INFO/job_scheduler] Job 2498 finished 1 steps successfully
[2017-06-01 19:49:03,011: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:49:03,024: INFO/job_scheduler] Job 2498: complete_job: Updating cache
[2017-06-01 19:49:03,024: INFO/job_scheduler] Job 2498 completing (errored=False, cancelled=False)
[2017-06-01 19:49:03,101: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:49:05,268: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:05,269: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:05,293: INFO/command_plan] add_jobs: created Job 2499 (Test for host connectivity)
[2017-06-01 19:49:05,391: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:49:05,415: INFO/job_scheduler] Job 2499: RunJobThread.run
[2017-06-01 19:49:05,483: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:05,485: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:05,507: INFO/command_plan] add_jobs: created Job 2500 (Test for host connectivity)
[2017-06-01 19:49:05,581: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:49:05,643: INFO/job_scheduler] Job 2500: RunJobThread.run
[2017-06-01 19:49:05,686: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:05,686: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:05,719: INFO/command_plan] add_jobs: created Job 2501 (Test for host connectivity)
[2017-06-01 19:49:05,761: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:49:05,819: INFO/job_scheduler] Job 2501: RunJobThread.run
[2017-06-01 19:49:05,963: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:05,965: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:06,064: INFO/command_plan] add_jobs: created Job 2502 (Test for host connectivity)
[2017-06-01 19:49:06,127: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:49:06,188: INFO/job_scheduler] Job 2502: RunJobThread.run
[2017-06-01 19:49:07,774: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:49:08,095: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:49:08,280: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:49:08,372: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:49:09,097: INFO/job_scheduler] Job 2499 finished 1 steps successfully
[2017-06-01 19:49:09,154: INFO/job_scheduler] Job 2499: complete_job: Updating cache
[2017-06-01 19:49:09,154: INFO/job_scheduler] Job 2499 completing (errored=False, cancelled=False)
[2017-06-01 19:49:09,249: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:49:09,466: INFO/job_scheduler] Job 2500 finished 1 steps successfully
[2017-06-01 19:49:09,506: INFO/job_scheduler] Job 2500: complete_job: Updating cache
[2017-06-01 19:49:09,508: INFO/job_scheduler] Job 2500 completing (errored=False, cancelled=False)
[2017-06-01 19:49:09,572: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:49:09,770: INFO/job_scheduler] Job 2501 finished 1 steps successfully
[2017-06-01 19:49:09,783: INFO/job_scheduler] Job 2502 finished 1 steps successfully
[2017-06-01 19:49:09,831: INFO/job_scheduler] Job 2501: complete_job: Updating cache
[2017-06-01 19:49:09,831: INFO/job_scheduler] Job 2501 completing (errored=False, cancelled=False)
[2017-06-01 19:49:09,889: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:49:09,910: INFO/job_scheduler] Job 2502: complete_job: Updating cache
[2017-06-01 19:49:09,910: INFO/job_scheduler] Job 2502 completing (errored=False, cancelled=False)
[2017-06-01 19:49:10,031: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:49:12,714: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:12,716: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:12,739: INFO/command_plan] add_jobs: created Job 2503 (Test for host connectivity)
[2017-06-01 19:49:12,763: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:49:12,845: INFO/job_scheduler] Job 2503: RunJobThread.run
[2017-06-01 19:49:12,939: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:12,940: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:12,959: INFO/command_plan] add_jobs: created Job 2504 (Test for host connectivity)
[2017-06-01 19:49:13,052: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:49:13,083: INFO/job_scheduler] Job 2504: RunJobThread.run
[2017-06-01 19:49:13,121: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:13,121: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:13,141: INFO/command_plan] add_jobs: created Job 2505 (Test for host connectivity)
[2017-06-01 19:49:13,289: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:13,289: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:13,313: INFO/command_plan] add_jobs: created Job 2506 (Test for host connectivity)
[2017-06-01 19:49:13,332: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:49:13,335: INFO/job_scheduler] Job 2505: RunJobThread.run
[2017-06-01 19:49:13,336: INFO/job_scheduler] Job 2506: RunJobThread.run
[2017-06-01 19:49:13,356: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:49:15,278: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:49:15,382: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:49:15,641: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:49:15,679: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:49:16,656: INFO/job_scheduler] Job 2503 finished 1 steps successfully
[2017-06-01 19:49:16,701: INFO/job_scheduler] Job 2503: complete_job: Updating cache
[2017-06-01 19:49:16,701: INFO/job_scheduler] Job 2503 completing (errored=False, cancelled=False)
[2017-06-01 19:49:16,777: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:49:16,785: INFO/job_scheduler] Job 2504 finished 1 steps successfully
[2017-06-01 19:49:16,821: INFO/job_scheduler] Job 2504: complete_job: Updating cache
[2017-06-01 19:49:16,821: INFO/job_scheduler] Job 2504 completing (errored=False, cancelled=False)
[2017-06-01 19:49:16,957: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:49:17,086: INFO/job_scheduler] Job 2505 finished 1 steps successfully
[2017-06-01 19:49:17,089: INFO/job_scheduler] Job 2506 finished 1 steps successfully
[2017-06-01 19:49:17,129: INFO/job_scheduler] Job 2505: complete_job: Updating cache
[2017-06-01 19:49:17,130: INFO/job_scheduler] Job 2505 completing (errored=False, cancelled=False)
[2017-06-01 19:49:17,247: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:49:17,275: INFO/job_scheduler] Job 2506: complete_job: Updating cache
[2017-06-01 19:49:17,275: INFO/job_scheduler] Job 2506 completing (errored=False, cancelled=False)
[2017-06-01 19:49:17,356: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:49:19,305: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:19,306: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:19,339: INFO/command_plan] add_jobs: created Job 2507 (Test for host connectivity)
[2017-06-01 19:49:19,357: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:49:19,382: INFO/job_scheduler] Job 2507: RunJobThread.run
[2017-06-01 19:49:19,516: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:19,517: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:19,537: INFO/command_plan] add_jobs: created Job 2508 (Test for host connectivity)
[2017-06-01 19:49:19,570: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:49:19,609: INFO/job_scheduler] Job 2508: RunJobThread.run
[2017-06-01 19:49:19,697: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:19,700: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:19,712: INFO/command_plan] add_jobs: created Job 2509 (Test for host connectivity)
[2017-06-01 19:49:19,737: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:49:19,792: INFO/job_scheduler] Job 2509: RunJobThread.run
[2017-06-01 19:49:19,867: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:19,867: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:19,894: INFO/command_plan] add_jobs: created Job 2510 (Test for host connectivity)
[2017-06-01 19:49:19,915: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:49:19,931: INFO/job_scheduler] Job 2510: RunJobThread.run
[2017-06-01 19:49:21,876: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:49:21,958: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:49:22,092: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:49:22,202: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:49:23,196: INFO/job_scheduler] Job 2507 finished 1 steps successfully
[2017-06-01 19:49:23,208: INFO/job_scheduler] Job 2508 finished 1 steps successfully
[2017-06-01 19:49:23,228: INFO/job_scheduler] Job 2507: complete_job: Updating cache
[2017-06-01 19:49:23,228: INFO/job_scheduler] Job 2507 completing (errored=False, cancelled=False)
[2017-06-01 19:49:23,279: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:49:23,318: INFO/job_scheduler] Job 2508: complete_job: Updating cache
[2017-06-01 19:49:23,319: INFO/job_scheduler] Job 2508 completing (errored=False, cancelled=False)
[2017-06-01 19:49:23,471: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:49:23,493: INFO/job_scheduler] Job 2509 finished 1 steps successfully
[2017-06-01 19:49:23,504: INFO/job_scheduler] Job 2509: complete_job: Updating cache
[2017-06-01 19:49:23,504: INFO/job_scheduler] Job 2509 completing (errored=False, cancelled=False)
[2017-06-01 19:49:23,640: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:49:23,674: INFO/job_scheduler] Job 2510 finished 1 steps successfully
[2017-06-01 19:49:23,695: INFO/job_scheduler] Job 2510: complete_job: Updating cache
[2017-06-01 19:49:23,695: INFO/job_scheduler] Job 2510 completing (errored=False, cancelled=False)
[2017-06-01 19:49:23,753: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:49:25,806: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:25,808: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:25,828: INFO/command_plan] add_jobs: created Job 2511 (Test for host connectivity)
[2017-06-01 19:49:25,896: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:49:25,924: INFO/job_scheduler] Job 2511: RunJobThread.run
[2017-06-01 19:49:25,976: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:25,977: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:25,989: INFO/command_plan] add_jobs: created Job 2512 (Test for host connectivity)
[2017-06-01 19:49:26,034: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:49:26,039: INFO/job_scheduler] Job 2512: RunJobThread.run
[2017-06-01 19:49:26,187: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:26,188: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:26,345: INFO/command_plan] add_jobs: created Job 2513 (Test for host connectivity)
[2017-06-01 19:49:26,405: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:49:26,436: INFO/job_scheduler] Job 2513: RunJobThread.run
[2017-06-01 19:49:26,554: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:26,554: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:26,586: INFO/command_plan] add_jobs: created Job 2514 (Test for host connectivity)
[2017-06-01 19:49:26,679: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:49:26,717: INFO/job_scheduler] Job 2514: RunJobThread.run
[2017-06-01 19:49:28,446: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:49:28,465: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:49:28,818: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:49:28,959: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:49:29,858: INFO/job_scheduler] Job 2511 finished 1 steps successfully
[2017-06-01 19:49:29,886: INFO/job_scheduler] Job 2512 finished 1 steps successfully
[2017-06-01 19:49:29,911: INFO/job_scheduler] Job 2511: complete_job: Updating cache
[2017-06-01 19:49:29,911: INFO/job_scheduler] Job 2511 completing (errored=False, cancelled=False)
[2017-06-01 19:49:29,975: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:49:29,994: INFO/job_scheduler] Job 2512: complete_job: Updating cache
[2017-06-01 19:49:29,999: INFO/job_scheduler] Job 2512 completing (errored=False, cancelled=False)
[2017-06-01 19:49:30,136: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:49:30,260: INFO/job_scheduler] Job 2513 finished 1 steps successfully
[2017-06-01 19:49:30,326: INFO/job_scheduler] Job 2513: complete_job: Updating cache
[2017-06-01 19:49:30,326: INFO/job_scheduler] Job 2513 completing (errored=False, cancelled=False)
[2017-06-01 19:49:30,377: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:49:30,483: INFO/job_scheduler] Job 2514 finished 1 steps successfully
[2017-06-01 19:49:30,506: INFO/job_scheduler] Job 2514: complete_job: Updating cache
[2017-06-01 19:49:30,506: INFO/job_scheduler] Job 2514 completing (errored=False, cancelled=False)
[2017-06-01 19:49:30,555: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:49:31,528: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:31,530: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:31,557: INFO/command_plan] add_jobs: created Job 2515 (Test for host connectivity)
[2017-06-01 19:49:31,585: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:49:31,606: INFO/job_scheduler] Job 2515: RunJobThread.run
[2017-06-01 19:49:31,689: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:31,690: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:31,735: INFO/command_plan] add_jobs: created Job 2516 (Test for host connectivity)
[2017-06-01 19:49:31,775: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:49:31,790: INFO/job_scheduler] Job 2516: RunJobThread.run
[2017-06-01 19:49:31,886: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:31,887: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:31,901: INFO/command_plan] add_jobs: created Job 2517 (Test for host connectivity)
[2017-06-01 19:49:31,936: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:49:32,019: INFO/job_scheduler] Job 2517: RunJobThread.run
[2017-06-01 19:49:32,097: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:32,098: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:32,120: INFO/command_plan] add_jobs: created Job 2518 (Test for host connectivity)
[2017-06-01 19:49:32,193: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:49:32,236: INFO/job_scheduler] Job 2518: RunJobThread.run
[2017-06-01 19:49:34,007: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:49:34,122: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:49:34,404: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:49:34,460: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:49:35,328: INFO/job_scheduler] Job 2515 finished 1 steps successfully
[2017-06-01 19:49:35,350: INFO/job_scheduler] Job 2515: complete_job: Updating cache
[2017-06-01 19:49:35,351: INFO/job_scheduler] Job 2515 completing (errored=False, cancelled=False)
[2017-06-01 19:49:35,375: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:49:35,449: INFO/job_scheduler] Job 2516 finished 1 steps successfully
[2017-06-01 19:49:35,480: INFO/job_scheduler] Job 2516: complete_job: Updating cache
[2017-06-01 19:49:35,481: INFO/job_scheduler] Job 2516 completing (errored=False, cancelled=False)
[2017-06-01 19:49:35,551: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:49:35,914: INFO/job_scheduler] Job 2518 finished 1 steps successfully
[2017-06-01 19:49:35,915: INFO/job_scheduler] Job 2517 finished 1 steps successfully
[2017-06-01 19:49:35,967: INFO/job_scheduler] Job 2518: complete_job: Updating cache
[2017-06-01 19:49:35,967: INFO/job_scheduler] Job 2518 completing (errored=False, cancelled=False)
[2017-06-01 19:49:36,026: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:49:36,030: INFO/job_scheduler] Job 2517: complete_job: Updating cache
[2017-06-01 19:49:36,033: INFO/job_scheduler] Job 2517 completing (errored=False, cancelled=False)
[2017-06-01 19:49:36,177: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:49:38,051: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:38,051: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:38,067: INFO/command_plan] add_jobs: created Job 2519 (Test for host connectivity)
[2017-06-01 19:49:38,106: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:49:38,134: INFO/job_scheduler] Job 2519: RunJobThread.run
[2017-06-01 19:49:38,222: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:38,223: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:38,289: INFO/command_plan] add_jobs: created Job 2520 (Test for host connectivity)
[2017-06-01 19:49:38,370: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:49:38,395: INFO/job_scheduler] Job 2520: RunJobThread.run
[2017-06-01 19:49:38,495: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:38,502: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:38,535: INFO/command_plan] add_jobs: created Job 2521 (Test for host connectivity)
[2017-06-01 19:49:38,585: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:49:38,626: INFO/job_scheduler] Job 2521: RunJobThread.run
[2017-06-01 19:49:38,675: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:38,675: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:38,807: INFO/command_plan] add_jobs: created Job 2522 (Test for host connectivity)
[2017-06-01 19:49:38,876: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:49:38,880: INFO/job_scheduler] Job 2522: RunJobThread.run
[2017-06-01 19:49:40,517: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:49:40,883: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:49:40,920: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:49:41,152: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:49:41,847: INFO/job_scheduler] Job 2519 finished 1 steps successfully
[2017-06-01 19:49:41,916: INFO/job_scheduler] Job 2519: complete_job: Updating cache
[2017-06-01 19:49:41,916: INFO/job_scheduler] Job 2519 completing (errored=False, cancelled=False)
[2017-06-01 19:49:42,048: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:49:42,388: INFO/job_scheduler] Job 2520 finished 1 steps successfully
[2017-06-01 19:49:42,450: INFO/job_scheduler] Job 2520: complete_job: Updating cache
[2017-06-01 19:49:42,452: INFO/job_scheduler] Job 2520 completing (errored=False, cancelled=False)
[2017-06-01 19:49:42,518: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:49:42,555: INFO/job_scheduler] Job 2521 finished 1 steps successfully
[2017-06-01 19:49:42,575: INFO/job_scheduler] Job 2521: complete_job: Updating cache
[2017-06-01 19:49:42,581: INFO/job_scheduler] Job 2521 completing (errored=False, cancelled=False)
[2017-06-01 19:49:42,647: INFO/job_scheduler] Job 2522 finished 1 steps successfully
[2017-06-01 19:49:42,677: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:49:42,742: INFO/job_scheduler] Job 2522: complete_job: Updating cache
[2017-06-01 19:49:42,742: INFO/job_scheduler] Job 2522 completing (errored=False, cancelled=False)
[2017-06-01 19:49:42,783: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:49:43,589: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:43,591: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:43,635: INFO/command_plan] add_jobs: created Job 2523 (Test for host connectivity)
[2017-06-01 19:49:43,698: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:49:43,750: INFO/job_scheduler] Job 2523: RunJobThread.run
[2017-06-01 19:49:43,806: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:43,807: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:43,833: INFO/command_plan] add_jobs: created Job 2524 (Test for host connectivity)
[2017-06-01 19:49:43,928: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:43,928: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:43,942: INFO/command_plan] add_jobs: created Job 2525 (Test for host connectivity)
[2017-06-01 19:49:43,959: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:49:44,008: INFO/job_scheduler] Job 2524: RunJobThread.run
[2017-06-01 19:49:44,019: INFO/job_scheduler] Job 2525: RunJobThread.run
[2017-06-01 19:49:44,026: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:49:44,077: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:44,078: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:44,103: INFO/command_plan] add_jobs: created Job 2526 (Test for host connectivity)
[2017-06-01 19:49:44,174: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:49:44,246: INFO/job_scheduler] Job 2526: RunJobThread.run
[2017-06-01 19:49:46,169: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:49:46,319: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:49:46,499: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:49:46,662: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:49:47,501: INFO/job_scheduler] Job 2523 finished 1 steps successfully
[2017-06-01 19:49:47,531: INFO/job_scheduler] Job 2523: complete_job: Updating cache
[2017-06-01 19:49:47,531: INFO/job_scheduler] Job 2523 completing (errored=False, cancelled=False)
[2017-06-01 19:49:47,589: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:49:47,723: INFO/job_scheduler] Job 2524 finished 1 steps successfully
[2017-06-01 19:49:47,765: INFO/job_scheduler] Job 2524: complete_job: Updating cache
[2017-06-01 19:49:47,765: INFO/job_scheduler] Job 2524 completing (errored=False, cancelled=False)
[2017-06-01 19:49:47,840: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:49:47,966: INFO/job_scheduler] Job 2525 finished 1 steps successfully
[2017-06-01 19:49:48,015: INFO/job_scheduler] Job 2525: complete_job: Updating cache
[2017-06-01 19:49:48,015: INFO/job_scheduler] Job 2525 completing (errored=False, cancelled=False)
[2017-06-01 19:49:48,104: INFO/job_scheduler] Job 2526 finished 1 steps successfully
[2017-06-01 19:49:48,104: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:49:48,132: INFO/job_scheduler] Job 2526: complete_job: Updating cache
[2017-06-01 19:49:48,133: INFO/job_scheduler] Job 2526 completing (errored=False, cancelled=False)
[2017-06-01 19:49:48,249: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:49:50,131: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:50,132: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:50,173: INFO/command_plan] add_jobs: created Job 2527 (Test for host connectivity)
[2017-06-01 19:49:50,245: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:49:50,253: INFO/job_scheduler] Job 2527: RunJobThread.run
[2017-06-01 19:49:50,354: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:50,354: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:50,364: INFO/command_plan] add_jobs: created Job 2528 (Test for host connectivity)
[2017-06-01 19:49:50,400: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:49:50,431: INFO/job_scheduler] Job 2528: RunJobThread.run
[2017-06-01 19:49:50,566: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:50,567: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:50,584: INFO/command_plan] add_jobs: created Job 2529 (Test for host connectivity)
[2017-06-01 19:49:50,605: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:49:50,671: INFO/job_scheduler] Job 2529: RunJobThread.run
[2017-06-01 19:49:50,738: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:50,738: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:50,762: INFO/command_plan] add_jobs: created Job 2530 (Test for host connectivity)
[2017-06-01 19:49:50,885: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:49:50,896: INFO/job_scheduler] Job 2530: RunJobThread.run
[2017-06-01 19:49:52,713: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:49:52,883: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:49:53,012: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:49:53,203: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:49:54,055: INFO/job_scheduler] Job 2527 finished 1 steps successfully
[2017-06-01 19:49:54,138: INFO/job_scheduler] Job 2527: complete_job: Updating cache
[2017-06-01 19:49:54,139: INFO/job_scheduler] Job 2527 completing (errored=False, cancelled=False)
[2017-06-01 19:49:54,201: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:49:54,265: INFO/job_scheduler] Job 2528 finished 1 steps successfully
[2017-06-01 19:49:54,353: INFO/job_scheduler] Job 2528: complete_job: Updating cache
[2017-06-01 19:49:54,354: INFO/job_scheduler] Job 2528 completing (errored=False, cancelled=False)
[2017-06-01 19:49:54,392: INFO/job_scheduler] Job 2529 finished 1 steps successfully
[2017-06-01 19:49:54,421: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:49:54,434: INFO/job_scheduler] Job 2529: complete_job: Updating cache
[2017-06-01 19:49:54,437: INFO/job_scheduler] Job 2529 completing (errored=False, cancelled=False)
[2017-06-01 19:49:54,536: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:49:54,622: INFO/job_scheduler] Job 2530 finished 1 steps successfully
[2017-06-01 19:49:54,670: INFO/job_scheduler] Job 2530: complete_job: Updating cache
[2017-06-01 19:49:54,670: INFO/job_scheduler] Job 2530 completing (errored=False, cancelled=False)
[2017-06-01 19:49:54,767: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:49:56,819: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:56,820: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:56,862: INFO/command_plan] add_jobs: created Job 2531 (Test for host connectivity)
[2017-06-01 19:49:56,897: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:49:56,903: INFO/job_scheduler] Job 2531: RunJobThread.run
[2017-06-01 19:49:56,964: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:56,965: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:56,983: INFO/command_plan] add_jobs: created Job 2532 (Test for host connectivity)
[2017-06-01 19:49:57,124: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:49:57,139: INFO/job_scheduler] Job 2532: RunJobThread.run
[2017-06-01 19:49:57,202: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:57,205: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:57,227: INFO/command_plan] add_jobs: created Job 2533 (Test for host connectivity)
[2017-06-01 19:49:57,317: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:49:57,357: INFO/job_scheduler] Job 2533: RunJobThread.run
[2017-06-01 19:49:57,417: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:49:57,417: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:49:57,440: INFO/command_plan] add_jobs: created Job 2534 (Test for host connectivity)
[2017-06-01 19:49:57,560: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:49:57,586: INFO/job_scheduler] Job 2534: RunJobThread.run
[2017-06-01 19:49:59,388: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:49:59,585: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:49:59,730: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:49:59,971: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:50:00,707: INFO/job_scheduler] Job 2531 finished 1 steps successfully
[2017-06-01 19:50:00,744: INFO/job_scheduler] Job 2531: complete_job: Updating cache
[2017-06-01 19:50:00,745: INFO/job_scheduler] Job 2531 completing (errored=False, cancelled=False)
[2017-06-01 19:50:00,789: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:50:01,010: INFO/job_scheduler] Job 2532 finished 1 steps successfully
[2017-06-01 19:50:01,053: INFO/job_scheduler] Job 2532: complete_job: Updating cache
[2017-06-01 19:50:01,058: INFO/job_scheduler] Job 2532 completing (errored=False, cancelled=False)
[2017-06-01 19:50:01,094: INFO/job_scheduler] Job 2533 finished 1 steps successfully
[2017-06-01 19:50:01,154: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:50:01,174: INFO/job_scheduler] Job 2533: complete_job: Updating cache
[2017-06-01 19:50:01,174: INFO/job_scheduler] Job 2533 completing (errored=False, cancelled=False)
[2017-06-01 19:50:01,287: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:50:01,343: INFO/job_scheduler] Job 2534 finished 1 steps successfully
[2017-06-01 19:50:01,361: INFO/job_scheduler] Job 2534: complete_job: Updating cache
[2017-06-01 19:50:01,362: INFO/job_scheduler] Job 2534 completing (errored=False, cancelled=False)
[2017-06-01 19:50:01,404: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:50:03,490: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:03,490: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:03,519: INFO/command_plan] add_jobs: created Job 2535 (Test for host connectivity)
[2017-06-01 19:50:03,643: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:50:03,649: INFO/job_scheduler] Job 2535: RunJobThread.run
[2017-06-01 19:50:03,675: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:03,679: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:03,720: INFO/command_plan] add_jobs: created Job 2536 (Test for host connectivity)
[2017-06-01 19:50:03,760: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:50:03,808: INFO/job_scheduler] Job 2536: RunJobThread.run
[2017-06-01 19:50:03,870: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:03,871: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:03,920: INFO/command_plan] add_jobs: created Job 2537 (Test for host connectivity)
[2017-06-01 19:50:03,974: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:50:04,025: INFO/job_scheduler] Job 2537: RunJobThread.run
[2017-06-01 19:50:04,083: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:04,084: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:04,168: INFO/command_plan] add_jobs: created Job 2538 (Test for host connectivity)
[2017-06-01 19:50:04,234: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:50:04,281: INFO/job_scheduler] Job 2538: RunJobThread.run
[2017-06-01 19:50:06,070: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:50:06,185: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:50:06,349: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:50:06,593: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:50:07,383: INFO/job_scheduler] Job 2535 finished 1 steps successfully
[2017-06-01 19:50:07,402: INFO/job_scheduler] Job 2535: complete_job: Updating cache
[2017-06-01 19:50:07,402: INFO/job_scheduler] Job 2535 completing (errored=False, cancelled=False)
[2017-06-01 19:50:07,470: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:50:07,530: INFO/job_scheduler] Job 2536 finished 1 steps successfully
[2017-06-01 19:50:07,570: INFO/job_scheduler] Job 2536: complete_job: Updating cache
[2017-06-01 19:50:07,570: INFO/job_scheduler] Job 2536 completing (errored=False, cancelled=False)
[2017-06-01 19:50:07,639: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:50:07,737: INFO/job_scheduler] Job 2537 finished 1 steps successfully
[2017-06-01 19:50:07,788: INFO/job_scheduler] Job 2537: complete_job: Updating cache
[2017-06-01 19:50:07,789: INFO/job_scheduler] Job 2537 completing (errored=False, cancelled=False)
[2017-06-01 19:50:07,839: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:50:08,032: INFO/job_scheduler] Job 2538 finished 1 steps successfully
[2017-06-01 19:50:08,069: INFO/job_scheduler] Job 2538: complete_job: Updating cache
[2017-06-01 19:50:08,070: INFO/job_scheduler] Job 2538 completing (errored=False, cancelled=False)
[2017-06-01 19:50:08,160: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:50:10,182: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:10,182: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:10,194: INFO/command_plan] add_jobs: created Job 2539 (Test for host connectivity)
[2017-06-01 19:50:10,232: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:50:10,290: INFO/job_scheduler] Job 2539: RunJobThread.run
[2017-06-01 19:50:10,354: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:10,359: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:10,389: INFO/command_plan] add_jobs: created Job 2540 (Test for host connectivity)
[2017-06-01 19:50:10,442: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:50:10,452: INFO/job_scheduler] Job 2540: RunJobThread.run
[2017-06-01 19:50:10,542: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:10,542: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:10,572: INFO/command_plan] add_jobs: created Job 2541 (Test for host connectivity)
[2017-06-01 19:50:10,602: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:50:10,608: INFO/job_scheduler] Job 2541: RunJobThread.run
[2017-06-01 19:50:10,708: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:10,712: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:10,780: INFO/command_plan] add_jobs: created Job 2542 (Test for host connectivity)
[2017-06-01 19:50:10,943: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:50:10,950: INFO/job_scheduler] Job 2542: RunJobThread.run
[2017-06-01 19:50:12,581: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:50:12,862: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:50:13,095: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:50:13,178: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:50:13,926: INFO/job_scheduler] Job 2539 finished 1 steps successfully
[2017-06-01 19:50:13,960: INFO/job_scheduler] Job 2539: complete_job: Updating cache
[2017-06-01 19:50:13,960: INFO/job_scheduler] Job 2539 completing (errored=False, cancelled=False)
[2017-06-01 19:50:14,013: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:50:14,278: INFO/job_scheduler] Job 2540 finished 1 steps successfully
[2017-06-01 19:50:14,324: INFO/job_scheduler] Job 2540: complete_job: Updating cache
[2017-06-01 19:50:14,325: INFO/job_scheduler] Job 2540 completing (errored=False, cancelled=False)
[2017-06-01 19:50:14,403: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:50:14,554: INFO/job_scheduler] Job 2542 finished 1 steps successfully
[2017-06-01 19:50:14,577: INFO/job_scheduler] Job 2542: complete_job: Updating cache
[2017-06-01 19:50:14,578: INFO/job_scheduler] Job 2542 completing (errored=False, cancelled=False)
[2017-06-01 19:50:14,643: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:50:14,695: INFO/job_scheduler] Job 2541 finished 1 steps successfully
[2017-06-01 19:50:14,749: INFO/job_scheduler] Job 2541: complete_job: Updating cache
[2017-06-01 19:50:14,759: INFO/job_scheduler] Job 2541 completing (errored=False, cancelled=False)
[2017-06-01 19:50:14,811: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:50:15,654: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:15,655: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:15,689: INFO/command_plan] add_jobs: created Job 2543 (Test for host connectivity)
[2017-06-01 19:50:15,758: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:50:15,792: INFO/job_scheduler] Job 2543: RunJobThread.run
[2017-06-01 19:50:15,842: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:15,842: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:15,874: INFO/command_plan] add_jobs: created Job 2544 (Test for host connectivity)
[2017-06-01 19:50:15,997: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:15,997: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:16,010: INFO/command_plan] add_jobs: created Job 2545 (Test for host connectivity)
[2017-06-01 19:50:16,052: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-01 19:50:16,068: INFO/job_scheduler] Job 2544: RunJobThread.run
[2017-06-01 19:50:16,108: INFO/job_scheduler] Job 2545: RunJobThread.run
[2017-06-01 19:50:16,132: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:50:16,178: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:16,179: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:16,190: INFO/command_plan] add_jobs: created Job 2546 (Test for host connectivity)
[2017-06-01 19:50:16,213: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:50:16,323: INFO/job_scheduler] Job 2546: RunJobThread.run
[2017-06-01 19:50:18,226: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:50:18,421: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:50:18,553: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:50:18,633: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:50:19,695: INFO/job_scheduler] Job 2543 finished 1 steps successfully
[2017-06-01 19:50:19,793: INFO/job_scheduler] Job 2543: complete_job: Updating cache
[2017-06-01 19:50:19,793: INFO/job_scheduler] Job 2543 completing (errored=False, cancelled=False)
[2017-06-01 19:50:19,818: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:50:19,859: INFO/job_scheduler] Job 2544 finished 1 steps successfully
[2017-06-01 19:50:19,904: INFO/job_scheduler] Job 2545 finished 1 steps successfully
[2017-06-01 19:50:19,979: INFO/job_scheduler] Job 2546 finished 1 steps successfully
[2017-06-01 19:50:20,026: INFO/job_scheduler] Job 2544: complete_job: Updating cache
[2017-06-01 19:50:20,027: INFO/job_scheduler] Job 2544 completing (errored=False, cancelled=False)
[2017-06-01 19:50:20,093: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:50:20,152: INFO/job_scheduler] Job 2545: complete_job: Updating cache
[2017-06-01 19:50:20,153: INFO/job_scheduler] Job 2545 completing (errored=False, cancelled=False)
[2017-06-01 19:50:20,352: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:50:20,368: INFO/job_scheduler] Job 2546: complete_job: Updating cache
[2017-06-01 19:50:20,368: INFO/job_scheduler] Job 2546 completing (errored=False, cancelled=False)
[2017-06-01 19:50:20,419: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:50:22,152: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:22,152: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:22,168: INFO/command_plan] add_jobs: created Job 2547 (Test for host connectivity)
[2017-06-01 19:50:22,196: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:50:22,240: INFO/job_scheduler] Job 2547: RunJobThread.run
[2017-06-01 19:50:22,293: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:22,295: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:22,346: INFO/command_plan] add_jobs: created Job 2548 (Test for host connectivity)
[2017-06-01 19:50:22,424: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:50:22,430: INFO/job_scheduler] Job 2548: RunJobThread.run
[2017-06-01 19:50:22,486: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:22,487: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:22,527: INFO/command_plan] add_jobs: created Job 2549 (Test for host connectivity)
[2017-06-01 19:50:22,585: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:50:22,698: INFO/job_scheduler] Job 2549: RunJobThread.run
[2017-06-01 19:50:22,760: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:22,762: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:22,786: INFO/command_plan] add_jobs: created Job 2550 (Test for host connectivity)
[2017-06-01 19:50:22,872: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:50:22,923: INFO/job_scheduler] Job 2550: RunJobThread.run
[2017-06-01 19:50:24,712: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:50:24,842: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:50:24,989: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:50:25,243: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:50:26,036: INFO/job_scheduler] Job 2547 finished 1 steps successfully
[2017-06-01 19:50:26,058: INFO/job_scheduler] Job 2547: complete_job: Updating cache
[2017-06-01 19:50:26,059: INFO/job_scheduler] Job 2547 completing (errored=False, cancelled=False)
[2017-06-01 19:50:26,111: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:50:26,222: INFO/job_scheduler] Job 2548 finished 1 steps successfully
[2017-06-01 19:50:26,272: INFO/job_scheduler] Job 2548: complete_job: Updating cache
[2017-06-01 19:50:26,272: INFO/job_scheduler] Job 2548 completing (errored=False, cancelled=False)
[2017-06-01 19:50:26,340: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:50:26,503: INFO/job_scheduler] Job 2549 finished 1 steps successfully
[2017-06-01 19:50:26,584: INFO/job_scheduler] Job 2549: complete_job: Updating cache
[2017-06-01 19:50:26,584: INFO/job_scheduler] Job 2549 completing (errored=False, cancelled=False)
[2017-06-01 19:50:26,644: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:50:26,798: INFO/job_scheduler] Job 2550 finished 1 steps successfully
[2017-06-01 19:50:26,842: INFO/job_scheduler] Job 2550: complete_job: Updating cache
[2017-06-01 19:50:26,842: INFO/job_scheduler] Job 2550 completing (errored=False, cancelled=False)
[2017-06-01 19:50:26,903: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:50:28,866: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:28,868: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:28,875: INFO/command_plan] add_jobs: created Job 2551 (Test for host connectivity)
[2017-06-01 19:50:28,905: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:50:28,909: INFO/job_scheduler] Job 2551: RunJobThread.run
[2017-06-01 19:50:29,004: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:29,005: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:29,017: INFO/command_plan] add_jobs: created Job 2552 (Test for host connectivity)
[2017-06-01 19:50:29,066: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:50:29,073: INFO/job_scheduler] Job 2552: RunJobThread.run
[2017-06-01 19:50:29,186: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:29,187: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:29,387: INFO/command_plan] add_jobs: created Job 2553 (Test for host connectivity)
[2017-06-01 19:50:29,499: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:50:29,508: INFO/job_scheduler] Job 2553: RunJobThread.run
[2017-06-01 19:50:29,579: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:29,579: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:29,610: INFO/command_plan] add_jobs: created Job 2554 (Test for host connectivity)
[2017-06-01 19:50:29,647: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:50:29,739: INFO/job_scheduler] Job 2554: RunJobThread.run
[2017-06-01 19:50:31,469: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:50:31,568: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:50:31,956: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:50:31,999: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:50:32,918: INFO/job_scheduler] Job 2551 finished 1 steps successfully
[2017-06-01 19:50:32,941: INFO/job_scheduler] Job 2551: complete_job: Updating cache
[2017-06-01 19:50:32,941: INFO/job_scheduler] Job 2551 completing (errored=False, cancelled=False)
[2017-06-01 19:50:32,975: INFO/job_scheduler] Job 2552 finished 1 steps successfully
[2017-06-01 19:50:33,047: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:50:33,060: INFO/job_scheduler] Job 2552: complete_job: Updating cache
[2017-06-01 19:50:33,061: INFO/job_scheduler] Job 2552 completing (errored=False, cancelled=False)
[2017-06-01 19:50:33,185: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:50:33,344: INFO/job_scheduler] Job 2553 finished 1 steps successfully
[2017-06-01 19:50:33,401: INFO/job_scheduler] Job 2553: complete_job: Updating cache
[2017-06-01 19:50:33,401: INFO/job_scheduler] Job 2553 completing (errored=False, cancelled=False)
[2017-06-01 19:50:33,462: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:50:33,475: INFO/job_scheduler] Job 2554 finished 1 steps successfully
[2017-06-01 19:50:33,545: INFO/job_scheduler] Job 2554: complete_job: Updating cache
[2017-06-01 19:50:33,545: INFO/job_scheduler] Job 2554 completing (errored=False, cancelled=False)
[2017-06-01 19:50:33,617: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:50:35,584: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:35,585: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:35,608: INFO/command_plan] add_jobs: created Job 2555 (Test for host connectivity)
[2017-06-01 19:50:35,621: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:50:35,624: INFO/job_scheduler] Job 2555: RunJobThread.run
[2017-06-01 19:50:35,708: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:35,708: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:35,724: INFO/command_plan] add_jobs: created Job 2556 (Test for host connectivity)
[2017-06-01 19:50:35,767: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:50:35,784: INFO/job_scheduler] Job 2556: RunJobThread.run
[2017-06-01 19:50:35,902: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:35,903: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:35,919: INFO/command_plan] add_jobs: created Job 2557 (Test for host connectivity)
[2017-06-01 19:50:36,060: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:36,060: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:36,075: INFO/command_plan] add_jobs: created Job 2558 (Test for host connectivity)
[2017-06-01 19:50:36,084: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:50:36,096: INFO/job_scheduler] Job 2557: RunJobThread.run
[2017-06-01 19:50:36,096: INFO/job_scheduler] Job 2558: RunJobThread.run
[2017-06-01 19:50:36,221: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:50:38,036: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:50:38,257: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:50:38,461: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:50:38,552: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:50:39,344: INFO/job_scheduler] Job 2555 finished 1 steps successfully
[2017-06-01 19:50:39,407: INFO/job_scheduler] Job 2555: complete_job: Updating cache
[2017-06-01 19:50:39,407: INFO/job_scheduler] Job 2555 completing (errored=False, cancelled=False)
[2017-06-01 19:50:39,476: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:50:39,614: INFO/job_scheduler] Job 2556 finished 1 steps successfully
[2017-06-01 19:50:39,692: INFO/job_scheduler] Job 2556: complete_job: Updating cache
[2017-06-01 19:50:39,693: INFO/job_scheduler] Job 2556 completing (errored=False, cancelled=False)
[2017-06-01 19:50:39,749: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:50:39,791: INFO/job_scheduler] Job 2558 finished 1 steps successfully
[2017-06-01 19:50:39,822: INFO/job_scheduler] Job 2558: complete_job: Updating cache
[2017-06-01 19:50:39,829: INFO/job_scheduler] Job 2558 completing (errored=False, cancelled=False)
[2017-06-01 19:50:39,882: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:50:39,942: INFO/job_scheduler] Job 2557 finished 1 steps successfully
[2017-06-01 19:50:39,973: INFO/job_scheduler] Job 2557: complete_job: Updating cache
[2017-06-01 19:50:39,973: INFO/job_scheduler] Job 2557 completing (errored=False, cancelled=False)
[2017-06-01 19:50:40,030: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:50:42,074: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:42,074: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:42,095: INFO/command_plan] add_jobs: created Job 2559 (Test for host connectivity)
[2017-06-01 19:50:42,152: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:50:42,186: INFO/job_scheduler] Job 2559: RunJobThread.run
[2017-06-01 19:50:42,238: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:42,239: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:42,264: INFO/command_plan] add_jobs: created Job 2560 (Test for host connectivity)
[2017-06-01 19:50:42,349: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:50:42,355: INFO/job_scheduler] Job 2560: RunJobThread.run
[2017-06-01 19:50:42,483: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:42,483: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:42,503: INFO/command_plan] add_jobs: created Job 2561 (Test for host connectivity)
[2017-06-01 19:50:42,563: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:50:42,610: INFO/job_scheduler] Job 2561: RunJobThread.run
[2017-06-01 19:50:42,656: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:42,656: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:42,669: INFO/command_plan] add_jobs: created Job 2562 (Test for host connectivity)
[2017-06-01 19:50:42,694: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:50:42,712: INFO/job_scheduler] Job 2562: RunJobThread.run
[2017-06-01 19:50:44,570: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:50:44,832: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:50:45,025: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:50:45,138: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:50:45,936: INFO/job_scheduler] Job 2559 finished 1 steps successfully
[2017-06-01 19:50:45,952: INFO/job_scheduler] Job 2559: complete_job: Updating cache
[2017-06-01 19:50:45,952: INFO/job_scheduler] Job 2559 completing (errored=False, cancelled=False)
[2017-06-01 19:50:45,978: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:50:46,240: INFO/job_scheduler] Job 2560 finished 1 steps successfully
[2017-06-01 19:50:46,386: INFO/job_scheduler] Job 2560: complete_job: Updating cache
[2017-06-01 19:50:46,387: INFO/job_scheduler] Job 2560 completing (errored=False, cancelled=False)
[2017-06-01 19:50:46,455: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:50:46,537: INFO/job_scheduler] Job 2562 finished 1 steps successfully
[2017-06-01 19:50:46,556: INFO/job_scheduler] Job 2561 finished 1 steps successfully
[2017-06-01 19:50:46,613: INFO/job_scheduler] Job 2562: complete_job: Updating cache
[2017-06-01 19:50:46,614: INFO/job_scheduler] Job 2562 completing (errored=False, cancelled=False)
[2017-06-01 19:50:46,677: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:50:46,688: INFO/job_scheduler] Job 2561: complete_job: Updating cache
[2017-06-01 19:50:46,688: INFO/job_scheduler] Job 2561 completing (errored=False, cancelled=False)
[2017-06-01 19:50:46,744: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:50:49,382: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:49,382: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:49,393: INFO/command_plan] add_jobs: created Job 2563 (Test for host connectivity)
[2017-06-01 19:50:49,462: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:50:49,476: INFO/job_scheduler] Job 2563: RunJobThread.run
[2017-06-01 19:50:49,506: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:49,506: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:49,519: INFO/command_plan] add_jobs: created Job 2564 (Test for host connectivity)
[2017-06-01 19:50:49,601: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:50:49,626: INFO/job_scheduler] Job 2564: RunJobThread.run
[2017-06-01 19:50:49,705: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:49,705: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:49,729: INFO/command_plan] add_jobs: created Job 2565 (Test for host connectivity)
[2017-06-01 19:50:49,829: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:49,829: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:49,858: INFO/command_plan] add_jobs: created Job 2566 (Test for host connectivity)
[2017-06-01 19:50:49,892: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:50:49,910: INFO/job_scheduler] Job 2565: RunJobThread.run
[2017-06-01 19:50:49,936: INFO/job_scheduler] Job 2566: RunJobThread.run
[2017-06-01 19:50:50,007: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:50:51,909: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:50:52,013: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:50:52,281: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:50:52,362: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:50:53,202: INFO/job_scheduler] Job 2563 finished 1 steps successfully
[2017-06-01 19:50:53,236: INFO/job_scheduler] Job 2563: complete_job: Updating cache
[2017-06-01 19:50:53,236: INFO/job_scheduler] Job 2563 completing (errored=False, cancelled=False)
[2017-06-01 19:50:53,294: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:50:53,437: INFO/job_scheduler] Job 2564 finished 1 steps successfully
[2017-06-01 19:50:53,469: INFO/job_scheduler] Job 2564: complete_job: Updating cache
[2017-06-01 19:50:53,471: INFO/job_scheduler] Job 2564 completing (errored=False, cancelled=False)
[2017-06-01 19:50:53,602: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:50:53,771: INFO/job_scheduler] Job 2566 finished 1 steps successfully
[2017-06-01 19:50:53,834: INFO/job_scheduler] Job 2566: complete_job: Updating cache
[2017-06-01 19:50:53,834: INFO/job_scheduler] Job 2566 completing (errored=False, cancelled=False)
[2017-06-01 19:50:53,866: INFO/job_scheduler] Job 2565 finished 1 steps successfully
[2017-06-01 19:50:53,893: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:50:53,912: INFO/job_scheduler] Job 2565: complete_job: Updating cache
[2017-06-01 19:50:53,913: INFO/job_scheduler] Job 2565 completing (errored=False, cancelled=False)
[2017-06-01 19:50:54,022: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:50:55,683: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:55,685: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:55,691: INFO/command_plan] add_jobs: created Job 2567 (Test for host connectivity)
[2017-06-01 19:50:55,753: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:50:55,799: INFO/job_scheduler] Job 2567: RunJobThread.run
[2017-06-01 19:50:55,854: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:55,855: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:55,874: INFO/command_plan] add_jobs: created Job 2568 (Test for host connectivity)
[2017-06-01 19:50:55,910: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:50:55,934: INFO/job_scheduler] Job 2568: RunJobThread.run
[2017-06-01 19:50:56,026: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:56,028: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:56,042: INFO/command_plan] add_jobs: created Job 2569 (Test for host connectivity)
[2017-06-01 19:50:56,105: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:50:56,165: INFO/job_scheduler] Job 2569: RunJobThread.run
[2017-06-01 19:50:56,255: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:50:56,255: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:50:56,281: INFO/command_plan] add_jobs: created Job 2570 (Test for host connectivity)
[2017-06-01 19:50:56,335: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:50:56,426: INFO/job_scheduler] Job 2570: RunJobThread.run
[2017-06-01 19:50:58,098: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:50:58,293: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:50:58,545: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:50:58,588: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:50:59,433: INFO/job_scheduler] Job 2567 finished 1 steps successfully
[2017-06-01 19:50:59,458: INFO/job_scheduler] Job 2567: complete_job: Updating cache
[2017-06-01 19:50:59,458: INFO/job_scheduler] Job 2567 completing (errored=False, cancelled=False)
[2017-06-01 19:50:59,499: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:50:59,635: INFO/job_scheduler] Job 2568 finished 1 steps successfully
[2017-06-01 19:50:59,724: INFO/job_scheduler] Job 2568: complete_job: Updating cache
[2017-06-01 19:50:59,724: INFO/job_scheduler] Job 2568 completing (errored=False, cancelled=False)
[2017-06-01 19:50:59,833: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:51:00,031: INFO/job_scheduler] Job 2569 finished 1 steps successfully
[2017-06-01 19:51:00,066: INFO/job_scheduler] Job 2570 finished 1 steps successfully
[2017-06-01 19:51:00,117: INFO/job_scheduler] Job 2569: complete_job: Updating cache
[2017-06-01 19:51:00,118: INFO/job_scheduler] Job 2569 completing (errored=False, cancelled=False)
[2017-06-01 19:51:00,192: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:51:00,208: INFO/job_scheduler] Job 2570: complete_job: Updating cache
[2017-06-01 19:51:00,212: INFO/job_scheduler] Job 2570 completing (errored=False, cancelled=False)
[2017-06-01 19:51:00,268: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:51:02,161: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:02,161: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:02,175: INFO/command_plan] add_jobs: created Job 2571 (Test for host connectivity)
[2017-06-01 19:51:02,193: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:51:02,211: INFO/job_scheduler] Job 2571: RunJobThread.run
[2017-06-01 19:51:02,368: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:02,369: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:02,384: INFO/command_plan] add_jobs: created Job 2572 (Test for host connectivity)
[2017-06-01 19:51:02,431: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:51:02,462: INFO/job_scheduler] Job 2572: RunJobThread.run
[2017-06-01 19:51:02,559: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:02,562: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:02,593: INFO/command_plan] add_jobs: created Job 2573 (Test for host connectivity)
[2017-06-01 19:51:02,676: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:51:02,715: INFO/job_scheduler] Job 2573: RunJobThread.run
[2017-06-01 19:51:02,752: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:02,754: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:02,768: INFO/command_plan] add_jobs: created Job 2574 (Test for host connectivity)
[2017-06-01 19:51:02,835: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:51:02,912: INFO/job_scheduler] Job 2574: RunJobThread.run
[2017-06-01 19:51:04,720: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:51:04,910: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:51:05,162: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:51:05,223: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:51:06,049: INFO/job_scheduler] Job 2571 finished 1 steps successfully
[2017-06-01 19:51:06,090: INFO/job_scheduler] Job 2571: complete_job: Updating cache
[2017-06-01 19:51:06,090: INFO/job_scheduler] Job 2571 completing (errored=False, cancelled=False)
[2017-06-01 19:51:06,132: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:51:06,285: INFO/job_scheduler] Job 2572 finished 1 steps successfully
[2017-06-01 19:51:06,364: INFO/job_scheduler] Job 2572: complete_job: Updating cache
[2017-06-01 19:51:06,365: INFO/job_scheduler] Job 2572 completing (errored=False, cancelled=False)
[2017-06-01 19:51:06,423: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:51:06,560: INFO/job_scheduler] Job 2573 finished 1 steps successfully
[2017-06-01 19:51:06,606: INFO/job_scheduler] Job 2573: complete_job: Updating cache
[2017-06-01 19:51:06,607: INFO/job_scheduler] Job 2573 completing (errored=False, cancelled=False)
[2017-06-01 19:51:06,630: INFO/job_scheduler] Job 2574 finished 1 steps successfully
[2017-06-01 19:51:06,665: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:51:06,688: INFO/job_scheduler] Job 2574: complete_job: Updating cache
[2017-06-01 19:51:06,688: INFO/job_scheduler] Job 2574 completing (errored=False, cancelled=False)
[2017-06-01 19:51:06,740: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:51:08,675: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:08,683: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:08,702: INFO/command_plan] add_jobs: created Job 2575 (Test for host connectivity)
[2017-06-01 19:51:08,770: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:51:08,822: INFO/job_scheduler] Job 2575: RunJobThread.run
[2017-06-01 19:51:08,853: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:08,853: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:08,986: INFO/command_plan] add_jobs: created Job 2576 (Test for host connectivity)
[2017-06-01 19:51:09,037: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:51:09,126: INFO/job_scheduler] Job 2576: RunJobThread.run
[2017-06-01 19:51:09,174: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:09,175: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:09,189: INFO/command_plan] add_jobs: created Job 2577 (Test for host connectivity)
[2017-06-01 19:51:09,247: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:51:09,265: INFO/job_scheduler] Job 2577: RunJobThread.run
[2017-06-01 19:51:09,427: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:09,431: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:09,574: INFO/command_plan] add_jobs: created Job 2578 (Test for host connectivity)
[2017-06-01 19:51:09,675: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:51:09,692: INFO/job_scheduler] Job 2578: RunJobThread.run
[2017-06-01 19:51:11,194: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:51:11,615: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:51:11,753: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:51:11,850: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:51:12,529: INFO/job_scheduler] Job 2575 finished 1 steps successfully
[2017-06-01 19:51:12,555: INFO/job_scheduler] Job 2575: complete_job: Updating cache
[2017-06-01 19:51:12,555: INFO/job_scheduler] Job 2575 completing (errored=False, cancelled=False)
[2017-06-01 19:51:12,589: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:51:13,083: INFO/job_scheduler] Job 2576 finished 1 steps successfully
[2017-06-01 19:51:13,138: INFO/job_scheduler] Job 2576: complete_job: Updating cache
[2017-06-01 19:51:13,138: INFO/job_scheduler] Job 2576 completing (errored=False, cancelled=False)
[2017-06-01 19:51:13,164: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:51:13,202: INFO/job_scheduler] Job 2577 finished 1 steps successfully
[2017-06-01 19:51:13,247: INFO/job_scheduler] Job 2577: complete_job: Updating cache
[2017-06-01 19:51:13,247: INFO/job_scheduler] Job 2577 completing (errored=False, cancelled=False)
[2017-06-01 19:51:13,249: INFO/job_scheduler] Job 2578 finished 1 steps successfully
[2017-06-01 19:51:13,299: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:51:13,330: INFO/job_scheduler] Job 2578: complete_job: Updating cache
[2017-06-01 19:51:13,331: INFO/job_scheduler] Job 2578 completing (errored=False, cancelled=False)
[2017-06-01 19:51:13,411: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:51:14,265: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:14,265: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:14,280: INFO/command_plan] add_jobs: created Job 2579 (Test for host connectivity)
[2017-06-01 19:51:14,329: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:51:14,343: INFO/job_scheduler] Job 2579: RunJobThread.run
[2017-06-01 19:51:14,485: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:14,488: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:14,499: INFO/command_plan] add_jobs: created Job 2580 (Test for host connectivity)
[2017-06-01 19:51:14,590: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:51:14,634: INFO/job_scheduler] Job 2580: RunJobThread.run
[2017-06-01 19:51:14,715: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:14,716: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:14,746: INFO/command_plan] add_jobs: created Job 2581 (Test for host connectivity)
[2017-06-01 19:51:14,955: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:14,955: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:14,972: INFO/command_plan] add_jobs: created Job 2582 (Test for host connectivity)
[2017-06-01 19:51:14,998: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-01 19:51:15,026: INFO/job_scheduler] Job 2581: RunJobThread.run
[2017-06-01 19:51:15,049: INFO/job_scheduler] Job 2582: RunJobThread.run
[2017-06-01 19:51:15,057: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:51:16,960: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:51:17,162: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:51:17,324: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:51:17,418: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:51:18,300: INFO/job_scheduler] Job 2579 finished 1 steps successfully
[2017-06-01 19:51:18,366: INFO/job_scheduler] Job 2579: complete_job: Updating cache
[2017-06-01 19:51:18,369: INFO/job_scheduler] Job 2579 completing (errored=False, cancelled=False)
[2017-06-01 19:51:18,405: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:51:18,550: INFO/job_scheduler] Job 2580 finished 1 steps successfully
[2017-06-01 19:51:18,611: INFO/job_scheduler] Job 2580: complete_job: Updating cache
[2017-06-01 19:51:18,615: INFO/job_scheduler] Job 2580 completing (errored=False, cancelled=False)
[2017-06-01 19:51:18,667: INFO/job_scheduler] Job 2582 finished 1 steps successfully
[2017-06-01 19:51:18,679: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:51:18,697: INFO/job_scheduler] Job 2582: complete_job: Updating cache
[2017-06-01 19:51:18,697: INFO/job_scheduler] Job 2582 completing (errored=False, cancelled=False)
[2017-06-01 19:51:18,821: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:51:18,899: INFO/job_scheduler] Job 2581 finished 1 steps successfully
[2017-06-01 19:51:18,930: INFO/job_scheduler] Job 2581: complete_job: Updating cache
[2017-06-01 19:51:18,931: INFO/job_scheduler] Job 2581 completing (errored=False, cancelled=False)
[2017-06-01 19:51:18,981: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:51:20,918: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:20,920: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:20,938: INFO/command_plan] add_jobs: created Job 2583 (Test for host connectivity)
[2017-06-01 19:51:21,056: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:21,056: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:21,089: INFO/command_plan] add_jobs: created Job 2584 (Test for host connectivity)
[2017-06-01 19:51:21,132: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:51:21,158: INFO/job_scheduler] Job 2584: RunJobThread.run
[2017-06-01 19:51:21,178: INFO/job_scheduler] Job 2583: RunJobThread.run
[2017-06-01 19:51:21,226: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:21,226: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:21,262: INFO/command_plan] add_jobs: created Job 2585 (Test for host connectivity)
[2017-06-01 19:51:21,374: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:51:21,396: INFO/job_scheduler] Job 2585: RunJobThread.run
[2017-06-01 19:51:21,510: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:21,511: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:21,528: INFO/command_plan] add_jobs: created Job 2586 (Test for host connectivity)
[2017-06-01 19:51:21,537: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:51:21,572: INFO/job_scheduler] Job 2586: RunJobThread.run
[2017-06-01 19:51:21,622: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-01 19:51:23,513: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:51:23,609: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:51:23,735: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:51:24,117: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:51:24,960: INFO/job_scheduler] Job 2583 finished 1 steps successfully
[2017-06-01 19:51:25,010: INFO/job_scheduler] Job 2583: complete_job: Updating cache
[2017-06-01 19:51:25,011: INFO/job_scheduler] Job 2583 completing (errored=False, cancelled=False)
[2017-06-01 19:51:25,031: INFO/job_scheduler] Job 2584 finished 1 steps successfully
[2017-06-01 19:51:25,064: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:51:25,083: INFO/job_scheduler] Job 2585 finished 1 steps successfully
[2017-06-01 19:51:25,203: INFO/job_scheduler] Job 2584: complete_job: Updating cache
[2017-06-01 19:51:25,204: INFO/job_scheduler] Job 2584 completing (errored=False, cancelled=False)
[2017-06-01 19:51:25,282: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:51:25,375: INFO/job_scheduler] Job 2585: complete_job: Updating cache
[2017-06-01 19:51:25,375: INFO/job_scheduler] Job 2585 completing (errored=False, cancelled=False)
[2017-06-01 19:51:25,427: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:51:25,671: INFO/job_scheduler] Job 2586 finished 1 steps successfully
[2017-06-01 19:51:25,744: INFO/job_scheduler] Job 2586: complete_job: Updating cache
[2017-06-01 19:51:25,745: INFO/job_scheduler] Job 2586 completing (errored=False, cancelled=False)
[2017-06-01 19:51:25,821: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:51:27,360: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:27,360: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:27,493: INFO/command_plan] add_jobs: created Job 2587 (Test for host connectivity)
[2017-06-01 19:51:27,540: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:51:27,551: INFO/job_scheduler] Job 2587: RunJobThread.run
[2017-06-01 19:51:27,609: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:27,611: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:27,618: INFO/command_plan] add_jobs: created Job 2588 (Test for host connectivity)
[2017-06-01 19:51:27,656: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:51:27,672: INFO/job_scheduler] Job 2588: RunJobThread.run
[2017-06-01 19:51:27,775: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:27,780: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:27,803: INFO/command_plan] add_jobs: created Job 2589 (Test for host connectivity)
[2017-06-01 19:51:27,867: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:51:27,898: INFO/job_scheduler] Job 2589: RunJobThread.run
[2017-06-01 19:51:27,990: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:27,990: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:28,001: INFO/command_plan] add_jobs: created Job 2590 (Test for host connectivity)
[2017-06-01 19:51:28,056: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:51:28,105: INFO/job_scheduler] Job 2590: RunJobThread.run
[2017-06-01 19:51:29,874: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:51:30,095: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:51:30,193: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:51:30,377: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:51:31,174: INFO/job_scheduler] Job 2587 finished 1 steps successfully
[2017-06-01 19:51:31,218: INFO/job_scheduler] Job 2587: complete_job: Updating cache
[2017-06-01 19:51:31,218: INFO/job_scheduler] Job 2587 completing (errored=False, cancelled=False)
[2017-06-01 19:51:31,284: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:51:31,499: INFO/job_scheduler] Job 2588 finished 1 steps successfully
[2017-06-01 19:51:31,551: INFO/job_scheduler] Job 2588: complete_job: Updating cache
[2017-06-01 19:51:31,553: INFO/job_scheduler] Job 2588 completing (errored=False, cancelled=False)
[2017-06-01 19:51:31,618: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:51:31,705: INFO/job_scheduler] Job 2589 finished 1 steps successfully
[2017-06-01 19:51:31,741: INFO/job_scheduler] Job 2590 finished 1 steps successfully
[2017-06-01 19:51:31,768: INFO/job_scheduler] Job 2589: complete_job: Updating cache
[2017-06-01 19:51:31,769: INFO/job_scheduler] Job 2589 completing (errored=False, cancelled=False)
[2017-06-01 19:51:31,851: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:51:31,871: INFO/job_scheduler] Job 2590: complete_job: Updating cache
[2017-06-01 19:51:31,872: INFO/job_scheduler] Job 2590 completing (errored=False, cancelled=False)
[2017-06-01 19:51:32,027: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:51:33,872: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:33,872: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:33,894: INFO/command_plan] add_jobs: created Job 2591 (Test for host connectivity)
[2017-06-01 19:51:34,020: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:34,021: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:34,037: INFO/command_plan] add_jobs: created Job 2592 (Test for host connectivity)
[2017-06-01 19:51:34,052: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:51:34,059: INFO/job_scheduler] Job 2592: RunJobThread.run
[2017-06-01 19:51:34,095: INFO/job_scheduler] Job 2591: RunJobThread.run
[2017-06-01 19:51:34,119: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:51:34,211: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:34,212: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:34,234: INFO/command_plan] add_jobs: created Job 2593 (Test for host connectivity)
[2017-06-01 19:51:34,328: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:51:34,354: INFO/job_scheduler] Job 2593: RunJobThread.run
[2017-06-01 19:51:34,470: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:34,475: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:34,492: INFO/command_plan] add_jobs: created Job 2594 (Test for host connectivity)
[2017-06-01 19:51:34,541: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:51:34,580: INFO/job_scheduler] Job 2594: RunJobThread.run
[2017-06-01 19:51:36,498: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:51:36,583: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:51:36,667: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:51:36,899: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:51:37,836: INFO/job_scheduler] Job 2591 finished 1 steps successfully
[2017-06-01 19:51:37,891: INFO/job_scheduler] Job 2591: complete_job: Updating cache
[2017-06-01 19:51:37,891: INFO/job_scheduler] Job 2591 completing (errored=False, cancelled=False)
[2017-06-01 19:51:37,939: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:51:37,959: INFO/job_scheduler] Job 2592 finished 1 steps successfully
[2017-06-01 19:51:37,998: INFO/job_scheduler] Job 2592: complete_job: Updating cache
[2017-06-01 19:51:37,998: INFO/job_scheduler] Job 2592 completing (errored=False, cancelled=False)
[2017-06-01 19:51:38,095: INFO/job_scheduler] Job 2593 finished 1 steps successfully
[2017-06-01 19:51:38,116: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:51:38,200: INFO/job_scheduler] Job 2593: complete_job: Updating cache
[2017-06-01 19:51:38,200: INFO/job_scheduler] Job 2593 completing (errored=False, cancelled=False)
[2017-06-01 19:51:38,278: INFO/job_scheduler] Job 2594 finished 1 steps successfully
[2017-06-01 19:51:38,318: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:51:38,353: INFO/job_scheduler] Job 2594: complete_job: Updating cache
[2017-06-01 19:51:38,353: INFO/job_scheduler] Job 2594 completing (errored=False, cancelled=False)
[2017-06-01 19:51:38,425: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:51:40,361: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:40,362: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:40,374: INFO/command_plan] add_jobs: created Job 2595 (Test for host connectivity)
[2017-06-01 19:51:40,427: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:51:40,446: INFO/job_scheduler] Job 2595: RunJobThread.run
[2017-06-01 19:51:40,632: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:40,632: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:40,700: INFO/command_plan] add_jobs: created Job 2596 (Test for host connectivity)
[2017-06-01 19:51:40,786: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:51:40,793: INFO/job_scheduler] Job 2596: RunJobThread.run
[2017-06-01 19:51:40,961: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:40,966: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:40,984: INFO/command_plan] add_jobs: created Job 2597 (Test for host connectivity)
[2017-06-01 19:51:41,071: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:51:41,077: INFO/job_scheduler] Job 2597: RunJobThread.run
[2017-06-01 19:51:41,146: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:41,146: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:41,159: INFO/command_plan] add_jobs: created Job 2598 (Test for host connectivity)
[2017-06-01 19:51:41,203: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:51:41,237: INFO/job_scheduler] Job 2598: RunJobThread.run
[2017-06-01 19:51:42,814: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:51:43,275: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:51:43,420: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:51:43,520: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:51:44,156: INFO/job_scheduler] Job 2595 finished 1 steps successfully
[2017-06-01 19:51:44,172: INFO/job_scheduler] Job 2595: complete_job: Updating cache
[2017-06-01 19:51:44,173: INFO/job_scheduler] Job 2595 completing (errored=False, cancelled=False)
[2017-06-01 19:51:44,208: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:51:44,728: INFO/job_scheduler] Job 2596 finished 1 steps successfully
[2017-06-01 19:51:44,789: INFO/job_scheduler] Job 2596: complete_job: Updating cache
[2017-06-01 19:51:44,789: INFO/job_scheduler] Job 2596 completing (errored=False, cancelled=False)
[2017-06-01 19:51:44,844: INFO/job_scheduler] Job 2597 finished 1 steps successfully
[2017-06-01 19:51:44,941: INFO/job_scheduler] Job 2598 finished 1 steps successfully
[2017-06-01 19:51:44,958: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:51:44,977: INFO/job_scheduler] Job 2597: complete_job: Updating cache
[2017-06-01 19:51:44,977: INFO/job_scheduler] Job 2597 completing (errored=False, cancelled=False)
[2017-06-01 19:51:45,056: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:51:45,072: INFO/job_scheduler] Job 2598: complete_job: Updating cache
[2017-06-01 19:51:45,072: INFO/job_scheduler] Job 2598 completing (errored=False, cancelled=False)
[2017-06-01 19:51:45,138: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:51:45,823: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:45,830: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:45,849: INFO/command_plan] add_jobs: created Job 2599 (Test for host connectivity)
[2017-06-01 19:51:45,867: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 19:51:45,887: INFO/job_scheduler] Job 2599: RunJobThread.run
[2017-06-01 19:51:45,945: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:45,951: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:45,974: INFO/command_plan] add_jobs: created Job 2600 (Test for host connectivity)
[2017-06-01 19:51:46,009: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-01 19:51:46,013: INFO/job_scheduler] Job 2600: RunJobThread.run
[2017-06-01 19:51:46,165: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:46,168: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:46,197: INFO/command_plan] add_jobs: created Job 2601 (Test for host connectivity)
[2017-06-01 19:51:46,279: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:51:46,286: INFO/job_scheduler] Job 2601: RunJobThread.run
[2017-06-01 19:51:46,385: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:46,386: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:46,413: INFO/command_plan] add_jobs: created Job 2602 (Test for host connectivity)
[2017-06-01 19:51:46,518: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:51:46,555: INFO/job_scheduler] Job 2602: RunJobThread.run
[2017-06-01 19:51:48,287: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:51:48,386: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:51:48,727: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:51:49,101: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:51:49,753: INFO/job_scheduler] Job 2600 finished 1 steps successfully
[2017-06-01 19:51:49,773: INFO/job_scheduler] Job 2599 finished 1 steps successfully
[2017-06-01 19:51:49,796: INFO/job_scheduler] Job 2600: complete_job: Updating cache
[2017-06-01 19:51:49,797: INFO/job_scheduler] Job 2600 completing (errored=False, cancelled=False)
[2017-06-01 19:51:49,895: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:51:49,921: INFO/job_scheduler] Job 2599: complete_job: Updating cache
[2017-06-01 19:51:49,921: INFO/job_scheduler] Job 2599 completing (errored=False, cancelled=False)
[2017-06-01 19:51:49,974: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:51:50,192: INFO/job_scheduler] Job 2601 finished 1 steps successfully
[2017-06-01 19:51:50,238: INFO/job_scheduler] Job 2601: complete_job: Updating cache
[2017-06-01 19:51:50,239: INFO/job_scheduler] Job 2601 completing (errored=False, cancelled=False)
[2017-06-01 19:51:50,355: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:51:50,613: INFO/job_scheduler] Job 2602 finished 1 steps successfully
[2017-06-01 19:51:50,663: INFO/job_scheduler] Job 2602: complete_job: Updating cache
[2017-06-01 19:51:50,663: INFO/job_scheduler] Job 2602 completing (errored=False, cancelled=False)
[2017-06-01 19:51:50,721: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 19:51:52,365: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:52,365: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:52,432: INFO/command_plan] add_jobs: created Job 2603 (Test for host connectivity)
[2017-06-01 19:51:52,482: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:52,483: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:52,538: INFO/command_plan] add_jobs: created Job 2604 (Test for host connectivity)
[2017-06-01 19:51:52,549: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-01 19:51:52,573: INFO/job_scheduler] Job 2603: RunJobThread.run
[2017-06-01 19:51:52,575: INFO/job_scheduler] Job 2604: RunJobThread.run
[2017-06-01 19:51:52,588: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:51:52,702: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:52,702: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:52,726: INFO/command_plan] add_jobs: created Job 2605 (Test for host connectivity)
[2017-06-01 19:51:52,795: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-01 19:51:52,826: INFO/job_scheduler] Job 2605: RunJobThread.run
[2017-06-01 19:51:52,896: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 19:51:52,904: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 19:51:52,932: INFO/command_plan] add_jobs: created Job 2606 (Test for host connectivity)
[2017-06-01 19:51:53,039: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-01 19:51:53,099: INFO/job_scheduler] Job 2606: RunJobThread.run
[2017-06-01 19:51:55,041: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 19:51:55,133: ERROR/job] Failed configuration check on 'lotus-32vm7.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:51:55,252: ERROR/job] Failed configuration check on 'lotus-32vm8.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:51:55,452: ERROR/job] Failed configuration check on 'lotus-32vm6.lotus.hpdd.lab.intel.com': EPEL repository not detected in yum configuration
[2017-06-01 19:51:56,376: INFO/job_scheduler] Job 2604 finished 1 steps successfully
[2017-06-01 19:51:56,418: INFO/job_scheduler] Job 2603 finished 1 steps successfully
[2017-06-01 19:51:56,436: INFO/job_scheduler] Job 2604: complete_job: Updating cache
[2017-06-01 19:51:56,437: INFO/job_scheduler] Job 2604 completing (errored=False, cancelled=False)
[2017-06-01 19:51:56,539: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-01 19:51:56,586: INFO/job_scheduler] Job 2603: complete_job: Updating cache
[2017-06-01 19:51:56,586: INFO/job_scheduler] Job 2603 completing (errored=False, cancelled=False)
[2017-06-01 19:51:56,688: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-01 19:51:56,753: INFO/job_scheduler] Job 2605 finished 1 steps successfully
[2017-06-01 19:51:56,779: INFO/job_scheduler] Job 2605: complete_job: Updating cache
[2017-06-01 19:51:56,779: INFO/job_scheduler] Job 2605 completing (errored=False, cancelled=False)
[2017-06-01 19:51:56,787: INFO/job_scheduler] Job 2606 finished 1 steps successfully
[2017-06-01 19:51:56,837: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-01 19:51:56,871: INFO/job_scheduler] Job 2606: complete_job: Updating cache
[2017-06-01 19:51:56,871: INFO/job_scheduler] Job 2606 completing (errored=False, cancelled=False)
[2017-06-01 19:51:56,946: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:07:51,490: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:07:51,491: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:07:51,507: INFO/command_plan] add_jobs: created Job 2607 (Test for host connectivity)
[2017-06-01 23:07:51,597: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:07:51,678: INFO/job_scheduler] Job 2607: RunJobThread.run
[2017-06-01 23:07:54,008: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:07:55,323: INFO/job_scheduler] Job 2607 finished 1 steps successfully
[2017-06-01 23:07:55,364: INFO/job_scheduler] Job 2607: complete_job: Updating cache
[2017-06-01 23:07:55,364: INFO/job_scheduler] Job 2607 completing (errored=False, cancelled=False)
[2017-06-01 23:07:55,414: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:07:55,574: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:07:55,576: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:07:55,593: INFO/command_plan] add_jobs: created Job 2608 (Test for host connectivity)
[2017-06-01 23:07:55,622: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:07:55,660: INFO/job_scheduler] Job 2608: RunJobThread.run
[2017-06-01 23:07:57,870: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:07:59,184: INFO/job_scheduler] Job 2608 finished 1 steps successfully
[2017-06-01 23:07:59,255: INFO/job_scheduler] Job 2608: complete_job: Updating cache
[2017-06-01 23:07:59,255: INFO/job_scheduler] Job 2608 completing (errored=False, cancelled=False)
[2017-06-01 23:07:59,292: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:08:00,448: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:08:00,448: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:08:00,464: INFO/command_plan] add_jobs: created Job 2609 (Test for host connectivity)
[2017-06-01 23:08:00,512: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:08:00,558: INFO/job_scheduler] Job 2609: RunJobThread.run
[2017-06-01 23:08:02,759: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:08:04,118: INFO/job_scheduler] Job 2609 finished 1 steps successfully
[2017-06-01 23:08:04,146: INFO/job_scheduler] Job 2609: complete_job: Updating cache
[2017-06-01 23:08:04,146: INFO/job_scheduler] Job 2609 completing (errored=False, cancelled=False)
[2017-06-01 23:08:04,174: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:08:05,322: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:08:05,322: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:08:05,329: INFO/command_plan] add_jobs: created Job 2610 (Test for host connectivity)
[2017-06-01 23:08:05,343: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:08:05,366: INFO/job_scheduler] Job 2610: RunJobThread.run
[2017-06-01 23:08:07,582: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:08:08,899: INFO/job_scheduler] Job 2610 finished 1 steps successfully
[2017-06-01 23:08:08,931: INFO/job_scheduler] Job 2610: complete_job: Updating cache
[2017-06-01 23:08:08,931: INFO/job_scheduler] Job 2610 completing (errored=False, cancelled=False)
[2017-06-01 23:08:08,996: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:08:10,145: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:08:10,147: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:08:10,154: INFO/command_plan] add_jobs: created Job 2611 (Test for host connectivity)
[2017-06-01 23:08:10,165: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:08:10,192: INFO/job_scheduler] Job 2611: RunJobThread.run
[2017-06-01 23:08:12,332: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:08:13,640: INFO/job_scheduler] Job 2611 finished 1 steps successfully
[2017-06-01 23:08:13,653: INFO/job_scheduler] Job 2611: complete_job: Updating cache
[2017-06-01 23:08:13,654: INFO/job_scheduler] Job 2611 completing (errored=False, cancelled=False)
[2017-06-01 23:08:13,687: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:08:14,830: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:08:14,830: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:08:14,854: INFO/command_plan] add_jobs: created Job 2612 (Test for host connectivity)
[2017-06-01 23:08:14,918: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:08:14,972: INFO/job_scheduler] Job 2612: RunJobThread.run
[2017-06-01 23:08:17,169: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:08:18,490: INFO/job_scheduler] Job 2612 finished 1 steps successfully
[2017-06-01 23:08:18,519: INFO/job_scheduler] Job 2612: complete_job: Updating cache
[2017-06-01 23:08:18,520: INFO/job_scheduler] Job 2612 completing (errored=False, cancelled=False)
[2017-06-01 23:08:18,556: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:08:19,721: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:08:19,723: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:08:19,736: INFO/command_plan] add_jobs: created Job 2613 (Test for host connectivity)
[2017-06-01 23:08:19,780: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:08:19,785: INFO/job_scheduler] Job 2613: RunJobThread.run
[2017-06-01 23:08:21,961: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:08:23,276: INFO/job_scheduler] Job 2613 finished 1 steps successfully
[2017-06-01 23:08:23,311: INFO/job_scheduler] Job 2613: complete_job: Updating cache
[2017-06-01 23:08:23,311: INFO/job_scheduler] Job 2613 completing (errored=False, cancelled=False)
[2017-06-01 23:08:23,350: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:08:24,504: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:08:24,504: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:08:24,519: INFO/command_plan] add_jobs: created Job 2614 (Test for host connectivity)
[2017-06-01 23:08:24,596: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:08:24,654: INFO/job_scheduler] Job 2614: RunJobThread.run
[2017-06-01 23:08:26,840: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:08:28,137: INFO/job_scheduler] Job 2614 finished 1 steps successfully
[2017-06-01 23:08:28,160: INFO/job_scheduler] Job 2614: complete_job: Updating cache
[2017-06-01 23:08:28,160: INFO/job_scheduler] Job 2614 completing (errored=False, cancelled=False)
[2017-06-01 23:08:28,187: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:08:29,322: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:08:29,324: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:08:29,350: INFO/command_plan] add_jobs: created Job 2615 (Test for host connectivity)
[2017-06-01 23:08:29,433: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:08:29,475: INFO/job_scheduler] Job 2615: RunJobThread.run
[2017-06-01 23:08:31,631: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:08:32,943: INFO/job_scheduler] Job 2615 finished 1 steps successfully
[2017-06-01 23:08:32,976: INFO/job_scheduler] Job 2615: complete_job: Updating cache
[2017-06-01 23:08:32,977: INFO/job_scheduler] Job 2615 completing (errored=False, cancelled=False)
[2017-06-01 23:08:33,001: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:08:34,181: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:08:34,183: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:08:34,192: INFO/command_plan] add_jobs: created Job 2616 (Test for host connectivity)
[2017-06-01 23:08:34,219: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:08:34,234: INFO/job_scheduler] Job 2616: RunJobThread.run
[2017-06-01 23:08:36,382: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:08:37,647: INFO/job_scheduler] Job 2616 finished 1 steps successfully
[2017-06-01 23:08:37,700: INFO/job_scheduler] Job 2616: complete_job: Updating cache
[2017-06-01 23:08:37,701: INFO/job_scheduler] Job 2616 completing (errored=False, cancelled=False)
[2017-06-01 23:08:37,738: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:08:38,923: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:08:38,923: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:08:38,934: INFO/command_plan] add_jobs: created Job 2617 (Test for host connectivity)
[2017-06-01 23:08:39,015: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:08:39,022: INFO/job_scheduler] Job 2617: RunJobThread.run
[2017-06-01 23:08:41,165: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:08:42,489: INFO/job_scheduler] Job 2617 finished 1 steps successfully
[2017-06-01 23:08:42,541: INFO/job_scheduler] Job 2617: complete_job: Updating cache
[2017-06-01 23:08:42,541: INFO/job_scheduler] Job 2617 completing (errored=False, cancelled=False)
[2017-06-01 23:08:42,576: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:08:43,735: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:08:43,736: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:08:43,797: INFO/command_plan] add_jobs: created Job 2618 (Test for host connectivity)
[2017-06-01 23:08:43,858: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:08:43,880: INFO/job_scheduler] Job 2618: RunJobThread.run
[2017-06-01 23:08:46,042: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:08:47,399: INFO/job_scheduler] Job 2618 finished 1 steps successfully
[2017-06-01 23:08:47,458: INFO/job_scheduler] Job 2618: complete_job: Updating cache
[2017-06-01 23:08:47,459: INFO/job_scheduler] Job 2618 completing (errored=False, cancelled=False)
[2017-06-01 23:08:47,491: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:08:48,617: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:08:48,618: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:08:48,634: INFO/command_plan] add_jobs: created Job 2619 (Test for host connectivity)
[2017-06-01 23:08:48,755: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:08:48,768: INFO/job_scheduler] Job 2619: RunJobThread.run
[2017-06-01 23:08:50,939: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:08:52,257: INFO/job_scheduler] Job 2619 finished 1 steps successfully
[2017-06-01 23:08:52,282: INFO/job_scheduler] Job 2619: complete_job: Updating cache
[2017-06-01 23:08:52,282: INFO/job_scheduler] Job 2619 completing (errored=False, cancelled=False)
[2017-06-01 23:08:52,333: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:08:53,477: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:08:53,479: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:08:53,490: INFO/command_plan] add_jobs: created Job 2620 (Test for host connectivity)
[2017-06-01 23:08:53,603: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:08:53,613: INFO/job_scheduler] Job 2620: RunJobThread.run
[2017-06-01 23:08:55,761: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:08:57,081: INFO/job_scheduler] Job 2620 finished 1 steps successfully
[2017-06-01 23:08:57,140: INFO/job_scheduler] Job 2620: complete_job: Updating cache
[2017-06-01 23:08:57,140: INFO/job_scheduler] Job 2620 completing (errored=False, cancelled=False)
[2017-06-01 23:08:57,165: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:08:58,292: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:08:58,293: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:08:58,307: INFO/command_plan] add_jobs: created Job 2621 (Test for host connectivity)
[2017-06-01 23:08:58,429: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:08:58,441: INFO/job_scheduler] Job 2621: RunJobThread.run
[2017-06-01 23:09:00,553: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:09:01,867: INFO/job_scheduler] Job 2621 finished 1 steps successfully
[2017-06-01 23:09:01,889: INFO/job_scheduler] Job 2621: complete_job: Updating cache
[2017-06-01 23:09:01,889: INFO/job_scheduler] Job 2621 completing (errored=False, cancelled=False)
[2017-06-01 23:09:01,914: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:09:03,074: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:09:03,074: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:09:03,099: INFO/command_plan] add_jobs: created Job 2622 (Test for host connectivity)
[2017-06-01 23:09:03,139: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:09:03,162: INFO/job_scheduler] Job 2622: RunJobThread.run
[2017-06-01 23:09:05,350: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:09:06,665: INFO/job_scheduler] Job 2622 finished 1 steps successfully
[2017-06-01 23:09:06,727: INFO/job_scheduler] Job 2622: complete_job: Updating cache
[2017-06-01 23:09:06,727: INFO/job_scheduler] Job 2622 completing (errored=False, cancelled=False)
[2017-06-01 23:09:06,766: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:09:07,885: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:09:07,887: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:09:07,893: INFO/command_plan] add_jobs: created Job 2623 (Test for host connectivity)
[2017-06-01 23:09:07,949: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:09:07,970: INFO/job_scheduler] Job 2623: RunJobThread.run
[2017-06-01 23:09:10,181: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:09:11,530: INFO/job_scheduler] Job 2623 finished 1 steps successfully
[2017-06-01 23:09:11,584: INFO/job_scheduler] Job 2623: complete_job: Updating cache
[2017-06-01 23:09:11,584: INFO/job_scheduler] Job 2623 completing (errored=False, cancelled=False)
[2017-06-01 23:09:11,609: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:09:12,757: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:09:12,759: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:09:12,773: INFO/command_plan] add_jobs: created Job 2624 (Test for host connectivity)
[2017-06-01 23:09:12,852: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:09:12,866: INFO/job_scheduler] Job 2624: RunJobThread.run
[2017-06-01 23:09:15,011: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:09:16,327: INFO/job_scheduler] Job 2624 finished 1 steps successfully
[2017-06-01 23:09:16,388: INFO/job_scheduler] Job 2624: complete_job: Updating cache
[2017-06-01 23:09:16,388: INFO/job_scheduler] Job 2624 completing (errored=False, cancelled=False)
[2017-06-01 23:09:16,413: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:09:17,558: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:09:17,558: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:09:17,571: INFO/command_plan] add_jobs: created Job 2625 (Test for host connectivity)
[2017-06-01 23:09:17,605: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:09:17,645: INFO/job_scheduler] Job 2625: RunJobThread.run
[2017-06-01 23:09:19,810: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:09:21,162: INFO/job_scheduler] Job 2625 finished 1 steps successfully
[2017-06-01 23:09:21,220: INFO/job_scheduler] Job 2625: complete_job: Updating cache
[2017-06-01 23:09:21,221: INFO/job_scheduler] Job 2625 completing (errored=False, cancelled=False)
[2017-06-01 23:09:21,247: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:09:22,398: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:09:22,398: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:09:22,427: INFO/command_plan] add_jobs: created Job 2626 (Test for host connectivity)
[2017-06-01 23:09:22,530: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:09:22,554: INFO/job_scheduler] Job 2626: RunJobThread.run
[2017-06-01 23:09:24,694: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:09:26,013: INFO/job_scheduler] Job 2626 finished 1 steps successfully
[2017-06-01 23:09:26,045: INFO/job_scheduler] Job 2626: complete_job: Updating cache
[2017-06-01 23:09:26,045: INFO/job_scheduler] Job 2626 completing (errored=False, cancelled=False)
[2017-06-01 23:09:26,087: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:09:27,201: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:09:27,201: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:09:27,219: INFO/command_plan] add_jobs: created Job 2627 (Test for host connectivity)
[2017-06-01 23:09:27,291: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:09:27,297: INFO/job_scheduler] Job 2627: RunJobThread.run
[2017-06-01 23:09:29,427: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:09:30,781: INFO/job_scheduler] Job 2627 finished 1 steps successfully
[2017-06-01 23:09:30,844: INFO/job_scheduler] Job 2627: complete_job: Updating cache
[2017-06-01 23:09:30,844: INFO/job_scheduler] Job 2627 completing (errored=False, cancelled=False)
[2017-06-01 23:09:30,878: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:09:31,999: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:09:32,000: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:09:32,010: INFO/command_plan] add_jobs: created Job 2628 (Test for host connectivity)
[2017-06-01 23:09:32,051: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:09:32,108: INFO/job_scheduler] Job 2628: RunJobThread.run
[2017-06-01 23:09:34,273: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:09:35,582: INFO/job_scheduler] Job 2628 finished 1 steps successfully
[2017-06-01 23:09:35,593: INFO/job_scheduler] Job 2628: complete_job: Updating cache
[2017-06-01 23:09:35,594: INFO/job_scheduler] Job 2628 completing (errored=False, cancelled=False)
[2017-06-01 23:09:35,628: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:09:36,776: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:09:36,779: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:09:36,795: INFO/command_plan] add_jobs: created Job 2629 (Test for host connectivity)
[2017-06-01 23:09:36,870: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:09:36,875: INFO/job_scheduler] Job 2629: RunJobThread.run
[2017-06-01 23:09:39,062: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:09:40,328: INFO/job_scheduler] Job 2629 finished 1 steps successfully
[2017-06-01 23:09:40,391: INFO/job_scheduler] Job 2629: complete_job: Updating cache
[2017-06-01 23:09:40,392: INFO/job_scheduler] Job 2629 completing (errored=False, cancelled=False)
[2017-06-01 23:09:40,435: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:09:41,560: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:09:41,562: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:09:41,575: INFO/command_plan] add_jobs: created Job 2630 (Test for host connectivity)
[2017-06-01 23:09:41,610: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:09:41,686: INFO/job_scheduler] Job 2630: RunJobThread.run
[2017-06-01 23:09:43,882: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:09:45,203: INFO/job_scheduler] Job 2630 finished 1 steps successfully
[2017-06-01 23:09:45,240: INFO/job_scheduler] Job 2630: complete_job: Updating cache
[2017-06-01 23:09:45,240: INFO/job_scheduler] Job 2630 completing (errored=False, cancelled=False)
[2017-06-01 23:09:45,275: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:09:46,433: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:09:46,434: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:09:46,450: INFO/command_plan] add_jobs: created Job 2631 (Test for host connectivity)
[2017-06-01 23:09:46,494: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:09:46,519: INFO/job_scheduler] Job 2631: RunJobThread.run
[2017-06-01 23:09:48,746: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:09:50,069: INFO/job_scheduler] Job 2631 finished 1 steps successfully
[2017-06-01 23:09:50,089: INFO/job_scheduler] Job 2631: complete_job: Updating cache
[2017-06-01 23:09:50,089: INFO/job_scheduler] Job 2631 completing (errored=False, cancelled=False)
[2017-06-01 23:09:50,117: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:09:51,270: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:09:51,272: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:09:51,281: INFO/command_plan] add_jobs: created Job 2632 (Test for host connectivity)
[2017-06-01 23:09:51,341: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:09:51,370: INFO/job_scheduler] Job 2632: RunJobThread.run
[2017-06-01 23:09:53,549: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:09:54,909: INFO/job_scheduler] Job 2632 finished 1 steps successfully
[2017-06-01 23:09:54,922: INFO/job_scheduler] Job 2632: complete_job: Updating cache
[2017-06-01 23:09:54,922: INFO/job_scheduler] Job 2632 completing (errored=False, cancelled=False)
[2017-06-01 23:09:54,947: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:09:56,079: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:09:56,081: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:09:56,097: INFO/command_plan] add_jobs: created Job 2633 (Test for host connectivity)
[2017-06-01 23:09:56,190: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:09:56,206: INFO/job_scheduler] Job 2633: RunJobThread.run
[2017-06-01 23:09:58,377: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:09:59,700: INFO/job_scheduler] Job 2633 finished 1 steps successfully
[2017-06-01 23:09:59,729: INFO/job_scheduler] Job 2633: complete_job: Updating cache
[2017-06-01 23:09:59,729: INFO/job_scheduler] Job 2633 completing (errored=False, cancelled=False)
[2017-06-01 23:09:59,764: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:10:00,908: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:10:00,908: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:10:00,921: INFO/command_plan] add_jobs: created Job 2634 (Test for host connectivity)
[2017-06-01 23:10:00,980: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:10:01,050: INFO/job_scheduler] Job 2634: RunJobThread.run
[2017-06-01 23:10:03,157: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:10:04,472: INFO/job_scheduler] Job 2634 finished 1 steps successfully
[2017-06-01 23:10:04,495: INFO/job_scheduler] Job 2634: complete_job: Updating cache
[2017-06-01 23:10:04,495: INFO/job_scheduler] Job 2634 completing (errored=False, cancelled=False)
[2017-06-01 23:10:04,552: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:10:05,760: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:10:05,772: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:10:05,811: INFO/command_plan] add_jobs: created Job 2635 (Test for host connectivity)
[2017-06-01 23:10:05,913: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:10:05,935: INFO/job_scheduler] Job 2635: RunJobThread.run
[2017-06-01 23:10:08,084: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:10:09,401: INFO/job_scheduler] Job 2635 finished 1 steps successfully
[2017-06-01 23:10:09,461: INFO/job_scheduler] Job 2635: complete_job: Updating cache
[2017-06-01 23:10:09,461: INFO/job_scheduler] Job 2635 completing (errored=False, cancelled=False)
[2017-06-01 23:10:09,512: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:10:10,663: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:10:10,665: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:10:10,679: INFO/command_plan] add_jobs: created Job 2636 (Test for host connectivity)
[2017-06-01 23:10:10,786: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:10:10,793: INFO/job_scheduler] Job 2636: RunJobThread.run
[2017-06-01 23:10:12,960: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:10:14,289: INFO/job_scheduler] Job 2636 finished 1 steps successfully
[2017-06-01 23:10:14,344: INFO/job_scheduler] Job 2636: complete_job: Updating cache
[2017-06-01 23:10:14,344: INFO/job_scheduler] Job 2636 completing (errored=False, cancelled=False)
[2017-06-01 23:10:14,379: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:10:15,523: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:10:15,523: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:10:15,545: INFO/command_plan] add_jobs: created Job 2637 (Test for host connectivity)
[2017-06-01 23:10:15,599: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:10:15,602: INFO/job_scheduler] Job 2637: RunJobThread.run
[2017-06-01 23:10:17,759: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:10:19,074: INFO/job_scheduler] Job 2637 finished 1 steps successfully
[2017-06-01 23:10:19,093: INFO/job_scheduler] Job 2637: complete_job: Updating cache
[2017-06-01 23:10:19,094: INFO/job_scheduler] Job 2637 completing (errored=False, cancelled=False)
[2017-06-01 23:10:19,118: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:10:20,281: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:10:20,283: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:10:20,312: INFO/command_plan] add_jobs: created Job 2638 (Test for host connectivity)
[2017-06-01 23:10:20,338: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:10:20,360: INFO/job_scheduler] Job 2638: RunJobThread.run
[2017-06-01 23:10:22,594: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:10:23,914: INFO/job_scheduler] Job 2638 finished 1 steps successfully
[2017-06-01 23:10:23,925: INFO/job_scheduler] Job 2638: complete_job: Updating cache
[2017-06-01 23:10:23,926: INFO/job_scheduler] Job 2638 completing (errored=False, cancelled=False)
[2017-06-01 23:10:23,951: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:10:25,092: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:10:25,094: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:10:25,117: INFO/command_plan] add_jobs: created Job 2639 (Test for host connectivity)
[2017-06-01 23:10:25,168: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:10:25,174: INFO/job_scheduler] Job 2639: RunJobThread.run
[2017-06-01 23:10:27,384: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:10:28,702: INFO/job_scheduler] Job 2639 finished 1 steps successfully
[2017-06-01 23:10:28,724: INFO/job_scheduler] Job 2639: complete_job: Updating cache
[2017-06-01 23:10:28,725: INFO/job_scheduler] Job 2639 completing (errored=False, cancelled=False)
[2017-06-01 23:10:28,750: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:10:29,862: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:10:29,864: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:10:29,874: INFO/command_plan] add_jobs: created Job 2640 (Test for host connectivity)
[2017-06-01 23:10:29,918: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:10:29,982: INFO/job_scheduler] Job 2640: RunJobThread.run
[2017-06-01 23:10:32,160: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:10:33,492: INFO/job_scheduler] Job 2640 finished 1 steps successfully
[2017-06-01 23:10:33,557: INFO/job_scheduler] Job 2640: complete_job: Updating cache
[2017-06-01 23:10:33,557: INFO/job_scheduler] Job 2640 completing (errored=False, cancelled=False)
[2017-06-01 23:10:33,597: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:10:34,723: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:10:34,724: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:10:34,740: INFO/command_plan] add_jobs: created Job 2641 (Test for host connectivity)
[2017-06-01 23:10:34,822: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:10:34,870: INFO/job_scheduler] Job 2641: RunJobThread.run
[2017-06-01 23:10:37,054: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:10:38,406: INFO/job_scheduler] Job 2641 finished 1 steps successfully
[2017-06-01 23:10:38,431: INFO/job_scheduler] Job 2641: complete_job: Updating cache
[2017-06-01 23:10:38,431: INFO/job_scheduler] Job 2641 completing (errored=False, cancelled=False)
[2017-06-01 23:10:38,484: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:10:39,673: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:10:39,673: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:10:39,734: INFO/command_plan] add_jobs: created Job 2642 (Test for host connectivity)
[2017-06-01 23:10:39,763: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:10:39,779: INFO/job_scheduler] Job 2642: RunJobThread.run
[2017-06-01 23:10:41,917: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:10:43,220: INFO/job_scheduler] Job 2642 finished 1 steps successfully
[2017-06-01 23:10:43,247: INFO/job_scheduler] Job 2642: complete_job: Updating cache
[2017-06-01 23:10:43,247: INFO/job_scheduler] Job 2642 completing (errored=False, cancelled=False)
[2017-06-01 23:10:43,283: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:10:44,440: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:10:44,441: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:10:44,456: INFO/command_plan] add_jobs: created Job 2643 (Test for host connectivity)
[2017-06-01 23:10:44,501: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:10:44,517: INFO/job_scheduler] Job 2643: RunJobThread.run
[2017-06-01 23:10:46,721: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:10:48,041: INFO/job_scheduler] Job 2643 finished 1 steps successfully
[2017-06-01 23:10:48,070: INFO/job_scheduler] Job 2643: complete_job: Updating cache
[2017-06-01 23:10:48,071: INFO/job_scheduler] Job 2643 completing (errored=False, cancelled=False)
[2017-06-01 23:10:48,107: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:10:49,264: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:10:49,266: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:10:49,279: INFO/command_plan] add_jobs: created Job 2644 (Test for host connectivity)
[2017-06-01 23:10:49,346: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:10:49,373: INFO/job_scheduler] Job 2644: RunJobThread.run
[2017-06-01 23:10:51,581: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:10:52,897: INFO/job_scheduler] Job 2644 finished 1 steps successfully
[2017-06-01 23:10:52,920: INFO/job_scheduler] Job 2644: complete_job: Updating cache
[2017-06-01 23:10:52,920: INFO/job_scheduler] Job 2644 completing (errored=False, cancelled=False)
[2017-06-01 23:10:53,001: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:10:54,170: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:10:54,170: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:10:54,178: INFO/command_plan] add_jobs: created Job 2645 (Test for host connectivity)
[2017-06-01 23:10:54,256: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:10:54,262: INFO/job_scheduler] Job 2645: RunJobThread.run
[2017-06-01 23:10:56,416: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:10:57,729: INFO/job_scheduler] Job 2645 finished 1 steps successfully
[2017-06-01 23:10:57,769: INFO/job_scheduler] Job 2645: complete_job: Updating cache
[2017-06-01 23:10:57,769: INFO/job_scheduler] Job 2645 completing (errored=False, cancelled=False)
[2017-06-01 23:10:57,795: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:10:58,931: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:10:58,933: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:10:58,943: INFO/command_plan] add_jobs: created Job 2646 (Test for host connectivity)
[2017-06-01 23:10:59,026: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:10:59,053: INFO/job_scheduler] Job 2646: RunJobThread.run
[2017-06-01 23:11:01,224: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:11:02,591: INFO/job_scheduler] Job 2646 finished 1 steps successfully
[2017-06-01 23:11:02,601: INFO/job_scheduler] Job 2646: complete_job: Updating cache
[2017-06-01 23:11:02,601: INFO/job_scheduler] Job 2646 completing (errored=False, cancelled=False)
[2017-06-01 23:11:02,675: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:11:03,815: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:11:03,815: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:11:03,835: INFO/command_plan] add_jobs: created Job 2647 (Test for host connectivity)
[2017-06-01 23:11:03,926: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:11:03,964: INFO/job_scheduler] Job 2647: RunJobThread.run
[2017-06-01 23:11:06,126: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:11:07,446: INFO/job_scheduler] Job 2647 finished 1 steps successfully
[2017-06-01 23:11:07,467: INFO/job_scheduler] Job 2647: complete_job: Updating cache
[2017-06-01 23:11:07,467: INFO/job_scheduler] Job 2647 completing (errored=False, cancelled=False)
[2017-06-01 23:11:07,503: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:11:08,681: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:11:08,682: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:11:08,691: INFO/command_plan] add_jobs: created Job 2648 (Test for host connectivity)
[2017-06-01 23:11:08,732: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:11:08,735: INFO/job_scheduler] Job 2648: RunJobThread.run
[2017-06-01 23:11:10,914: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:11:12,227: INFO/job_scheduler] Job 2648 finished 1 steps successfully
[2017-06-01 23:11:12,284: INFO/job_scheduler] Job 2648: complete_job: Updating cache
[2017-06-01 23:11:12,284: INFO/job_scheduler] Job 2648 completing (errored=False, cancelled=False)
[2017-06-01 23:11:12,317: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:11:13,442: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:11:13,443: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:11:13,469: INFO/command_plan] add_jobs: created Job 2649 (Test for host connectivity)
[2017-06-01 23:11:13,527: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:11:13,584: INFO/job_scheduler] Job 2649: RunJobThread.run
[2017-06-01 23:11:15,770: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:11:17,038: INFO/job_scheduler] Job 2649 finished 1 steps successfully
[2017-06-01 23:11:17,073: INFO/job_scheduler] Job 2649: complete_job: Updating cache
[2017-06-01 23:11:17,073: INFO/job_scheduler] Job 2649 completing (errored=False, cancelled=False)
[2017-06-01 23:11:17,150: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:11:18,301: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:11:18,302: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:11:18,314: INFO/command_plan] add_jobs: created Job 2650 (Test for host connectivity)
[2017-06-01 23:11:18,387: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:11:18,439: INFO/job_scheduler] Job 2650: RunJobThread.run
[2017-06-01 23:11:20,626: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:11:21,930: INFO/job_scheduler] Job 2650 finished 1 steps successfully
[2017-06-01 23:11:21,955: INFO/job_scheduler] Job 2650: complete_job: Updating cache
[2017-06-01 23:11:21,955: INFO/job_scheduler] Job 2650 completing (errored=False, cancelled=False)
[2017-06-01 23:11:22,005: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:11:23,129: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:11:23,130: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:11:23,138: INFO/command_plan] add_jobs: created Job 2651 (Test for host connectivity)
[2017-06-01 23:11:23,202: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:11:23,213: INFO/job_scheduler] Job 2651: RunJobThread.run
[2017-06-01 23:11:25,352: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:11:26,706: INFO/job_scheduler] Job 2651 finished 1 steps successfully
[2017-06-01 23:11:26,738: INFO/job_scheduler] Job 2651: complete_job: Updating cache
[2017-06-01 23:11:26,738: INFO/job_scheduler] Job 2651 completing (errored=False, cancelled=False)
[2017-06-01 23:11:26,795: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:11:27,983: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:11:27,985: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:11:27,995: INFO/command_plan] add_jobs: created Job 2652 (Test for host connectivity)
[2017-06-01 23:11:28,027: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:11:28,057: INFO/job_scheduler] Job 2652: RunJobThread.run
[2017-06-01 23:11:30,228: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:11:31,543: INFO/job_scheduler] Job 2652 finished 1 steps successfully
[2017-06-01 23:11:31,579: INFO/job_scheduler] Job 2652: complete_job: Updating cache
[2017-06-01 23:11:31,579: INFO/job_scheduler] Job 2652 completing (errored=False, cancelled=False)
[2017-06-01 23:11:31,613: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:11:32,781: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:11:32,781: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:11:32,798: INFO/command_plan] add_jobs: created Job 2653 (Test for host connectivity)
[2017-06-01 23:11:32,883: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:11:32,907: INFO/job_scheduler] Job 2653: RunJobThread.run
[2017-06-01 23:11:35,204: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:11:36,411: INFO/job_scheduler] Job 2653 finished 1 steps successfully
[2017-06-01 23:11:36,427: INFO/job_scheduler] Job 2653: complete_job: Updating cache
[2017-06-01 23:11:36,427: INFO/job_scheduler] Job 2653 completing (errored=False, cancelled=False)
[2017-06-01 23:11:36,452: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:11:37,618: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:11:37,619: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:11:37,627: INFO/command_plan] add_jobs: created Job 2654 (Test for host connectivity)
[2017-06-01 23:11:37,679: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:11:37,694: INFO/job_scheduler] Job 2654: RunJobThread.run
[2017-06-01 23:11:39,978: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:11:41,331: INFO/job_scheduler] Job 2654 finished 1 steps successfully
[2017-06-01 23:11:41,393: INFO/job_scheduler] Job 2654: complete_job: Updating cache
[2017-06-01 23:11:41,393: INFO/job_scheduler] Job 2654 completing (errored=False, cancelled=False)
[2017-06-01 23:11:41,420: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:11:42,562: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:11:42,563: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:11:42,589: INFO/command_plan] add_jobs: created Job 2655 (Test for host connectivity)
[2017-06-01 23:11:42,640: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:11:42,642: INFO/job_scheduler] Job 2655: RunJobThread.run
[2017-06-01 23:11:44,813: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:11:46,116: INFO/job_scheduler] Job 2655 finished 1 steps successfully
[2017-06-01 23:11:46,151: INFO/job_scheduler] Job 2655: complete_job: Updating cache
[2017-06-01 23:11:46,151: INFO/job_scheduler] Job 2655 completing (errored=False, cancelled=False)
[2017-06-01 23:11:46,218: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:11:47,359: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:11:47,360: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:11:47,376: INFO/command_plan] add_jobs: created Job 2656 (Test for host connectivity)
[2017-06-01 23:11:47,440: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:11:47,467: INFO/job_scheduler] Job 2656: RunJobThread.run
[2017-06-01 23:11:49,640: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:11:50,963: INFO/job_scheduler] Job 2656 finished 1 steps successfully
[2017-06-01 23:11:50,983: INFO/job_scheduler] Job 2656: complete_job: Updating cache
[2017-06-01 23:11:50,983: INFO/job_scheduler] Job 2656 completing (errored=False, cancelled=False)
[2017-06-01 23:11:51,135: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:11:52,340: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:11:52,341: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:11:52,350: INFO/command_plan] add_jobs: created Job 2657 (Test for host connectivity)
[2017-06-01 23:11:52,405: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:11:52,414: INFO/job_scheduler] Job 2657: RunJobThread.run
[2017-06-01 23:11:54,579: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:11:55,904: INFO/job_scheduler] Job 2657 finished 1 steps successfully
[2017-06-01 23:11:55,950: INFO/job_scheduler] Job 2657: complete_job: Updating cache
[2017-06-01 23:11:55,950: INFO/job_scheduler] Job 2657 completing (errored=False, cancelled=False)
[2017-06-01 23:11:56,025: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:11:57,176: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:11:57,176: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:11:57,194: INFO/command_plan] add_jobs: created Job 2658 (Test for host connectivity)
[2017-06-01 23:11:57,249: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:11:57,261: INFO/job_scheduler] Job 2658: RunJobThread.run
[2017-06-01 23:11:59,441: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:12:00,758: INFO/job_scheduler] Job 2658 finished 1 steps successfully
[2017-06-01 23:12:00,824: INFO/job_scheduler] Job 2658: complete_job: Updating cache
[2017-06-01 23:12:00,824: INFO/job_scheduler] Job 2658 completing (errored=False, cancelled=False)
[2017-06-01 23:12:00,863: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:12:02,046: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:12:02,046: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:12:02,057: INFO/command_plan] add_jobs: created Job 2659 (Test for host connectivity)
[2017-06-01 23:12:02,079: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:12:02,113: INFO/job_scheduler] Job 2659: RunJobThread.run
[2017-06-01 23:12:04,305: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:12:05,730: INFO/job_scheduler] Job 2659 finished 1 steps successfully
[2017-06-01 23:12:05,757: INFO/job_scheduler] Job 2659: complete_job: Updating cache
[2017-06-01 23:12:05,757: INFO/job_scheduler] Job 2659 completing (errored=False, cancelled=False)
[2017-06-01 23:12:05,800: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:12:06,974: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:12:06,974: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:12:06,981: INFO/command_plan] add_jobs: created Job 2660 (Test for host connectivity)
[2017-06-01 23:12:07,017: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:12:07,029: INFO/job_scheduler] Job 2660: RunJobThread.run
[2017-06-01 23:12:09,223: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:12:10,575: INFO/job_scheduler] Job 2660 finished 1 steps successfully
[2017-06-01 23:12:10,589: INFO/job_scheduler] Job 2660: complete_job: Updating cache
[2017-06-01 23:12:10,590: INFO/job_scheduler] Job 2660 completing (errored=False, cancelled=False)
[2017-06-01 23:12:10,648: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:12:11,787: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:12:11,789: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:12:11,798: INFO/command_plan] add_jobs: created Job 2661 (Test for host connectivity)
[2017-06-01 23:12:11,824: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:12:11,835: INFO/job_scheduler] Job 2661: RunJobThread.run
[2017-06-01 23:12:14,043: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:12:15,357: INFO/job_scheduler] Job 2661 finished 1 steps successfully
[2017-06-01 23:12:15,372: INFO/job_scheduler] Job 2661: complete_job: Updating cache
[2017-06-01 23:12:15,372: INFO/job_scheduler] Job 2661 completing (errored=False, cancelled=False)
[2017-06-01 23:12:15,399: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:12:16,548: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:12:16,549: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:12:16,571: INFO/command_plan] add_jobs: created Job 2662 (Test for host connectivity)
[2017-06-01 23:12:16,623: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:12:16,665: INFO/job_scheduler] Job 2662: RunJobThread.run
[2017-06-01 23:12:18,857: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:12:20,170: INFO/job_scheduler] Job 2662 finished 1 steps successfully
[2017-06-01 23:12:20,197: INFO/job_scheduler] Job 2662: complete_job: Updating cache
[2017-06-01 23:12:20,197: INFO/job_scheduler] Job 2662 completing (errored=False, cancelled=False)
[2017-06-01 23:12:20,231: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:12:21,397: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:12:21,398: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:12:21,424: INFO/command_plan] add_jobs: created Job 2663 (Test for host connectivity)
[2017-06-01 23:12:21,514: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:12:21,519: INFO/job_scheduler] Job 2663: RunJobThread.run
[2017-06-01 23:12:23,662: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:12:24,978: INFO/job_scheduler] Job 2663 finished 1 steps successfully
[2017-06-01 23:12:25,038: INFO/job_scheduler] Job 2663: complete_job: Updating cache
[2017-06-01 23:12:25,038: INFO/job_scheduler] Job 2663 completing (errored=False, cancelled=False)
[2017-06-01 23:12:25,115: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:12:26,252: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:12:26,253: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:12:26,279: INFO/command_plan] add_jobs: created Job 2664 (Test for host connectivity)
[2017-06-01 23:12:26,335: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:12:26,370: INFO/job_scheduler] Job 2664: RunJobThread.run
[2017-06-01 23:12:28,547: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:12:29,890: INFO/job_scheduler] Job 2664 finished 1 steps successfully
[2017-06-01 23:12:29,912: INFO/job_scheduler] Job 2664: complete_job: Updating cache
[2017-06-01 23:12:29,912: INFO/job_scheduler] Job 2664 completing (errored=False, cancelled=False)
[2017-06-01 23:12:29,948: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:12:31,070: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:12:31,070: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:12:31,078: INFO/command_plan] add_jobs: created Job 2665 (Test for host connectivity)
[2017-06-01 23:12:31,176: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:12:31,180: INFO/job_scheduler] Job 2665: RunJobThread.run
[2017-06-01 23:12:33,349: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:12:34,701: INFO/job_scheduler] Job 2665 finished 1 steps successfully
[2017-06-01 23:12:34,761: INFO/job_scheduler] Job 2665: complete_job: Updating cache
[2017-06-01 23:12:34,762: INFO/job_scheduler] Job 2665 completing (errored=False, cancelled=False)
[2017-06-01 23:12:34,813: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:12:35,953: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:12:35,955: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:12:35,978: INFO/command_plan] add_jobs: created Job 2666 (Test for host connectivity)
[2017-06-01 23:12:36,030: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:12:36,072: INFO/job_scheduler] Job 2666: RunJobThread.run
[2017-06-01 23:12:38,216: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:12:39,528: INFO/job_scheduler] Job 2666 finished 1 steps successfully
[2017-06-01 23:12:39,553: INFO/job_scheduler] Job 2666: complete_job: Updating cache
[2017-06-01 23:12:39,553: INFO/job_scheduler] Job 2666 completing (errored=False, cancelled=False)
[2017-06-01 23:12:39,619: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:12:40,764: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:12:40,766: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:12:40,778: INFO/command_plan] add_jobs: created Job 2667 (Test for host connectivity)
[2017-06-01 23:12:40,848: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:12:40,896: INFO/job_scheduler] Job 2667: RunJobThread.run
[2017-06-01 23:12:43,037: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:12:44,352: INFO/job_scheduler] Job 2667 finished 1 steps successfully
[2017-06-01 23:12:44,410: INFO/job_scheduler] Job 2667: complete_job: Updating cache
[2017-06-01 23:12:44,411: INFO/job_scheduler] Job 2667 completing (errored=False, cancelled=False)
[2017-06-01 23:12:44,437: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:12:45,572: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:12:45,574: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:12:45,585: INFO/command_plan] add_jobs: created Job 2668 (Test for host connectivity)
[2017-06-01 23:12:45,617: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:12:45,621: INFO/job_scheduler] Job 2668: RunJobThread.run
[2017-06-01 23:12:47,812: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:12:49,133: INFO/job_scheduler] Job 2668 finished 1 steps successfully
[2017-06-01 23:12:49,193: INFO/job_scheduler] Job 2668: complete_job: Updating cache
[2017-06-01 23:12:49,193: INFO/job_scheduler] Job 2668 completing (errored=False, cancelled=False)
[2017-06-01 23:12:49,227: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:12:50,380: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:12:50,382: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:12:50,393: INFO/command_plan] add_jobs: created Job 2669 (Test for host connectivity)
[2017-06-01 23:12:50,474: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:12:50,476: INFO/job_scheduler] Job 2669: RunJobThread.run
[2017-06-01 23:12:52,653: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:12:53,966: INFO/job_scheduler] Job 2669 finished 1 steps successfully
[2017-06-01 23:12:53,984: INFO/job_scheduler] Job 2669: complete_job: Updating cache
[2017-06-01 23:12:53,984: INFO/job_scheduler] Job 2669 completing (errored=False, cancelled=False)
[2017-06-01 23:12:54,066: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:12:55,242: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:12:55,242: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:12:55,270: INFO/command_plan] add_jobs: created Job 2670 (Test for host connectivity)
[2017-06-01 23:12:55,343: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:12:55,353: INFO/job_scheduler] Job 2670: RunJobThread.run
[2017-06-01 23:12:57,600: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:12:58,917: INFO/job_scheduler] Job 2670 finished 1 steps successfully
[2017-06-01 23:12:58,958: INFO/job_scheduler] Job 2670: complete_job: Updating cache
[2017-06-01 23:12:58,959: INFO/job_scheduler] Job 2670 completing (errored=False, cancelled=False)
[2017-06-01 23:12:58,983: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:13:00,124: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:13:00,126: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:13:00,133: INFO/command_plan] add_jobs: created Job 2671 (Test for host connectivity)
[2017-06-01 23:13:00,156: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:13:00,158: INFO/job_scheduler] Job 2671: RunJobThread.run
[2017-06-01 23:13:02,345: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:13:03,701: INFO/job_scheduler] Job 2671 finished 1 steps successfully
[2017-06-01 23:13:03,741: INFO/job_scheduler] Job 2671: complete_job: Updating cache
[2017-06-01 23:13:03,741: INFO/job_scheduler] Job 2671 completing (errored=False, cancelled=False)
[2017-06-01 23:13:03,768: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:13:04,921: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:13:04,923: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:13:04,932: INFO/command_plan] add_jobs: created Job 2672 (Test for host connectivity)
[2017-06-01 23:13:04,986: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:13:05,051: INFO/job_scheduler] Job 2672: RunJobThread.run
[2017-06-01 23:13:07,157: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:13:08,471: INFO/job_scheduler] Job 2672 finished 1 steps successfully
[2017-06-01 23:13:08,508: INFO/job_scheduler] Job 2672: complete_job: Updating cache
[2017-06-01 23:13:08,508: INFO/job_scheduler] Job 2672 completing (errored=False, cancelled=False)
[2017-06-01 23:13:08,532: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:13:09,688: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:13:09,690: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:13:09,699: INFO/command_plan] add_jobs: created Job 2673 (Test for host connectivity)
[2017-06-01 23:13:09,769: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:13:09,799: INFO/job_scheduler] Job 2673: RunJobThread.run
[2017-06-01 23:13:11,958: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:13:13,323: INFO/job_scheduler] Job 2673 finished 1 steps successfully
[2017-06-01 23:13:13,382: INFO/job_scheduler] Job 2673: complete_job: Updating cache
[2017-06-01 23:13:13,382: INFO/job_scheduler] Job 2673 completing (errored=False, cancelled=False)
[2017-06-01 23:13:13,407: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:13:14,543: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:13:14,545: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:13:14,558: INFO/command_plan] add_jobs: created Job 2674 (Test for host connectivity)
[2017-06-01 23:13:14,627: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:13:14,642: INFO/job_scheduler] Job 2674: RunJobThread.run
[2017-06-01 23:13:16,760: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:13:18,060: INFO/job_scheduler] Job 2674 finished 1 steps successfully
[2017-06-01 23:13:18,108: INFO/job_scheduler] Job 2674: complete_job: Updating cache
[2017-06-01 23:13:18,108: INFO/job_scheduler] Job 2674 completing (errored=False, cancelled=False)
[2017-06-01 23:13:18,158: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:13:19,355: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:13:19,355: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:13:19,375: INFO/command_plan] add_jobs: created Job 2675 (Test for host connectivity)
[2017-06-01 23:13:19,438: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:13:19,480: INFO/job_scheduler] Job 2675: RunJobThread.run
[2017-06-01 23:13:21,860: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:13:23,178: INFO/job_scheduler] Job 2675 finished 1 steps successfully
[2017-06-01 23:13:23,239: INFO/job_scheduler] Job 2675: complete_job: Updating cache
[2017-06-01 23:13:23,239: INFO/job_scheduler] Job 2675 completing (errored=False, cancelled=False)
[2017-06-01 23:13:23,266: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:13:24,410: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:13:24,410: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:13:24,422: INFO/command_plan] add_jobs: created Job 2676 (Test for host connectivity)
[2017-06-01 23:13:24,433: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:13:24,436: INFO/job_scheduler] Job 2676: RunJobThread.run
[2017-06-01 23:13:26,589: ERROR/job] Failed configuration check on 'lotus-32vm5': EPEL repository not detected in yum configuration
[2017-06-01 23:13:27,909: INFO/job_scheduler] Job 2676 finished 1 steps successfully
[2017-06-01 23:13:27,930: INFO/job_scheduler] Job 2676: complete_job: Updating cache
[2017-06-01 23:13:27,930: INFO/job_scheduler] Job 2676 completing (errored=False, cancelled=False)
[2017-06-01 23:13:27,962: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:13:29,151: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:13:29,151: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:13:29,163: INFO/command_plan] add_jobs: created Job 2677 (Test for host connectivity)
[2017-06-01 23:13:29,182: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:13:29,192: INFO/job_scheduler] Job 2677: RunJobThread.run
[2017-06-01 23:13:55,809: INFO/job_scheduler] Job 2677 finished 1 steps successfully
[2017-06-01 23:13:55,868: INFO/job_scheduler] Job 2677: complete_job: Updating cache
[2017-06-01 23:13:55,868: INFO/job_scheduler] Job 2677 completing (errored=False, cancelled=False)
[2017-06-01 23:13:55,902: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:14:20,217: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:14:20,218: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:14:20,232: INFO/command_plan] add_jobs: created Job 2678 (Test for host connectivity)
[2017-06-01 23:14:20,282: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:14:20,315: INFO/job_scheduler] Job 2678: RunJobThread.run
[2017-06-01 23:14:42,972: INFO/job_scheduler] Job 2678 finished 1 steps successfully
[2017-06-01 23:14:43,012: INFO/job_scheduler] Job 2678: complete_job: Updating cache
[2017-06-01 23:14:43,012: INFO/job_scheduler] Job 2678 completing (errored=False, cancelled=False)
[2017-06-01 23:14:43,038: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-01 23:14:43,241: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-01 23:14:43,241: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-01 23:14:43,253: INFO/command_plan] add_jobs: created Job 2679 (Test for host connectivity)
[2017-06-01 23:14:43,370: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-01 23:14:43,372: INFO/job_scheduler] Job 2679: RunJobThread.run
[2017-06-01 23:15:06,265: INFO/job_scheduler] Job 2679 finished 1 steps successfully
[2017-06-01 23:15:06,296: INFO/job_scheduler] Job 2679: complete_job: Updating cache
[2017-06-01 23:15:06,296: INFO/job_scheduler] Job 2679 completing (errored=False, cancelled=False)
[2017-06-01 23:15:06,339: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-02 06:06:40,848: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-02 06:06:40,849: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-02 06:06:40,874: INFO/command_plan] add_jobs: created Job 2680 (Test for host connectivity)
[2017-06-02 06:06:40,943: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-02 06:06:40,943: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-02 06:06:40,955: INFO/command_plan] add_jobs: created Job 2681 (Test for host connectivity)
[2017-06-02 06:06:40,973: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-02 06:06:40,987: INFO/job_scheduler] Job 2680: RunJobThread.run
[2017-06-02 06:06:40,991: INFO/job_scheduler] Job 2681: RunJobThread.run
[2017-06-02 06:06:41,001: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-02 06:06:41,100: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-02 06:06:41,101: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-02 06:06:41,131: INFO/command_plan] add_jobs: created Job 2682 (Test for host connectivity)
[2017-06-02 06:06:41,188: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-02 06:06:41,220: INFO/job_scheduler] Job 2682: RunJobThread.run
[2017-06-02 06:06:41,304: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-02 06:06:41,305: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-02 06:06:41,314: INFO/command_plan] add_jobs: created Job 2683 (Test for host connectivity)
[2017-06-02 06:06:41,344: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-02 06:06:41,426: INFO/job_scheduler] Job 2683: RunJobThread.run
[2017-06-02 06:06:59,247: INFO/job_scheduler] Job 2680 finished 1 steps successfully
[2017-06-02 06:06:59,268: INFO/job_scheduler] Job 2680: complete_job: Updating cache
[2017-06-02 06:06:59,268: INFO/job_scheduler] Job 2680 completing (errored=False, cancelled=False)
[2017-06-02 06:06:59,302: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-02 06:07:00,894: INFO/job_scheduler] Job 2681 finished 1 steps successfully
[2017-06-02 06:07:00,934: INFO/job_scheduler] Job 2681: complete_job: Updating cache
[2017-06-02 06:07:00,935: INFO/job_scheduler] Job 2681 completing (errored=False, cancelled=False)
[2017-06-02 06:07:00,973: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-02 06:07:02,052: INFO/job_scheduler] Job 2683 finished 1 steps successfully
[2017-06-02 06:07:02,109: INFO/job_scheduler] Job 2683: complete_job: Updating cache
[2017-06-02 06:07:02,110: INFO/job_scheduler] Job 2683 completing (errored=False, cancelled=False)
[2017-06-02 06:07:02,144: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-02 06:07:02,446: INFO/job_scheduler] Job 2682 finished 1 steps successfully
[2017-06-02 06:07:02,494: INFO/job_scheduler] Job 2682: complete_job: Updating cache
[2017-06-02 06:07:02,494: INFO/job_scheduler] Job 2682 completing (errored=False, cancelled=False)
[2017-06-02 06:07:02,543: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-02 06:07:02,787: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-02 06:07:02,788: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-02 06:07:02,810: INFO/command_plan] add_jobs: created Job 2684 (Test for host connectivity)
[2017-06-02 06:07:02,872: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-02 06:07:02,928: INFO/job_scheduler] Job 2684: RunJobThread.run
[2017-06-02 06:07:02,962: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-02 06:07:02,963: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-02 06:07:02,978: INFO/command_plan] add_jobs: created Job 2685 (Test for host connectivity)
[2017-06-02 06:07:03,031: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-02 06:07:03,058: INFO/job_scheduler] Job 2685: RunJobThread.run
[2017-06-02 06:07:03,190: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-02 06:07:03,191: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-02 06:07:03,271: INFO/command_plan] add_jobs: created Job 2686 (Test for host connectivity)
[2017-06-02 06:07:03,310: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-02 06:07:03,355: INFO/job_scheduler] Job 2686: RunJobThread.run
[2017-06-02 06:07:03,431: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-02 06:07:03,431: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-02 06:07:03,513: INFO/command_plan] add_jobs: created Job 2687 (Test for host connectivity)
[2017-06-02 06:07:03,581: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-02 06:07:03,584: INFO/job_scheduler] Job 2687: RunJobThread.run
[2017-06-02 06:07:19,974: INFO/job_scheduler] Job 2685 finished 1 steps successfully
[2017-06-02 06:07:20,031: INFO/job_scheduler] Job 2685: complete_job: Updating cache
[2017-06-02 06:07:20,031: INFO/job_scheduler] Job 2685 completing (errored=False, cancelled=False)
[2017-06-02 06:07:20,081: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-02 06:07:20,133: INFO/job_scheduler] Job 2684 finished 1 steps successfully
[2017-06-02 06:07:20,152: INFO/job_scheduler] Job 2684: complete_job: Updating cache
[2017-06-02 06:07:20,152: INFO/job_scheduler] Job 2684 completing (errored=False, cancelled=False)
[2017-06-02 06:07:20,256: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-02 06:07:20,712: INFO/job_scheduler] Job 2686 finished 1 steps successfully
[2017-06-02 06:07:20,712: INFO/job_scheduler] Job 2687 finished 1 steps successfully
[2017-06-02 06:07:20,764: INFO/job_scheduler] Job 2686: complete_job: Updating cache
[2017-06-02 06:07:20,764: INFO/job_scheduler] Job 2686 completing (errored=False, cancelled=False)
[2017-06-02 06:07:20,807: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-02 06:07:20,847: INFO/job_scheduler] Job 2687: complete_job: Updating cache
[2017-06-02 06:07:20,848: INFO/job_scheduler] Job 2687 completing (errored=False, cancelled=False)
[2017-06-02 06:07:21,048: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-02 06:07:36,561: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-02 06:07:36,563: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-02 06:07:36,613: INFO/command_plan] add_jobs: created Job 2688 (Test for host connectivity)
[2017-06-02 06:07:36,682: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-02 06:07:36,692: INFO/job_scheduler] Job 2688: RunJobThread.run
[2017-06-02 06:07:36,769: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-02 06:07:36,779: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-02 06:07:36,804: INFO/command_plan] add_jobs: created Job 2689 (Test for host connectivity)
[2017-06-02 06:07:36,837: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-02 06:07:36,906: INFO/job_scheduler] Job 2689: RunJobThread.run
[2017-06-02 06:07:36,958: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-02 06:07:36,958: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-02 06:07:36,982: INFO/command_plan] add_jobs: created Job 2690 (Test for host connectivity)
[2017-06-02 06:07:37,013: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-02 06:07:37,060: INFO/job_scheduler] Job 2690: RunJobThread.run
[2017-06-02 06:07:37,191: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-02 06:07:37,193: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-02 06:07:37,211: INFO/command_plan] add_jobs: created Job 2691 (Test for host connectivity)
[2017-06-02 06:07:37,293: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-02 06:07:37,322: INFO/job_scheduler] Job 2691: RunJobThread.run
[2017-06-02 06:07:53,727: INFO/job_scheduler] Job 2688 finished 1 steps successfully
[2017-06-02 06:07:53,760: INFO/job_scheduler] Job 2688: complete_job: Updating cache
[2017-06-02 06:07:53,760: INFO/job_scheduler] Job 2688 completing (errored=False, cancelled=False)
[2017-06-02 06:07:53,800: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-02 06:07:53,813: INFO/job_scheduler] Job 2689 finished 1 steps successfully
[2017-06-02 06:07:53,849: INFO/job_scheduler] Job 2689: complete_job: Updating cache
[2017-06-02 06:07:53,849: INFO/job_scheduler] Job 2689 completing (errored=False, cancelled=False)
[2017-06-02 06:07:53,882: INFO/job_scheduler] Job 2690 finished 1 steps successfully
[2017-06-02 06:07:53,949: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-02 06:07:53,968: INFO/job_scheduler] Job 2690: complete_job: Updating cache
[2017-06-02 06:07:53,968: INFO/job_scheduler] Job 2690 completing (errored=False, cancelled=False)
[2017-06-02 06:07:54,091: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-02 06:07:54,174: INFO/job_scheduler] Job 2691 finished 1 steps successfully
[2017-06-02 06:07:54,232: INFO/job_scheduler] Job 2691: complete_job: Updating cache
[2017-06-02 06:07:54,234: INFO/job_scheduler] Job 2691 completing (errored=False, cancelled=False)
[2017-06-02 06:07:54,290: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-02 06:07:57,566: INFO/job_scheduler] Getting FQDN for 'lotus-32vm5': lotus-32vm5
lotus-32vm5.lotus.hpdd.lab.intel.com

[2017-06-02 06:07:57,619: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-02 06:07:57,631: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-7 transitioning from undeployed to unconfigured
[2017-06-02 06:07:57,731: INFO/command_plan] Created command 2534 (Deploying agent to host lotus-32vm5.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-06-02 06:07:57,876: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-02 06:07:57,961: INFO/job_scheduler] Job 2692: RunJobThread.run
[2017-06-02 06:07:58,568: INFO/job_scheduler] Getting FQDN for 'lotus-32vm7.lotus.hpdd.lab.intel.com': lotus-32vm7
lotus-32vm7.lotus.hpdd.lab.intel.com

[2017-06-02 06:07:58,596: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-02 06:07:58,599: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-8 transitioning from undeployed to unconfigured
[2017-06-02 06:07:58,697: INFO/command_plan] Created command 2535 (Deploying agent to host lotus-32vm7.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-06-02 06:07:58,866: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-02 06:07:58,933: INFO/job_scheduler] Job 2693: RunJobThread.run
[2017-06-02 06:07:59,395: INFO/job_scheduler] Getting FQDN for 'lotus-32vm8.lotus.hpdd.lab.intel.com': lotus-32vm8
lotus-32vm8.lotus.hpdd.lab.intel.com

[2017-06-02 06:07:59,441: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-02 06:07:59,445: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-9 transitioning from undeployed to unconfigured
[2017-06-02 06:07:59,465: INFO/command_plan] Created command 2536 (Deploying agent to host lotus-32vm8.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-06-02 06:07:59,494: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-02 06:07:59,669: INFO/job_scheduler] Job 2694: RunJobThread.run
[2017-06-02 06:07:59,773: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-02 06:07:59,796: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-02 06:08:00,289: INFO/job_scheduler] Getting FQDN for 'lotus-32vm6.lotus.hpdd.lab.intel.com': lotus-32vm6
lotus-32vm6.lotus.hpdd.lab.intel.com

[2017-06-02 06:08:00,313: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-02 06:08:00,316: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-10 transitioning from undeployed to unconfigured
[2017-06-02 06:08:00,367: INFO/command_plan] Created command 2537 (Deploying agent to host lotus-32vm6.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-06-02 06:08:00,587: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-02 06:08:00,615: INFO/job_scheduler] Job 2695: RunJobThread.run
[2017-06-02 06:08:00,627: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-02 06:08:01,435: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-02 06:08:08,357: ERROR/job_scheduler] Job 2692 step 0 encountered an agent error: Failed to register host lotus-32vm5.lotus.hpdd.lab.intel.com: rc=1
''
'  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  8294    0  8294    0     0  31030      0 --:--:-- --:--:-- --:--:-- 31063
'
[2017-06-02 06:08:08,412: INFO/job_scheduler] Job 2692: complete_job: Updating cache
[2017-06-02 06:08:08,413: INFO/job_scheduler] Job 2692: locked_item 140350883045584 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-02 06:08:08,418: INFO/job_scheduler] Job 2692 completing (errored=True, cancelled=False)
[2017-06-02 06:08:08,535: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-02 06:08:09,122: ERROR/job_scheduler] Job 2693 step 0 encountered an agent error: Failed to register host lotus-32vm7.lotus.hpdd.lab.intel.com: rc=1
''
'  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  8294    0  8294    0     0  44977      0 --:--:-- --:--:-- --:--:-- 45076
'
[2017-06-02 06:08:09,147: INFO/job_scheduler] Job 2693: complete_job: Updating cache
[2017-06-02 06:08:09,147: INFO/job_scheduler] Job 2693: locked_item 140350883554704 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-02 06:08:09,151: INFO/job_scheduler] Job 2693 completing (errored=True, cancelled=False)
[2017-06-02 06:08:09,302: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-02 06:08:09,953: ERROR/job_scheduler] Job 2694 step 0 encountered an agent error: Failed to register host lotus-32vm8.lotus.hpdd.lab.intel.com: rc=1
''
'  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  8294    0  8294    0     0  54726      0 --:--:-- --:--:-- --:--:-- 54927
'
[2017-06-02 06:08:09,979: INFO/job_scheduler] Job 2694: complete_job: Updating cache
[2017-06-02 06:08:09,979: INFO/job_scheduler] Job 2694: locked_item 140350882044368 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-02 06:08:09,982: INFO/job_scheduler] Job 2694 completing (errored=True, cancelled=False)
[2017-06-02 06:08:10,015: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-02 06:08:10,653: ERROR/job_scheduler] Job 2695 step 0 encountered an agent error: Failed to register host lotus-32vm6.lotus.hpdd.lab.intel.com: rc=1
''
'  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  8294    0  8294    0     0  51534      0 --:--:-- --:--:-- --:--:-- 51837
'
[2017-06-02 06:08:10,687: INFO/job_scheduler] Job 2695: complete_job: Updating cache
[2017-06-02 06:08:10,687: INFO/job_scheduler] Job 2695: locked_item 140350884338256 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-02 06:08:10,690: INFO/job_scheduler] Job 2695 completing (errored=True, cancelled=False)
[2017-06-02 06:08:10,714: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-06 20:52:24,080: INFO/chroma_service] Stopping job_scheduler
[2017-06-06 20:52:24,080: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-06-06 20:52:24,080: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-06-06 20:52:24,080: INFO/queue] Stopping ServiceQueue agent_action_runner_rx
[2017-06-06 20:52:24,477: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-06-06 20:52:24,477: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-06-06 20:52:24,477: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-06-06 20:52:24,477: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-06-06 20:52:24,477: INFO/job_scheduler] Stopping...
[2017-06-06 20:52:24,478: INFO/queue] Stopping ServiceQueue job_scheduler_notifications
[2017-06-06 20:52:24,478: INFO/job_scheduler] Joining...
[2017-06-06 20:52:25,473: INFO/job_scheduler] Complete.
[2017-06-06 20:52:25,474: INFO/chroma_service] Joining job_scheduler
[2017-06-06 20:52:25,474: INFO/job_scheduler] Cancelling outstanding jobs...
[2017-06-07 13:56:32,284: INFO/queue] Purged 0 messages from 'job_scheduler_notifications' queue
[2017-06-07 13:56:32,298: INFO/queue] Purged 1 messages from 'agent_action_runner_rx' queue
[2017-06-07 14:03:59,888: INFO/chroma_service] Stopping job_scheduler
[2017-06-07 14:03:59,889: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-06-07 14:03:59,889: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-06-07 14:03:59,889: INFO/queue] Stopping ServiceQueue agent_action_runner_rx
[2017-06-07 14:03:59,937: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-06-07 14:03:59,937: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-06-07 14:03:59,937: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-06-07 14:03:59,937: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-06-07 14:03:59,938: INFO/job_scheduler] Stopping...
[2017-06-07 14:03:59,938: INFO/queue] Stopping ServiceQueue job_scheduler_notifications
[2017-06-07 14:03:59,938: INFO/job_scheduler] Joining...
[2017-06-07 14:04:00,883: INFO/job_scheduler] Complete.
[2017-06-07 14:04:00,883: INFO/chroma_service] Joining job_scheduler
[2017-06-07 14:04:00,884: INFO/job_scheduler] Cancelling outstanding jobs...
[2017-06-07 14:04:00,887: ERROR/chroma_service] Rogue thread still running, exiting hard
[2017-06-07 14:04:00,887: ERROR/chroma_service] ['MainThread', 'Thread-32', 'Thread-55', 'Thread-60', 'Thread-70', 'Thread-56', 'Thread-30', 'Thread-43']
[2017-06-07 14:15:44,255: INFO/queue] Purged 0 messages from 'job_scheduler_notifications' queue
[2017-06-07 14:15:44,267: INFO/queue] Purged 1 messages from 'agent_action_runner_rx' queue
[2017-06-14 15:12:38,773: INFO/chroma_service] Stopping job_scheduler
[2017-06-14 15:12:38,773: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-06-14 15:12:38,773: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-06-14 15:12:38,773: INFO/queue] Stopping ServiceQueue agent_action_runner_rx
[2017-06-14 15:12:39,336: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-06-14 15:12:39,336: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-06-14 15:12:39,336: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-06-14 15:12:39,336: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-06-14 15:12:39,336: INFO/job_scheduler] Stopping...
[2017-06-14 15:12:39,336: INFO/queue] Stopping ServiceQueue job_scheduler_notifications
[2017-06-14 15:12:39,336: INFO/job_scheduler] Joining...
[2017-06-14 15:12:40,327: INFO/job_scheduler] Complete.
[2017-06-14 15:12:40,327: INFO/chroma_service] Joining job_scheduler
[2017-06-14 15:12:40,327: INFO/job_scheduler] Cancelling outstanding jobs...
[2017-06-14 15:26:21,470: INFO/queue] Purged 0 messages from 'job_scheduler_notifications' queue
[2017-06-14 15:26:21,514: INFO/queue] Purged 0 messages from 'agent_action_runner_rx' queue
[2017-06-14 15:26:21,639: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: None/None
[2017-06-14 15:26:21,639: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: None/None
[2017-06-14 15:34:58,181: INFO/chroma_service] Stopping job_scheduler
[2017-06-14 15:34:58,181: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-06-14 15:34:58,181: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-06-14 15:34:58,182: INFO/queue] Stopping ServiceQueue agent_action_runner_rx
[2017-06-14 15:34:59,181: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-06-14 15:34:59,181: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-06-14 15:34:59,181: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-06-14 15:34:59,181: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-06-14 15:34:59,181: INFO/job_scheduler] Stopping...
[2017-06-14 15:34:59,181: INFO/queue] Stopping ServiceQueue job_scheduler_notifications
[2017-06-14 15:34:59,181: INFO/job_scheduler] Joining...
[2017-06-14 15:35:00,126: INFO/job_scheduler] Complete.
[2017-06-14 15:35:00,126: INFO/chroma_service] Joining job_scheduler
[2017-06-14 15:35:00,127: INFO/job_scheduler] Cancelling outstanding jobs...
[2017-06-14 15:35:00,129: ERROR/chroma_service] Rogue thread still running, exiting hard
[2017-06-14 15:35:00,129: ERROR/chroma_service] ['Thread-2215', 'Thread-2213', 'Thread-2214', 'Thread-2212', 'Thread-2221', 'Thread-2218', 'Thread-2209', 'MainThread']
[2017-06-14 15:51:07,222: INFO/queue] Purged 0 messages from 'job_scheduler_notifications' queue
[2017-06-14 15:51:07,260: INFO/queue] Purged 1 messages from 'agent_action_runner_rx' queue
[2017-06-14 15:54:18,459: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-14 15:54:18,460: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-14 15:54:18,470: INFO/command_plan] add_jobs: created Job 1 (Test for host connectivity)
[2017-06-14 15:54:18,491: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-14 15:54:18,504: INFO/job_scheduler] Job 1: RunJobThread.run
[2017-06-14 15:54:18,610: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-14 15:54:18,611: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-14 15:54:18,729: INFO/command_plan] add_jobs: created Job 2 (Test for host connectivity)
[2017-06-14 15:54:18,803: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-14 15:54:18,833: INFO/job_scheduler] Job 2: RunJobThread.run
[2017-06-14 15:54:18,880: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-14 15:54:18,881: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-14 15:54:18,895: INFO/command_plan] add_jobs: created Job 3 (Test for host connectivity)
[2017-06-14 15:54:18,934: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-14 15:54:18,944: INFO/job_scheduler] Job 3: RunJobThread.run
[2017-06-14 15:54:18,995: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-14 15:54:18,995: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-14 15:54:19,023: INFO/command_plan] add_jobs: created Job 4 (Test for host connectivity)
[2017-06-14 15:54:19,103: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-14 15:54:19,106: INFO/job_scheduler] Job 4: RunJobThread.run
[2017-06-14 15:54:32,448: INFO/job_scheduler] Job 4 finished 1 steps successfully
[2017-06-14 15:54:32,469: INFO/job_scheduler] Job 4: complete_job: Updating cache
[2017-06-14 15:54:32,469: INFO/job_scheduler] Job 4 completing (errored=False, cancelled=False)
[2017-06-14 15:54:32,502: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-14 15:54:32,540: INFO/job_scheduler] Job 1 finished 1 steps successfully
[2017-06-14 15:54:32,602: INFO/job_scheduler] Job 1: complete_job: Updating cache
[2017-06-14 15:54:32,602: INFO/job_scheduler] Job 1 completing (errored=False, cancelled=False)
[2017-06-14 15:54:32,615: INFO/job_scheduler] Job 2 finished 1 steps successfully
[2017-06-14 15:54:32,638: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-14 15:54:32,659: INFO/job_scheduler] Job 2: complete_job: Updating cache
[2017-06-14 15:54:32,659: INFO/job_scheduler] Job 2 completing (errored=False, cancelled=False)
[2017-06-14 15:54:32,694: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-14 15:54:32,723: INFO/job_scheduler] Job 3 finished 1 steps successfully
[2017-06-14 15:54:32,753: INFO/job_scheduler] Job 3: complete_job: Updating cache
[2017-06-14 15:54:32,753: INFO/job_scheduler] Job 3 completing (errored=False, cancelled=False)
[2017-06-14 15:54:32,796: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 15:54:32,976: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-14 15:54:32,977: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-14 15:54:32,994: INFO/command_plan] add_jobs: created Job 5 (Test for host connectivity)
[2017-06-14 15:54:33,039: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-14 15:54:33,039: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-14 15:54:33,062: INFO/command_plan] add_jobs: created Job 6 (Test for host connectivity)
[2017-06-14 15:54:33,071: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-14 15:54:33,086: INFO/job_scheduler] Job 5: RunJobThread.run
[2017-06-14 15:54:33,118: INFO/job_scheduler] Job 6: RunJobThread.run
[2017-06-14 15:54:33,190: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-14 15:54:33,191: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-14 15:54:33,202: INFO/command_plan] add_jobs: created Job 7 (Test for host connectivity)
[2017-06-14 15:54:33,236: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-14 15:54:33,239: INFO/job_scheduler] Job 7: RunJobThread.run
[2017-06-14 15:54:33,379: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-14 15:54:33,381: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-14 15:54:33,407: INFO/command_plan] add_jobs: created Job 8 (Test for host connectivity)
[2017-06-14 15:54:33,420: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-14 15:54:33,439: INFO/job_scheduler] Job 8: RunJobThread.run
[2017-06-14 15:54:33,463: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-14 15:54:43,848: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-14 15:54:43,849: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-14 15:54:43,859: INFO/command_plan] add_jobs: created Job 9 (Test for host connectivity)
[2017-06-14 15:54:43,919: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-14 15:54:43,920: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-14 15:54:43,935: INFO/command_plan] add_jobs: created Job 10 (Test for host connectivity)
[2017-06-14 15:54:43,951: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 4 tasked)
[2017-06-14 15:54:43,959: INFO/job_scheduler] Job 9: RunJobThread.run
[2017-06-14 15:54:43,991: INFO/job_scheduler] Job 10: RunJobThread.run
[2017-06-14 15:54:44,047: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-14 15:54:44,047: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-14 15:54:44,075: INFO/command_plan] add_jobs: created Job 11 (Test for host connectivity)
[2017-06-14 15:54:44,105: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 6 tasked)
[2017-06-14 15:54:44,135: INFO/job_scheduler] Job 11: RunJobThread.run
[2017-06-14 15:54:44,161: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 7 tasked)
[2017-06-14 15:54:44,347: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-14 15:54:44,348: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-14 15:54:44,418: INFO/command_plan] add_jobs: created Job 12 (Test for host connectivity)
[2017-06-14 15:54:44,475: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 7 tasked)
[2017-06-14 15:54:44,510: INFO/job_scheduler] Job 12: RunJobThread.run
[2017-06-14 15:54:45,386: INFO/job_scheduler] Job 6 finished 1 steps successfully
[2017-06-14 15:54:45,428: INFO/job_scheduler] Job 6: complete_job: Updating cache
[2017-06-14 15:54:45,428: INFO/job_scheduler] Job 6 completing (errored=False, cancelled=False)
[2017-06-14 15:54:45,479: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 7 tasked)
[2017-06-14 15:54:45,577: INFO/job_scheduler] Job 5 finished 1 steps successfully
[2017-06-14 15:54:45,634: INFO/job_scheduler] Job 7 finished 1 steps successfully
[2017-06-14 15:54:45,712: INFO/job_scheduler] Job 5: complete_job: Updating cache
[2017-06-14 15:54:45,712: INFO/job_scheduler] Job 5 completing (errored=False, cancelled=False)
[2017-06-14 15:54:45,777: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 6 tasked)
[2017-06-14 15:54:45,792: INFO/job_scheduler] Job 7: complete_job: Updating cache
[2017-06-14 15:54:45,793: INFO/job_scheduler] Job 7 completing (errored=False, cancelled=False)
[2017-06-14 15:54:45,834: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 5 tasked)
[2017-06-14 15:54:45,929: INFO/job_scheduler] Job 8 finished 1 steps successfully
[2017-06-14 15:54:45,984: INFO/job_scheduler] Job 8: complete_job: Updating cache
[2017-06-14 15:54:45,984: INFO/job_scheduler] Job 8 completing (errored=False, cancelled=False)
[2017-06-14 15:54:46,093: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-14 15:54:56,433: INFO/job_scheduler] Job 10 finished 1 steps successfully
[2017-06-14 15:54:56,450: INFO/job_scheduler] Job 10: complete_job: Updating cache
[2017-06-14 15:54:56,450: INFO/job_scheduler] Job 10 completing (errored=False, cancelled=False)
[2017-06-14 15:54:56,491: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-14 15:54:56,502: INFO/job_scheduler] Job 9 finished 1 steps successfully
[2017-06-14 15:54:56,516: INFO/job_scheduler] Job 9: complete_job: Updating cache
[2017-06-14 15:54:56,516: INFO/job_scheduler] Job 9 completing (errored=False, cancelled=False)
[2017-06-14 15:54:56,550: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-14 15:54:56,642: INFO/job_scheduler] Job 11 finished 1 steps successfully
[2017-06-14 15:54:56,733: INFO/job_scheduler] Job 11: complete_job: Updating cache
[2017-06-14 15:54:56,733: INFO/job_scheduler] Job 11 completing (errored=False, cancelled=False)
[2017-06-14 15:54:56,791: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-14 15:54:57,072: INFO/job_scheduler] Job 12 finished 1 steps successfully
[2017-06-14 15:54:57,116: INFO/job_scheduler] Job 12: complete_job: Updating cache
[2017-06-14 15:54:57,116: INFO/job_scheduler] Job 12 completing (errored=False, cancelled=False)
[2017-06-14 15:54:57,149: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 15:54:57,396: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-14 15:54:57,396: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-14 15:54:57,407: INFO/command_plan] add_jobs: created Job 13 (Test for host connectivity)
[2017-06-14 15:54:57,461: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-14 15:54:57,461: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-14 15:54:57,477: INFO/command_plan] add_jobs: created Job 14 (Test for host connectivity)
[2017-06-14 15:54:57,500: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-14 15:54:57,541: INFO/job_scheduler] Job 13: RunJobThread.run
[2017-06-14 15:54:57,541: INFO/job_scheduler] Job 14: RunJobThread.run
[2017-06-14 15:54:57,577: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-14 15:54:57,578: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-14 15:54:57,598: INFO/command_plan] add_jobs: created Job 15 (Test for host connectivity)
[2017-06-14 15:54:57,607: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-14 15:54:57,613: INFO/job_scheduler] Job 15: RunJobThread.run
[2017-06-14 15:54:57,758: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-14 15:54:57,775: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-14 15:54:57,790: INFO/command_plan] add_jobs: created Job 16 (Test for host connectivity)
[2017-06-14 15:54:57,811: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-14 15:54:57,827: INFO/job_scheduler] Job 16: RunJobThread.run
[2017-06-14 15:54:57,875: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-14 15:55:01,041: INFO/job_scheduler] Getting FQDN for 'lotus-32vm5.lotus.hpdd.lab.intel.com': lotus-32vm5
lotus-32vm5.lotus.hpdd.lab.intel.com

[2017-06-14 15:55:01,072: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-14 15:55:01,076: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-1 transitioning from undeployed to unconfigured
[2017-06-14 15:55:01,106: INFO/command_plan] Created command 17 (Deploying agent to host lotus-32vm5.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-06-14 15:55:01,150: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 4 tasked)
[2017-06-14 15:55:01,189: INFO/job_scheduler] Job 17: RunJobThread.run
[2017-06-14 15:55:01,833: INFO/job_scheduler] Getting FQDN for 'lotus-32vm6.lotus.hpdd.lab.intel.com': lotus-32vm6
lotus-32vm6.lotus.hpdd.lab.intel.com

[2017-06-14 15:55:01,855: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-14 15:55:01,857: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-2 transitioning from undeployed to unconfigured
[2017-06-14 15:55:01,875: INFO/command_plan] Created command 18 (Deploying agent to host lotus-32vm6.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-06-14 15:55:01,897: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 5 tasked)
[2017-06-14 15:55:01,980: INFO/job_scheduler] Job 18: RunJobThread.run
[2017-06-14 15:55:02,245: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 6 tasked)
[2017-06-14 15:55:02,640: INFO/job_scheduler] Getting FQDN for 'lotus-32vm7.lotus.hpdd.lab.intel.com': lotus-32vm7
lotus-32vm7.lotus.hpdd.lab.intel.com

[2017-06-14 15:55:02,673: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-14 15:55:02,677: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-3 transitioning from undeployed to unconfigured
[2017-06-14 15:55:02,693: INFO/command_plan] Created command 19 (Deploying agent to host lotus-32vm7.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-06-14 15:55:02,740: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 6 tasked)
[2017-06-14 15:55:02,791: INFO/job_scheduler] Job 19: RunJobThread.run
[2017-06-14 15:55:03,061: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 7 tasked)
[2017-06-14 15:55:03,484: INFO/job_scheduler] Getting FQDN for 'lotus-32vm8.lotus.hpdd.lab.intel.com': lotus-32vm8
lotus-32vm8.lotus.hpdd.lab.intel.com

[2017-06-14 15:55:03,513: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-14 15:55:03,517: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-4 transitioning from undeployed to unconfigured
[2017-06-14 15:55:03,534: INFO/command_plan] Created command 20 (Deploying agent to host lotus-32vm8.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-06-14 15:55:03,594: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 7 tasked)
[2017-06-14 15:55:03,653: INFO/job_scheduler] Job 20: RunJobThread.run
[2017-06-14 15:55:03,666: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 8 tasked)
[2017-06-14 15:55:04,382: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 8 tasked)
[2017-06-14 15:55:09,944: INFO/job_scheduler] Job 13 finished 1 steps successfully
[2017-06-14 15:55:10,088: INFO/job_scheduler] Job 13: complete_job: Updating cache
[2017-06-14 15:55:10,088: INFO/job_scheduler] Job 13 completing (errored=False, cancelled=False)
[2017-06-14 15:55:10,121: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 7 tasked)
[2017-06-14 15:55:10,953: INFO/job_scheduler] Job 14 finished 1 steps successfully
[2017-06-14 15:55:11,163: INFO/job_scheduler] Job 14: complete_job: Updating cache
[2017-06-14 15:55:11,163: INFO/job_scheduler] Job 14 completing (errored=False, cancelled=False)
[2017-06-14 15:55:11,212: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 6 tasked)
[2017-06-14 15:55:13,014: INFO/job_scheduler] Job 16 finished 1 steps successfully
[2017-06-14 15:55:13,179: INFO/job_scheduler] Job 16: complete_job: Updating cache
[2017-06-14 15:55:13,179: INFO/job_scheduler] Job 16 completing (errored=False, cancelled=False)
[2017-06-14 15:55:13,245: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 5 tasked)
[2017-06-14 15:55:13,573: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 15:55:13,573: INFO/job_scheduler]   Job 17 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-06-14 15:55:13,574: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 5 tasked)
[2017-06-14 15:55:13,590: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/12fcf9ae-7a95-498b-a720-d2b40a64909c
[2017-06-14 15:55:13,590: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/12fcf9ae-7a95-498b-a720-d2b40a64909c
[2017-06-14 15:55:13,681: INFO/job_scheduler] Job 17 finished 1 steps successfully
[2017-06-14 15:55:13,754: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 140623202459536
[2017-06-14 15:55:13,759: INFO/job] Job 17: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 15:55:13,760: INFO/job] Job 17: not_deleted=True
[2017-06-14 15:55:13,760: INFO/job_scheduler] Job 17: complete_job: Updating cache
[2017-06-14 15:55:13,760: INFO/job_scheduler] Job 17: locked_item 140623202459536 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-14 15:55:13,762: INFO/job_scheduler] Job 17 completing (errored=False, cancelled=False)
[2017-06-14 15:55:13,947: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-1
[2017-06-14 15:55:14,004: INFO/job_scheduler] _notify: Updating .boot_time of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com) from None to 2017-05-10 06:56:09+00:00
[2017-06-14 15:55:14,017: INFO/job_scheduler] _notify: Set boot_time=2017-05-10 06:56:09+00:00 on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-1) and saved
[2017-06-14 15:55:14,026: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-14 15:55:15,594: INFO/job_scheduler] Job 15 finished 1 steps successfully
[2017-06-14 15:55:15,795: INFO/job_scheduler] Job 15: complete_job: Updating cache
[2017-06-14 15:55:15,795: INFO/job_scheduler] Job 15 completing (errored=False, cancelled=False)
[2017-06-14 15:55:15,878: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-14 15:55:16,741: INFO/job_scheduler] _notify: Updating .properties of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-06-14 15:55:16,773: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-1) and saved
[2017-06-14 15:55:16,778: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-14 15:55:16,953: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-14 15:55:23,779: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 15:55:23,780: INFO/job_scheduler]   Job 18 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-06-14 15:55:23,780: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-14 15:55:23,790: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/5b929a0e-6e23-409c-8a01-7795c4b9b596
[2017-06-14 15:55:23,790: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/5b929a0e-6e23-409c-8a01-7795c4b9b596
[2017-06-14 15:55:23,914: INFO/job_scheduler] Job 18 finished 1 steps successfully
[2017-06-14 15:55:23,968: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 140623806379856
[2017-06-14 15:55:23,973: INFO/job] Job 18: StateChangeJob complete, setting state unconfigured on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-14 15:55:23,973: INFO/job] Job 18: not_deleted=True
[2017-06-14 15:55:23,973: INFO/job_scheduler] Job 18: complete_job: Updating cache
[2017-06-14 15:55:23,973: INFO/job_scheduler] Job 18: locked_item 140623806379856 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-14 15:55:23,976: INFO/job_scheduler] Job 18 completing (errored=False, cancelled=False)
[2017-06-14 15:55:24,037: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-2
[2017-06-14 15:55:24,037: INFO/job_scheduler] _notify: Updating .boot_time of item 2 (lotus-32vm6.lotus.hpdd.lab.intel.com) from None to 2017-04-25 07:09:19+00:00
[2017-06-14 15:55:24,042: INFO/job_scheduler] _notify: Set boot_time=2017-04-25 07:09:19+00:00 on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-2) and saved
[2017-06-14 15:55:24,054: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-14 15:55:27,503: INFO/job_scheduler] _notify: Updating .properties of item 2 (lotus-32vm6.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-06-14 15:55:27,509: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-2) and saved
[2017-06-14 15:55:27,514: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-14 15:55:27,598: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-14 15:55:28,666: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 15:55:28,666: INFO/job_scheduler]   Job 20 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-06-14 15:55:28,666: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-14 15:55:28,714: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/791bfe7f-8cf1-4696-bf74-8d18e8056fb5
[2017-06-14 15:55:28,714: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/791bfe7f-8cf1-4696-bf74-8d18e8056fb5
[2017-06-14 15:55:28,748: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/09fc0ec3-666e-491a-aa31-9cdd376dd28c
[2017-06-14 15:55:28,748: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/09fc0ec3-666e-491a-aa31-9cdd376dd28c
[2017-06-14 15:55:28,750: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 15:55:28,750: INFO/job_scheduler]   Job 19 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-06-14 15:55:28,750: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-14 15:55:28,768: INFO/job_scheduler] Job 20 finished 1 steps successfully
[2017-06-14 15:55:28,842: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 140624277360400
[2017-06-14 15:55:28,847: INFO/job] Job 20: StateChangeJob complete, setting state unconfigured on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 15:55:28,847: INFO/job] Job 20: not_deleted=True
[2017-06-14 15:55:28,848: INFO/job_scheduler] Job 20: complete_job: Updating cache
[2017-06-14 15:55:28,848: INFO/job_scheduler] Job 20: locked_item 140624277360400 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-14 15:55:28,853: INFO/job_scheduler] Job 20 completing (errored=False, cancelled=False)
[2017-06-14 15:55:28,873: INFO/job_scheduler] Job 19 finished 1 steps successfully
[2017-06-14 15:55:28,887: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-4
[2017-06-14 15:55:28,887: INFO/job_scheduler] _notify: Updating .boot_time of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com) from None to 2017-04-25 07:09:56+00:00
[2017-06-14 15:55:28,892: INFO/job_scheduler] _notify: Set boot_time=2017-04-25 07:09:56+00:00 on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-4) and saved
[2017-06-14 15:55:28,899: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-14 15:55:28,918: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 140623806376720
[2017-06-14 15:55:28,923: INFO/job] Job 19: StateChangeJob complete, setting state unconfigured on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-14 15:55:28,924: INFO/job] Job 19: not_deleted=True
[2017-06-14 15:55:28,924: INFO/job_scheduler] Job 19: complete_job: Updating cache
[2017-06-14 15:55:28,924: INFO/job_scheduler] Job 19: locked_item 140623806376720 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-14 15:55:28,931: INFO/job_scheduler] Job 19 completing (errored=False, cancelled=False)
[2017-06-14 15:55:29,052: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-3
[2017-06-14 15:55:29,052: INFO/job_scheduler] _notify: Updating .boot_time of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com) from None to 2017-04-25 07:09:31+00:00
[2017-06-14 15:55:29,065: INFO/job_scheduler] _notify: Set boot_time=2017-04-25 07:09:31+00:00 on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-3) and saved
[2017-06-14 15:55:29,070: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 15:55:31,911: INFO/job_scheduler] _notify: Updating .properties of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-06-14 15:55:31,917: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-4) and saved
[2017-06-14 15:55:31,922: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 15:55:31,975: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 15:55:33,080: INFO/job_scheduler] _notify: Updating .properties of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-06-14 15:55:33,085: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-3) and saved
[2017-06-14 15:55:33,090: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 15:55:33,140: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 15:55:42,036: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-14 15:55:42,045: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-1 transitioning from unconfigured to managed
[2017-06-14 15:55:42,220: INFO/command_plan] Created command 21 (Setup managed host lotus-32vm5.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-06-14 15:55:42,348: INFO/job_scheduler] run_next: 1 runnable jobs of (11 pending, 0 tasked)
[2017-06-14 15:55:42,376: INFO/job_scheduler] Job 21: RunJobThread.run
[2017-06-14 15:55:42,377: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-06-14 15:55:42,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 118eaf75-a270-429a-a6d7-6e710626f94b
[2017-06-14 15:55:42,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 118eaf75-a270-429a-a6d7-6e710626f94b
[2017-06-14 15:55:42,463: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-14 15:55:42,477: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-2 transitioning from unconfigured to managed
[2017-06-14 15:55:42,612: INFO/command_plan] Created command 22 (Setup managed host lotus-32vm6.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-06-14 15:55:42,790: INFO/job_scheduler] run_next: 1 runnable jobs of (21 pending, 1 tasked)
[2017-06-14 15:55:42,833: INFO/job_scheduler] Job 32: RunJobThread.run
[2017-06-14 15:55:42,834: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-06-14 15:55:42,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 453f6581-6429-455a-aa27-e4e55568a896
[2017-06-14 15:55:42,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 453f6581-6429-455a-aa27-e4e55568a896
[2017-06-14 15:55:42,977: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-14 15:55:42,982: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-3 transitioning from unconfigured to managed
[2017-06-14 15:55:43,107: INFO/command_plan] Created command 23 (Setup managed host lotus-32vm7.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-06-14 15:55:43,178: INFO/job_scheduler] run_next: 1 runnable jobs of (31 pending, 2 tasked)
[2017-06-14 15:55:43,225: INFO/job_scheduler] Job 43: RunJobThread.run
[2017-06-14 15:55:43,225: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-06-14 15:55:43,260: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c838f11c-fe2c-4dd9-a8e6-a0976999294a
[2017-06-14 15:55:43,260: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c838f11c-fe2c-4dd9-a8e6-a0976999294a
[2017-06-14 15:55:43,473: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-14 15:55:43,478: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-4 transitioning from unconfigured to managed
[2017-06-14 15:55:43,637: INFO/command_plan] Created command 24 (Setup managed host lotus-32vm8.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-06-14 15:55:43,730: INFO/job_scheduler] run_next: 1 runnable jobs of (41 pending, 3 tasked)
[2017-06-14 15:55:43,779: INFO/job_scheduler] Job 54: RunJobThread.run
[2017-06-14 15:55:43,780: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-06-14 15:55:43,808: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fef01b7d-5c13-4979-ab63-93d42aa8bd80
[2017-06-14 15:55:43,808: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fef01b7d-5c13-4979-ab63-93d42aa8bd80
[2017-06-14 15:58:05,151: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/09fc0ec3-666e-491a-aa31-9cdd376dd28c
[2017-06-14 15:58:05,151: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/09fc0ec3-666e-491a-aa31-9cdd376dd28c
[2017-06-14 15:58:05,151: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/09fc0ec3-666e-491a-aa31-9cdd376dd28c
[2017-06-14 15:58:05,151: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/09fc0ec3-666e-491a-aa31-9cdd376dd28c
[2017-06-14 15:58:05,151: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c838f11c-fe2c-4dd9-a8e6-a0976999294a
[2017-06-14 15:58:05,151: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c838f11c-fe2c-4dd9-a8e6-a0976999294a
[2017-06-14 15:58:05,177: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c838f11c-fe2c-4dd9-a8e6-a0976999294a
[2017-06-14 15:58:05,177: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c838f11c-fe2c-4dd9-a8e6-a0976999294a
[2017-06-14 15:58:05,177: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-06-14 15:58:05.177735+00:00:
[2017-06-14 15:58:05,177: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-06-14 15:58:05.177735+00:00:
[2017-06-14 15:58:05,178: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-06-14 15:58:05,178: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-06-14 15:58:05,199: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 15:58:05,199: INFO/job_scheduler]   Job 43 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-14 15:58:05,199: INFO/job_scheduler]   Job 44 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:05,199: INFO/job_scheduler]   Job 45 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:05,199: INFO/job_scheduler]   Job 46 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:05,199: INFO/job_scheduler]   Job 47 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:05,199: INFO/job_scheduler]   Job 50 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:05,200: INFO/job_scheduler]   Job 53 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 15:58:05,200: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-14 15:58:05,201: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-06-14 15:58:05.201676+00:00:
[2017-06-14 15:58:05,201: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-06-14 15:58:05.201676+00:00:
[2017-06-14 15:58:05,201: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-06-14 15:58:05,201: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-06-14 15:58:05,227: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 15:58:05,227: INFO/job_scheduler]   Job 43 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-14 15:58:05,227: INFO/job_scheduler]   Job 44 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:05,227: INFO/job_scheduler]   Job 45 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:05,227: INFO/job_scheduler]   Job 46 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:05,227: INFO/job_scheduler]   Job 47 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:05,227: INFO/job_scheduler]   Job 50 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:05,227: INFO/job_scheduler]   Job 53 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 15:58:05,228: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-14 15:58:05,229: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com restart_agent {}
[2017-06-14 15:58:05,260: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9bfa61f6-bb88-401a-ba01-b449230dc085
[2017-06-14 15:58:05,260: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9bfa61f6-bb88-401a-ba01-b449230dc085
[2017-06-14 15:58:05,326: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/09fc0ec3-666e-491a-aa31-9cdd376dd28c
[2017-06-14 15:58:05,326: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/09fc0ec3-666e-491a-aa31-9cdd376dd28c
[2017-06-14 15:58:05,346: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/09fc0ec3-666e-491a-aa31-9cdd376dd28c
[2017-06-14 15:58:05,346: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/09fc0ec3-666e-491a-aa31-9cdd376dd28c
[2017-06-14 15:58:05,346: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9bfa61f6-bb88-401a-ba01-b449230dc085
[2017-06-14 15:58:05,346: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9bfa61f6-bb88-401a-ba01-b449230dc085
[2017-06-14 15:58:05,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9bfa61f6-bb88-401a-ba01-b449230dc085
[2017-06-14 15:58:05,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9bfa61f6-bb88-401a-ba01-b449230dc085
[2017-06-14 15:58:05,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm7.lotus.hpdd.lab.intel.com (old 09fc0ec3-666e-491a-aa31-9cdd376dd28c)
[2017-06-14 15:58:05,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm7.lotus.hpdd.lab.intel.com (old 09fc0ec3-666e-491a-aa31-9cdd376dd28c)
[2017-06-14 15:58:13,668: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/12fcf9ae-7a95-498b-a720-d2b40a64909c
[2017-06-14 15:58:13,668: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/12fcf9ae-7a95-498b-a720-d2b40a64909c
[2017-06-14 15:58:13,669: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/12fcf9ae-7a95-498b-a720-d2b40a64909c
[2017-06-14 15:58:13,669: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/12fcf9ae-7a95-498b-a720-d2b40a64909c
[2017-06-14 15:58:13,669: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 118eaf75-a270-429a-a6d7-6e710626f94b
[2017-06-14 15:58:13,669: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 118eaf75-a270-429a-a6d7-6e710626f94b
[2017-06-14 15:58:13,700: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 118eaf75-a270-429a-a6d7-6e710626f94b
[2017-06-14 15:58:13,700: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 118eaf75-a270-429a-a6d7-6e710626f94b
[2017-06-14 15:58:13,701: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-06-14 15:58:13.701077+00:00:
[2017-06-14 15:58:13,701: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-06-14 15:58:13.701077+00:00:
[2017-06-14 15:58:13,701: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-06-14 15:58:13,701: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-06-14 15:58:13,714: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 15:58:13,714: INFO/job_scheduler]   Job 21 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-14 15:58:13,714: INFO/job_scheduler]   Job 22 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:13,714: INFO/job_scheduler]   Job 23 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:13,714: INFO/job_scheduler]   Job 24 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:13,714: INFO/job_scheduler]   Job 25 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:13,714: INFO/job_scheduler]   Job 28 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:13,715: INFO/job_scheduler]   Job 31 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 15:58:13,715: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-14 15:58:13,716: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-06-14 15:58:13.716831+00:00:
[2017-06-14 15:58:13,716: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-06-14 15:58:13.716831+00:00:
[2017-06-14 15:58:13,717: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-06-14 15:58:13,717: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-06-14 15:58:13,729: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 15:58:13,730: INFO/job_scheduler]   Job 21 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-14 15:58:13,730: INFO/job_scheduler]   Job 22 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:13,730: INFO/job_scheduler]   Job 23 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:13,730: INFO/job_scheduler]   Job 24 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:13,730: INFO/job_scheduler]   Job 25 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:13,730: INFO/job_scheduler]   Job 28 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:13,730: INFO/job_scheduler]   Job 31 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 15:58:13,731: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-14 15:58:13,732: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com restart_agent {}
[2017-06-14 15:58:13,748: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3f8743ff-51a4-41b6-83cb-e4214fb8c849
[2017-06-14 15:58:13,748: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3f8743ff-51a4-41b6-83cb-e4214fb8c849
[2017-06-14 15:58:13,849: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/12fcf9ae-7a95-498b-a720-d2b40a64909c
[2017-06-14 15:58:13,849: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/12fcf9ae-7a95-498b-a720-d2b40a64909c
[2017-06-14 15:58:13,850: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/12fcf9ae-7a95-498b-a720-d2b40a64909c
[2017-06-14 15:58:13,850: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/12fcf9ae-7a95-498b-a720-d2b40a64909c
[2017-06-14 15:58:13,850: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3f8743ff-51a4-41b6-83cb-e4214fb8c849
[2017-06-14 15:58:13,850: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3f8743ff-51a4-41b6-83cb-e4214fb8c849
[2017-06-14 15:58:13,866: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3f8743ff-51a4-41b6-83cb-e4214fb8c849
[2017-06-14 15:58:13,866: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3f8743ff-51a4-41b6-83cb-e4214fb8c849
[2017-06-14 15:58:13,866: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old 12fcf9ae-7a95-498b-a720-d2b40a64909c)
[2017-06-14 15:58:13,866: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old 12fcf9ae-7a95-498b-a720-d2b40a64909c)
[2017-06-14 15:58:15,122: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/5b929a0e-6e23-409c-8a01-7795c4b9b596
[2017-06-14 15:58:15,122: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/5b929a0e-6e23-409c-8a01-7795c4b9b596
[2017-06-14 15:58:15,122: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/5b929a0e-6e23-409c-8a01-7795c4b9b596
[2017-06-14 15:58:15,122: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/5b929a0e-6e23-409c-8a01-7795c4b9b596
[2017-06-14 15:58:15,122: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 453f6581-6429-455a-aa27-e4e55568a896
[2017-06-14 15:58:15,122: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 453f6581-6429-455a-aa27-e4e55568a896
[2017-06-14 15:58:15,149: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 453f6581-6429-455a-aa27-e4e55568a896
[2017-06-14 15:58:15,149: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 453f6581-6429-455a-aa27-e4e55568a896
[2017-06-14 15:58:15,149: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-06-14 15:58:15.149606+00:00:
[2017-06-14 15:58:15,149: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-06-14 15:58:15.149606+00:00:
[2017-06-14 15:58:15,149: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-06-14 15:58:15,149: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-06-14 15:58:15,166: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 15:58:15,167: INFO/job_scheduler]   Job 32 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-14 15:58:15,167: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-06-14 15:58:15.167724+00:00:
[2017-06-14 15:58:15,167: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-06-14 15:58:15.167724+00:00:
[2017-06-14 15:58:15,168: INFO/job_scheduler]   Job 33 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:15,167: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-06-14 15:58:15,168: INFO/job_scheduler]   Job 34 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:15,168: INFO/job_scheduler]   Job 35 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:15,167: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-06-14 15:58:15,176: INFO/job_scheduler]   Job 36 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:15,176: INFO/job_scheduler]   Job 39 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:15,177: INFO/job_scheduler]   Job 42 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 15:58:15,177: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-14 15:58:15,183: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 15:58:15,184: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com restart_agent {}
[2017-06-14 15:58:15,184: INFO/job_scheduler]   Job 32 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-14 15:58:15,193: INFO/job_scheduler]   Job 33 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:15,193: INFO/job_scheduler]   Job 34 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:15,193: INFO/job_scheduler]   Job 35 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:15,193: INFO/job_scheduler]   Job 36 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:15,193: INFO/job_scheduler]   Job 39 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:15,193: INFO/job_scheduler]   Job 42 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 15:58:15,194: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-14 15:58:15,199: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 084dcd08-7f32-4dd8-93c7-307adedb1fc8
[2017-06-14 15:58:15,199: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 084dcd08-7f32-4dd8-93c7-307adedb1fc8
[2017-06-14 15:58:15,294: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/5b929a0e-6e23-409c-8a01-7795c4b9b596
[2017-06-14 15:58:15,294: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/5b929a0e-6e23-409c-8a01-7795c4b9b596
[2017-06-14 15:58:15,295: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/5b929a0e-6e23-409c-8a01-7795c4b9b596
[2017-06-14 15:58:15,295: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/5b929a0e-6e23-409c-8a01-7795c4b9b596
[2017-06-14 15:58:15,295: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 084dcd08-7f32-4dd8-93c7-307adedb1fc8
[2017-06-14 15:58:15,295: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 084dcd08-7f32-4dd8-93c7-307adedb1fc8
[2017-06-14 15:58:15,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 084dcd08-7f32-4dd8-93c7-307adedb1fc8
[2017-06-14 15:58:15,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 084dcd08-7f32-4dd8-93c7-307adedb1fc8
[2017-06-14 15:58:15,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm6.lotus.hpdd.lab.intel.com (old 5b929a0e-6e23-409c-8a01-7795c4b9b596)
[2017-06-14 15:58:15,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm6.lotus.hpdd.lab.intel.com (old 5b929a0e-6e23-409c-8a01-7795c4b9b596)
[2017-06-14 15:58:24,061: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/791bfe7f-8cf1-4696-bf74-8d18e8056fb5
[2017-06-14 15:58:24,061: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/791bfe7f-8cf1-4696-bf74-8d18e8056fb5
[2017-06-14 15:58:24,062: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/791bfe7f-8cf1-4696-bf74-8d18e8056fb5
[2017-06-14 15:58:24,062: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/791bfe7f-8cf1-4696-bf74-8d18e8056fb5
[2017-06-14 15:58:24,062: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fef01b7d-5c13-4979-ab63-93d42aa8bd80
[2017-06-14 15:58:24,062: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fef01b7d-5c13-4979-ab63-93d42aa8bd80
[2017-06-14 15:58:24,093: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fef01b7d-5c13-4979-ab63-93d42aa8bd80
[2017-06-14 15:58:24,093: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fef01b7d-5c13-4979-ab63-93d42aa8bd80
[2017-06-14 15:58:24,093: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-06-14 15:58:24.093826+00:00:
[2017-06-14 15:58:24,093: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-06-14 15:58:24.093826+00:00:
[2017-06-14 15:58:24,094: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-06-14 15:58:24,094: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-06-14 15:58:24,110: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 15:58:24,110: INFO/job_scheduler]   Job 54 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-14 15:58:24,111: INFO/job_scheduler]   Job 55 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:24,111: INFO/job_scheduler]   Job 56 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:24,111: INFO/job_scheduler]   Job 57 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:24,111: INFO/job_scheduler]   Job 58 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:24,111: INFO/job_scheduler]   Job 62 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:24,111: INFO/job_scheduler]   Job 64 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 15:58:24,111: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-14 15:58:24,113: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-06-14 15:58:24.113274+00:00:
[2017-06-14 15:58:24,113: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-06-14 15:58:24.113274+00:00:
[2017-06-14 15:58:24,113: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-06-14 15:58:24,113: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-06-14 15:58:24,136: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 15:58:24,136: INFO/job_scheduler]   Job 54 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-14 15:58:24,136: INFO/job_scheduler]   Job 55 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:24,136: INFO/job_scheduler]   Job 56 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:24,136: INFO/job_scheduler]   Job 57 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:24,136: INFO/job_scheduler]   Job 58 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:24,136: INFO/job_scheduler]   Job 62 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:24,136: INFO/job_scheduler]   Job 64 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 15:58:24,137: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-14 15:58:24,140: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com restart_agent {}
[2017-06-14 15:58:24,153: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 75fa5726-76e6-4c4c-be2d-902e505a1d6f
[2017-06-14 15:58:24,153: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 75fa5726-76e6-4c4c-be2d-902e505a1d6f
[2017-06-14 15:58:24,254: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/791bfe7f-8cf1-4696-bf74-8d18e8056fb5
[2017-06-14 15:58:24,254: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/791bfe7f-8cf1-4696-bf74-8d18e8056fb5
[2017-06-14 15:58:24,254: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/791bfe7f-8cf1-4696-bf74-8d18e8056fb5
[2017-06-14 15:58:24,254: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/791bfe7f-8cf1-4696-bf74-8d18e8056fb5
[2017-06-14 15:58:24,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 75fa5726-76e6-4c4c-be2d-902e505a1d6f
[2017-06-14 15:58:24,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 75fa5726-76e6-4c4c-be2d-902e505a1d6f
[2017-06-14 15:58:24,272: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 75fa5726-76e6-4c4c-be2d-902e505a1d6f
[2017-06-14 15:58:24,272: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 75fa5726-76e6-4c4c-be2d-902e505a1d6f
[2017-06-14 15:58:24,272: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm8.lotus.hpdd.lab.intel.com (old 791bfe7f-8cf1-4696-bf74-8d18e8056fb5)
[2017-06-14 15:58:24,272: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm8.lotus.hpdd.lab.intel.com (old 791bfe7f-8cf1-4696-bf74-8d18e8056fb5)
[2017-06-14 15:58:34,023: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/09fc0ec3-666e-491a-aa31-9cdd376dd28c
[2017-06-14 15:58:34,023: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/09fc0ec3-666e-491a-aa31-9cdd376dd28c
[2017-06-14 15:58:34,023: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 09fc0ec3-666e-491a-aa31-9cdd376dd28c because session terminated
[2017-06-14 15:58:34,023: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 09fc0ec3-666e-491a-aa31-9cdd376dd28c because session terminated
[2017-06-14 15:58:34,025: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:58:34,025: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:58:34,423: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm7.lotus.hpdd.lab.intel.com new 932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:58:34,423: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm7.lotus.hpdd.lab.intel.com new 932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:58:34,502: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-14 15:58:34,534: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cd3eb2a3-222c-44e0-8ebb-4724c97e936a
[2017-06-14 15:58:34,534: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cd3eb2a3-222c-44e0-8ebb-4724c97e936a
[2017-06-14 15:58:35,602: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:58:35,602: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:58:35,603: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:58:35,603: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:58:35,603: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cd3eb2a3-222c-44e0-8ebb-4724c97e936a
[2017-06-14 15:58:35,603: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cd3eb2a3-222c-44e0-8ebb-4724c97e936a
[2017-06-14 15:58:35,651: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cd3eb2a3-222c-44e0-8ebb-4724c97e936a
[2017-06-14 15:58:35,651: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cd3eb2a3-222c-44e0-8ebb-4724c97e936a
[2017-06-14 15:58:35,652: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-06-14 15:58:35,679: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cff2de0e-3d97-4d40-9b49-c865d898a4f7
[2017-06-14 15:58:35,679: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cff2de0e-3d97-4d40-9b49-c865d898a4f7
[2017-06-14 15:58:38,266: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 15:58:38,266: INFO/job_scheduler]   Job 43 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-14 15:58:38,266: INFO/job_scheduler]   Job 44 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:38,266: INFO/job_scheduler]   Job 45 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:38,266: INFO/job_scheduler]   Job 46 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:38,266: INFO/job_scheduler]   Job 47 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:38,266: INFO/job_scheduler]   Job 50 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:38,266: INFO/job_scheduler]   Job 53 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 15:58:38,267: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-14 15:58:42,619: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/12fcf9ae-7a95-498b-a720-d2b40a64909c
[2017-06-14 15:58:42,619: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/12fcf9ae-7a95-498b-a720-d2b40a64909c
[2017-06-14 15:58:42,619: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 12fcf9ae-7a95-498b-a720-d2b40a64909c because session terminated
[2017-06-14 15:58:42,619: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 12fcf9ae-7a95-498b-a720-d2b40a64909c because session terminated
[2017-06-14 15:58:42,620: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/28bd4119-43c0-46cc-95d1-202fa8408bad
[2017-06-14 15:58:42,620: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/28bd4119-43c0-46cc-95d1-202fa8408bad
[2017-06-14 15:58:42,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new 28bd4119-43c0-46cc-95d1-202fa8408bad
[2017-06-14 15:58:42,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new 28bd4119-43c0-46cc-95d1-202fa8408bad
[2017-06-14 15:58:42,899: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-14 15:58:42,935: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fff85b73-abf0-40a6-bd6a-0f220c07ce19
[2017-06-14 15:58:42,935: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fff85b73-abf0-40a6-bd6a-0f220c07ce19
[2017-06-14 15:58:43,985: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/5b929a0e-6e23-409c-8a01-7795c4b9b596
[2017-06-14 15:58:43,985: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/5b929a0e-6e23-409c-8a01-7795c4b9b596
[2017-06-14 15:58:43,985: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 5b929a0e-6e23-409c-8a01-7795c4b9b596 because session terminated
[2017-06-14 15:58:43,985: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 5b929a0e-6e23-409c-8a01-7795c4b9b596 because session terminated
[2017-06-14 15:58:43,988: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 15:58:43,988: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 15:58:44,187: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/28bd4119-43c0-46cc-95d1-202fa8408bad
[2017-06-14 15:58:44,187: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/28bd4119-43c0-46cc-95d1-202fa8408bad
[2017-06-14 15:58:44,187: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/28bd4119-43c0-46cc-95d1-202fa8408bad
[2017-06-14 15:58:44,187: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/28bd4119-43c0-46cc-95d1-202fa8408bad
[2017-06-14 15:58:44,187: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fff85b73-abf0-40a6-bd6a-0f220c07ce19
[2017-06-14 15:58:44,187: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fff85b73-abf0-40a6-bd6a-0f220c07ce19
[2017-06-14 15:58:44,217: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fff85b73-abf0-40a6-bd6a-0f220c07ce19
[2017-06-14 15:58:44,217: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fff85b73-abf0-40a6-bd6a-0f220c07ce19
[2017-06-14 15:58:44,217: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-06-14 15:58:44,230: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ed2ad566-7d83-41ee-8199-6a9c7274088e
[2017-06-14 15:58:44,230: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ed2ad566-7d83-41ee-8199-6a9c7274088e
[2017-06-14 15:58:44,356: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm6.lotus.hpdd.lab.intel.com new 074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 15:58:44,356: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm6.lotus.hpdd.lab.intel.com new 074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 15:58:44,356: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-14 15:58:44,374: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3a9f0307-c793-4fc2-bd8a-91fa8c00d5c8
[2017-06-14 15:58:44,374: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3a9f0307-c793-4fc2-bd8a-91fa8c00d5c8
[2017-06-14 15:58:45,510: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 15:58:45,510: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 15:58:45,510: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 15:58:45,510: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 15:58:45,510: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3a9f0307-c793-4fc2-bd8a-91fa8c00d5c8
[2017-06-14 15:58:45,510: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3a9f0307-c793-4fc2-bd8a-91fa8c00d5c8
[2017-06-14 15:58:45,548: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3a9f0307-c793-4fc2-bd8a-91fa8c00d5c8
[2017-06-14 15:58:45,548: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3a9f0307-c793-4fc2-bd8a-91fa8c00d5c8
[2017-06-14 15:58:45,548: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-06-14 15:58:45,569: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0e8ed477-8fc5-4fc1-981c-123ca9c182eb
[2017-06-14 15:58:45,569: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0e8ed477-8fc5-4fc1-981c-123ca9c182eb
[2017-06-14 15:58:45,943: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 15:58:45,943: INFO/job_scheduler]   Job 21 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-14 15:58:45,943: INFO/job_scheduler]   Job 22 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:45,943: INFO/job_scheduler]   Job 23 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:45,943: INFO/job_scheduler]   Job 24 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:45,943: INFO/job_scheduler]   Job 25 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:45,943: INFO/job_scheduler]   Job 28 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:45,943: INFO/job_scheduler]   Job 31 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 15:58:45,944: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-14 15:58:47,271: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 15:58:47,271: INFO/job_scheduler]   Job 32 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-14 15:58:47,272: INFO/job_scheduler]   Job 33 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:47,272: INFO/job_scheduler]   Job 34 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:47,272: INFO/job_scheduler]   Job 35 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:47,272: INFO/job_scheduler]   Job 36 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:47,272: INFO/job_scheduler]   Job 39 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:47,272: INFO/job_scheduler]   Job 42 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 15:58:47,272: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-14 15:58:54,312: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm8.lotus.hpdd.lab.intel.com timeout after 30.0s
[2017-06-14 15:58:54,312: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm8.lotus.hpdd.lab.intel.com timeout after 30.0s
[2017-06-14 15:58:55,048: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/791bfe7f-8cf1-4696-bf74-8d18e8056fb5
[2017-06-14 15:58:55,048: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/791bfe7f-8cf1-4696-bf74-8d18e8056fb5
[2017-06-14 15:58:55,048: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 791bfe7f-8cf1-4696-bf74-8d18e8056fb5 because session terminated
[2017-06-14 15:58:55,048: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 791bfe7f-8cf1-4696-bf74-8d18e8056fb5 because session terminated
[2017-06-14 15:58:55,051: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a
[2017-06-14 15:58:55,051: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a
[2017-06-14 15:58:55,313: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm8.lotus.hpdd.lab.intel.com new e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a
[2017-06-14 15:58:55,313: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm8.lotus.hpdd.lab.intel.com new e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a
[2017-06-14 15:58:55,315: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-14 15:58:55,330: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2493b6b8-2433-4018-a4f2-f1ec5e2b3c9a
[2017-06-14 15:58:55,330: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2493b6b8-2433-4018-a4f2-f1ec5e2b3c9a
[2017-06-14 15:58:56,427: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a
[2017-06-14 15:58:56,427: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a
[2017-06-14 15:58:56,427: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a
[2017-06-14 15:58:56,427: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a
[2017-06-14 15:58:56,427: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2493b6b8-2433-4018-a4f2-f1ec5e2b3c9a
[2017-06-14 15:58:56,427: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2493b6b8-2433-4018-a4f2-f1ec5e2b3c9a
[2017-06-14 15:58:56,452: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2493b6b8-2433-4018-a4f2-f1ec5e2b3c9a
[2017-06-14 15:58:56,452: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2493b6b8-2433-4018-a4f2-f1ec5e2b3c9a
[2017-06-14 15:58:56,453: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-06-14 15:58:56,508: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 233fdcc1-e10f-46db-a26e-9476895f4cc1
[2017-06-14 15:58:56,508: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 233fdcc1-e10f-46db-a26e-9476895f4cc1
[2017-06-14 15:58:58,373: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 15:58:58,373: INFO/job_scheduler]   Job 54 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-14 15:58:58,373: INFO/job_scheduler]   Job 55 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:58,374: INFO/job_scheduler]   Job 56 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:58,374: INFO/job_scheduler]   Job 57 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:58,374: INFO/job_scheduler]   Job 58 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:58,374: INFO/job_scheduler]   Job 62 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 15:58:58,374: INFO/job_scheduler]   Job 64 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 15:58:58,374: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-14 15:59:17,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:17,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:17,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:17,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:17,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cff2de0e-3d97-4d40-9b49-c865d898a4f7
[2017-06-14 15:59:17,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cff2de0e-3d97-4d40-9b49-c865d898a4f7
[2017-06-14 15:59:17,292: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cff2de0e-3d97-4d40-9b49-c865d898a4f7
[2017-06-14 15:59:17,292: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cff2de0e-3d97-4d40-9b49-c865d898a4f7
[2017-06-14 15:59:17,703: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com install_packages {'repos': [u'iml-agent'], 'packages': [u'zfs', u'zfs-dkms', u'kernel-devel-[0-9]*_lustre', u'kernel-[0-9]*_lustre', u'lustre-osd-zfs-mount', u'kmod-lustre-osd-ldiskfs', u'lustre-dkms', u'lustre', u'chroma-agent-management']}
[2017-06-14 15:59:17,720: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3bb9e585-75cb-40a6-b220-b1d716b32567
[2017-06-14 15:59:17,720: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3bb9e585-75cb-40a6-b220-b1d716b32567
[2017-06-14 15:59:26,814: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/28bd4119-43c0-46cc-95d1-202fa8408bad
[2017-06-14 15:59:26,814: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/28bd4119-43c0-46cc-95d1-202fa8408bad
[2017-06-14 15:59:26,814: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/28bd4119-43c0-46cc-95d1-202fa8408bad
[2017-06-14 15:59:26,814: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/28bd4119-43c0-46cc-95d1-202fa8408bad
[2017-06-14 15:59:26,814: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ed2ad566-7d83-41ee-8199-6a9c7274088e
[2017-06-14 15:59:26,814: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ed2ad566-7d83-41ee-8199-6a9c7274088e
[2017-06-14 15:59:26,851: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ed2ad566-7d83-41ee-8199-6a9c7274088e
[2017-06-14 15:59:26,851: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ed2ad566-7d83-41ee-8199-6a9c7274088e
[2017-06-14 15:59:27,060: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-14 15:59:27,062: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-14 15:59:27,101: INFO/command_plan] add_jobs: created Job 65 (Trigger plugin poll for linux plugins)
[2017-06-14 15:59:27,651: INFO/job_scheduler] run_next: 1 runnable jobs of (41 pending, 4 tasked)
[2017-06-14 15:59:27,699: INFO/job_scheduler] Job 65: RunJobThread.run
[2017-06-14 15:59:27,734: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-14 15:59:27,765: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-14 15:59:27,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 578b7b37-89b5-441c-9e8b-34c042a75f76
[2017-06-14 15:59:27,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 578b7b37-89b5-441c-9e8b-34c042a75f76
[2017-06-14 15:59:27,794: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-14 15:59:27,820: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6e48d484-9ff6-46b4-b68e-c8e8016ca281
[2017-06-14 15:59:27,820: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6e48d484-9ff6-46b4-b68e-c8e8016ca281
[2017-06-14 15:59:27,862: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8e4bb2b9-98eb-47f8-ad0e-215e410000ef
[2017-06-14 15:59:27,862: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8e4bb2b9-98eb-47f8-ad0e-215e410000ef
[2017-06-14 15:59:27,884: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 15:59:27,884: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 15:59:27,885: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 15:59:27,885: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 15:59:27,885: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 578b7b37-89b5-441c-9e8b-34c042a75f76
[2017-06-14 15:59:27,885: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 578b7b37-89b5-441c-9e8b-34c042a75f76
[2017-06-14 15:59:27,887: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:27,887: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:27,887: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:27,887: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:27,888: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6e48d484-9ff6-46b4-b68e-c8e8016ca281
[2017-06-14 15:59:27,888: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6e48d484-9ff6-46b4-b68e-c8e8016ca281
[2017-06-14 15:59:27,899: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6e48d484-9ff6-46b4-b68e-c8e8016ca281
[2017-06-14 15:59:27,899: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6e48d484-9ff6-46b4-b68e-c8e8016ca281
[2017-06-14 15:59:27,899: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-14 15:59:27,913: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 578b7b37-89b5-441c-9e8b-34c042a75f76
[2017-06-14 15:59:27,913: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 578b7b37-89b5-441c-9e8b-34c042a75f76
[2017-06-14 15:59:27,913: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-14 15:59:27,961: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a
[2017-06-14 15:59:27,961: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a
[2017-06-14 15:59:27,961: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a
[2017-06-14 15:59:27,961: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a
[2017-06-14 15:59:27,961: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8e4bb2b9-98eb-47f8-ad0e-215e410000ef
[2017-06-14 15:59:27,961: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8e4bb2b9-98eb-47f8-ad0e-215e410000ef
[2017-06-14 15:59:27,983: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8e4bb2b9-98eb-47f8-ad0e-215e410000ef
[2017-06-14 15:59:27,983: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8e4bb2b9-98eb-47f8-ad0e-215e410000ef
[2017-06-14 15:59:27,983: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-14 15:59:27,983: INFO/job_scheduler] Job 65 finished 1 steps successfully
[2017-06-14 15:59:28,021: INFO/job_scheduler] Job 65: complete_job: Updating cache
[2017-06-14 15:59:28,022: INFO/job_scheduler] Job 65 completing (errored=False, cancelled=False)
[2017-06-14 15:59:28,124: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-14 15:59:28,169: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com install_packages {'repos': [u'iml-agent'], 'packages': [u'zfs', u'zfs-dkms', u'kernel-devel-[0-9]*_lustre', u'kernel-[0-9]*_lustre', u'lustre-osd-zfs-mount', u'kmod-lustre-osd-ldiskfs', u'lustre-dkms', u'lustre', u'chroma-agent-management']}
[2017-06-14 15:59:28,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0a065f0d-3071-4a23-b9a3-9fa373f77bfa
[2017-06-14 15:59:28,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0a065f0d-3071-4a23-b9a3-9fa373f77bfa
[2017-06-14 15:59:34,716: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 15:59:34,716: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 15:59:34,716: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 15:59:34,716: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 15:59:34,716: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0e8ed477-8fc5-4fc1-981c-123ca9c182eb
[2017-06-14 15:59:34,716: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0e8ed477-8fc5-4fc1-981c-123ca9c182eb
[2017-06-14 15:59:34,740: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0e8ed477-8fc5-4fc1-981c-123ca9c182eb
[2017-06-14 15:59:34,740: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0e8ed477-8fc5-4fc1-981c-123ca9c182eb
[2017-06-14 15:59:35,293: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com install_packages {'repos': [u'iml-agent'], 'packages': [u'zfs', u'zfs-dkms', u'kernel-devel-[0-9]*_lustre', u'kernel-[0-9]*_lustre', u'lustre-osd-zfs-mount', u'kmod-lustre-osd-ldiskfs', u'lustre-dkms', u'lustre', u'chroma-agent-management']}
[2017-06-14 15:59:35,311: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6df271f1-9b49-4802-aee2-b5c93ff44af3
[2017-06-14 15:59:35,311: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6df271f1-9b49-4802-aee2-b5c93ff44af3
[2017-06-14 15:59:44,896: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:44,896: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:44,896: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:44,896: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:44,896: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3bb9e585-75cb-40a6-b220-b1d716b32567
[2017-06-14 15:59:44,896: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3bb9e585-75cb-40a6-b220-b1d716b32567
[2017-06-14 15:59:44,909: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3bb9e585-75cb-40a6-b220-b1d716b32567
[2017-06-14 15:59:44,909: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3bb9e585-75cb-40a6-b220-b1d716b32567
[2017-06-14 15:59:44,954: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com kernel_status {}
[2017-06-14 15:59:44,968: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b88ff3c0-b2b5-41c6-a9ca-f5766f4887a2
[2017-06-14 15:59:44,968: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b88ff3c0-b2b5-41c6-a9ca-f5766f4887a2
[2017-06-14 15:59:47,141: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a
[2017-06-14 15:59:47,141: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a
[2017-06-14 15:59:47,142: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a
[2017-06-14 15:59:47,142: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a
[2017-06-14 15:59:47,142: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 233fdcc1-e10f-46db-a26e-9476895f4cc1
[2017-06-14 15:59:47,142: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 233fdcc1-e10f-46db-a26e-9476895f4cc1
[2017-06-14 15:59:47,169: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 233fdcc1-e10f-46db-a26e-9476895f4cc1
[2017-06-14 15:59:47,169: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 233fdcc1-e10f-46db-a26e-9476895f4cc1
[2017-06-14 15:59:47,849: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com install_packages {'repos': [u'iml-agent'], 'packages': [u'zfs', u'zfs-dkms', u'kernel-devel-[0-9]*_lustre', u'kernel-[0-9]*_lustre', u'lustre-osd-zfs-mount', u'kmod-lustre-osd-ldiskfs', u'lustre-dkms', u'lustre', u'chroma-agent-management']}
[2017-06-14 15:59:47,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 36ac2c91-887a-4c1a-81b1-6b4d66f8358b
[2017-06-14 15:59:47,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 36ac2c91-887a-4c1a-81b1-6b4d66f8358b
[2017-06-14 15:59:48,091: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:48,091: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:48,091: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:48,091: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:48,091: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b88ff3c0-b2b5-41c6-a9ca-f5766f4887a2
[2017-06-14 15:59:48,091: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b88ff3c0-b2b5-41c6-a9ca-f5766f4887a2
[2017-06-14 15:59:48,143: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b88ff3c0-b2b5-41c6-a9ca-f5766f4887a2
[2017-06-14 15:59:48,143: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b88ff3c0-b2b5-41c6-a9ca-f5766f4887a2
[2017-06-14 15:59:48,144: INFO/job_scheduler] Job 43 finished 4 steps successfully
[2017-06-14 15:59:48,253: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 140623806366736
[2017-06-14 15:59:48,257: INFO/job] Job 43: StateChangeJob complete, setting state packages_installed on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-14 15:59:48,257: INFO/job] Job 43: not_deleted=True
[2017-06-14 15:59:48,257: INFO/job_scheduler] Job 43: complete_job: Updating cache
[2017-06-14 15:59:48,257: INFO/job_scheduler] Job 43: locked_item 140623806239120 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-14 15:59:48,260: INFO/job_scheduler] Job 43 completing (errored=False, cancelled=False)
[2017-06-14 15:59:48,301: INFO/job_scheduler] run_next: 4 runnable jobs of (40 pending, 3 tasked)
[2017-06-14 15:59:48,394: INFO/job_scheduler] Job 44: RunJobThread.run
[2017-06-14 15:59:48,399: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-06-14 15:59:48,421: INFO/job_scheduler] Job 45: RunJobThread.run
[2017-06-14 15:59:48,422: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-06-14 15:59:48,426: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1e7d984e-8f0e-450e-b0ee-270d88426da3
[2017-06-14 15:59:48,426: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1e7d984e-8f0e-450e-b0ee-270d88426da3
[2017-06-14 15:59:48,437: INFO/job_scheduler] Job 46: RunJobThread.run
[2017-06-14 15:59:48,437: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-06-14 15:59:48,438: INFO/job_scheduler] Job 47: open DB connection during completion
[2017-06-14 15:59:48,483: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e90dd55c-39a2-4405-82ad-9adda5196970
[2017-06-14 15:59:48,483: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e90dd55c-39a2-4405-82ad-9adda5196970
[2017-06-14 15:59:48,519: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 140624277303760
[2017-06-14 15:59:48,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8dac0c29-a7e3-45c0-9726-c5085cd2d93b
[2017-06-14 15:59:48,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8dac0c29-a7e3-45c0-9726-c5085cd2d93b
[2017-06-14 15:59:48,541: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-06-14 15:59:48,558: INFO/job] Job 47: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-14 15:59:48,558: INFO/job] Job 47: not_deleted=True
[2017-06-14 15:59:48,558: INFO/job_scheduler] Job 47: complete_job: Updating cache
[2017-06-14 15:59:48,558: INFO/job_scheduler] Job 47: locked_item 140624277303760 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-14 15:59:48,573: INFO/job_scheduler] Job 47 completing (errored=False, cancelled=False)
[2017-06-14 15:59:48,609: INFO/job_scheduler] run_next: 1 runnable jobs of (36 pending, 6 tasked)
[2017-06-14 15:59:48,639: INFO/job_scheduler] Job 48: RunJobThread.run
[2017-06-14 15:59:48,640: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com load_lnet {}
[2017-06-14 15:59:48,685: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e0fafe0e-90b2-4a68-8349-bdc7b1426358
[2017-06-14 15:59:48,685: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e0fafe0e-90b2-4a68-8349-bdc7b1426358
[2017-06-14 15:59:49,757: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:49,757: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:49,758: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:49,758: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:49,758: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e0fafe0e-90b2-4a68-8349-bdc7b1426358
[2017-06-14 15:59:49,758: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e0fafe0e-90b2-4a68-8349-bdc7b1426358
[2017-06-14 15:59:49,760: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e0fafe0e-90b2-4a68-8349-bdc7b1426358
[2017-06-14 15:59:49,760: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e0fafe0e-90b2-4a68-8349-bdc7b1426358
[2017-06-14 15:59:49,761: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-14 15:59:49,789: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e22f9f08-c3c9-483d-b583-5dfd3e3affff
[2017-06-14 15:59:49,789: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e22f9f08-c3c9-483d-b583-5dfd3e3affff
[2017-06-14 15:59:50,886: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:50,886: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:50,947: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:50,947: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:50,947: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e22f9f08-c3c9-483d-b583-5dfd3e3affff
[2017-06-14 15:59:50,947: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e22f9f08-c3c9-483d-b583-5dfd3e3affff
[2017-06-14 15:59:50,963: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e22f9f08-c3c9-483d-b583-5dfd3e3affff
[2017-06-14 15:59:50,963: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e22f9f08-c3c9-483d-b583-5dfd3e3affff
[2017-06-14 15:59:51,318: INFO/job_scheduler] Job 48 finished 2 steps successfully
[2017-06-14 15:59:51,368: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 140624277303760
[2017-06-14 15:59:51,373: INFO/job] Job 48: StateChangeJob complete, setting state lnet_down on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-14 15:59:51,373: INFO/job] Job 48: not_deleted=True
[2017-06-14 15:59:51,373: INFO/job_scheduler] Job 48: complete_job: Updating cache
[2017-06-14 15:59:51,373: INFO/job_scheduler] Job 48: locked_item 140624277303760 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-14 15:59:51,377: INFO/job_scheduler] Job 48 completing (errored=False, cancelled=False)
[2017-06-14 15:59:51,407: INFO/job_scheduler] run_next: 1 runnable jobs of (35 pending, 6 tasked)
[2017-06-14 15:59:51,423: INFO/job_scheduler] Job 51: RunJobThread.run
[2017-06-14 15:59:51,423: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_lnet {}
[2017-06-14 15:59:51,447: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cc03f47c-79b0-4c6d-9176-85d4f9495bc5
[2017-06-14 15:59:51,447: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cc03f47c-79b0-4c6d-9176-85d4f9495bc5
[2017-06-14 15:59:54,521: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:54,521: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:54,522: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:54,522: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:54,522: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1e7d984e-8f0e-450e-b0ee-270d88426da3
[2017-06-14 15:59:54,522: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1e7d984e-8f0e-450e-b0ee-270d88426da3
[2017-06-14 15:59:54,550: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1e7d984e-8f0e-450e-b0ee-270d88426da3
[2017-06-14 15:59:54,550: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1e7d984e-8f0e-450e-b0ee-270d88426da3
[2017-06-14 15:59:54,551: INFO/job_scheduler] Job 44 finished 1 steps successfully
[2017-06-14 15:59:54,835: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 140623202470096
[2017-06-14 15:59:54,839: INFO/job] Job 44: StateChangeJob complete, setting state configured on lotus-32vm7.lotus.hpdd.lab.intel.com NTP configuration
[2017-06-14 15:59:54,839: INFO/job] Job 44: not_deleted=True
[2017-06-14 15:59:54,839: INFO/job_scheduler] Job 44: complete_job: Updating cache
[2017-06-14 15:59:54,839: INFO/job_scheduler] Job 44: locked_item 140623202470096 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-06-14 15:59:54,842: INFO/job_scheduler] Job 44 completing (errored=False, cancelled=False)
[2017-06-14 15:59:54,888: INFO/job_scheduler] run_next: 0 runnable jobs of (34 pending, 6 tasked)
[2017-06-14 15:59:55,572: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:55,572: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:55,572: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:55,572: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:55,573: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cc03f47c-79b0-4c6d-9176-85d4f9495bc5
[2017-06-14 15:59:55,573: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cc03f47c-79b0-4c6d-9176-85d4f9495bc5
[2017-06-14 15:59:55,573: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:55,573: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:55,574: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:55,574: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:55,574: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e90dd55c-39a2-4405-82ad-9adda5196970
[2017-06-14 15:59:55,574: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e90dd55c-39a2-4405-82ad-9adda5196970
[2017-06-14 15:59:55,607: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e90dd55c-39a2-4405-82ad-9adda5196970
[2017-06-14 15:59:55,607: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e90dd55c-39a2-4405-82ad-9adda5196970
[2017-06-14 15:59:55,608: INFO/job_scheduler] Job 45 finished 1 steps successfully
[2017-06-14 15:59:55,619: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cc03f47c-79b0-4c6d-9176-85d4f9495bc5
[2017-06-14 15:59:55,619: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cc03f47c-79b0-4c6d-9176-85d4f9495bc5
[2017-06-14 15:59:55,620: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-14 15:59:55,638: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b5ef7aa8-3fde-4549-917d-5130136f9b4b
[2017-06-14 15:59:55,638: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b5ef7aa8-3fde-4549-917d-5130136f9b4b
[2017-06-14 15:59:55,701: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 140623202468240
[2017-06-14 15:59:55,708: INFO/job] Job 45: StateChangeJob complete, setting state configured on lotus-32vm7.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-06-14 15:59:55,708: INFO/job] Job 45: not_deleted=True
[2017-06-14 15:59:55,708: INFO/job_scheduler] Job 45: complete_job: Updating cache
[2017-06-14 15:59:55,708: INFO/job_scheduler] Job 45: locked_item 140623202468240 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-06-14 15:59:55,715: INFO/job_scheduler] Job 45 completing (errored=False, cancelled=False)
[2017-06-14 15:59:55,741: INFO/job_scheduler] run_next: 0 runnable jobs of (34 pending, 5 tasked)
[2017-06-14 15:59:56,732: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:56,732: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:56,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:56,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 15:59:56,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b5ef7aa8-3fde-4549-917d-5130136f9b4b
[2017-06-14 15:59:56,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b5ef7aa8-3fde-4549-917d-5130136f9b4b
[2017-06-14 15:59:56,765: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b5ef7aa8-3fde-4549-917d-5130136f9b4b
[2017-06-14 15:59:56,765: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b5ef7aa8-3fde-4549-917d-5130136f9b4b
[2017-06-14 15:59:56,911: INFO/job_scheduler] Job 51 finished 2 steps successfully
[2017-06-14 15:59:56,960: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 140624277303760
[2017-06-14 15:59:56,964: INFO/job] Job 51: StateChangeJob complete, setting state lnet_up on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-14 15:59:56,964: INFO/job] Job 51: not_deleted=True
[2017-06-14 15:59:56,964: INFO/job_scheduler] Job 51: complete_job: Updating cache
[2017-06-14 15:59:56,964: INFO/job_scheduler] Job 51: locked_item 140624277303760 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-14 15:59:56,968: INFO/job_scheduler] Job 51 completing (errored=False, cancelled=False)
[2017-06-14 15:59:57,012: INFO/job_scheduler] run_next: 0 runnable jobs of (34 pending, 4 tasked)
[2017-06-14 15:59:57,373: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/28bd4119-43c0-46cc-95d1-202fa8408bad
[2017-06-14 15:59:57,373: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/28bd4119-43c0-46cc-95d1-202fa8408bad
[2017-06-14 15:59:57,373: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/28bd4119-43c0-46cc-95d1-202fa8408bad
[2017-06-14 15:59:57,373: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/28bd4119-43c0-46cc-95d1-202fa8408bad
[2017-06-14 15:59:57,373: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0a065f0d-3071-4a23-b9a3-9fa373f77bfa
[2017-06-14 15:59:57,373: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0a065f0d-3071-4a23-b9a3-9fa373f77bfa
[2017-06-14 15:59:57,391: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0a065f0d-3071-4a23-b9a3-9fa373f77bfa
[2017-06-14 15:59:57,391: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0a065f0d-3071-4a23-b9a3-9fa373f77bfa
[2017-06-14 15:59:57,496: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com kernel_status {}
[2017-06-14 15:59:57,541: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 64560673-6436-4d47-9dad-fdc05bb74672
[2017-06-14 15:59:57,541: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 64560673-6436-4d47-9dad-fdc05bb74672
[2017-06-14 15:59:59,649: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/28bd4119-43c0-46cc-95d1-202fa8408bad
[2017-06-14 15:59:59,649: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/28bd4119-43c0-46cc-95d1-202fa8408bad
[2017-06-14 15:59:59,649: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/28bd4119-43c0-46cc-95d1-202fa8408bad
[2017-06-14 15:59:59,649: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/28bd4119-43c0-46cc-95d1-202fa8408bad
[2017-06-14 15:59:59,650: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 64560673-6436-4d47-9dad-fdc05bb74672
[2017-06-14 15:59:59,650: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 64560673-6436-4d47-9dad-fdc05bb74672
[2017-06-14 15:59:59,664: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 64560673-6436-4d47-9dad-fdc05bb74672
[2017-06-14 15:59:59,664: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 64560673-6436-4d47-9dad-fdc05bb74672
[2017-06-14 15:59:59,664: INFO/job] Job 21 RebootIfNeededStep: Reboot of lotus-32vm5.lotus.hpdd.lab.intel.com required to switch from running kernel kernel-3.10.0-514.10.2.el7_lustre.x86_64 to required kernel-3.10.0-514.2.2.el7_lustre.x86_64
[2017-06-14 15:59:59,665: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com reboot_server {}
[2017-06-14 15:59:59,677: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 73ede109-7465-42aa-b7f7-1dbe3750b345
[2017-06-14 15:59:59,677: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 73ede109-7465-42aa-b7f7-1dbe3750b345
[2017-06-14 15:59:59,769: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/28bd4119-43c0-46cc-95d1-202fa8408bad
[2017-06-14 15:59:59,769: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/28bd4119-43c0-46cc-95d1-202fa8408bad
[2017-06-14 15:59:59,769: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/28bd4119-43c0-46cc-95d1-202fa8408bad
[2017-06-14 15:59:59,769: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/28bd4119-43c0-46cc-95d1-202fa8408bad
[2017-06-14 15:59:59,769: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 73ede109-7465-42aa-b7f7-1dbe3750b345
[2017-06-14 15:59:59,769: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 73ede109-7465-42aa-b7f7-1dbe3750b345
[2017-06-14 15:59:59,791: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 73ede109-7465-42aa-b7f7-1dbe3750b345
[2017-06-14 15:59:59,791: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 73ede109-7465-42aa-b7f7-1dbe3750b345
[2017-06-14 15:59:59,792: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old 28bd4119-43c0-46cc-95d1-202fa8408bad)
[2017-06-14 15:59:59,792: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old 28bd4119-43c0-46cc-95d1-202fa8408bad)
[2017-06-14 16:00:03,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:03,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:03,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:03,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:03,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6df271f1-9b49-4802-aee2-b5c93ff44af3
[2017-06-14 16:00:03,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6df271f1-9b49-4802-aee2-b5c93ff44af3
[2017-06-14 16:00:03,513: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6df271f1-9b49-4802-aee2-b5c93ff44af3
[2017-06-14 16:00:03,513: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6df271f1-9b49-4802-aee2-b5c93ff44af3
[2017-06-14 16:00:03,588: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com kernel_status {}
[2017-06-14 16:00:03,605: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ff777729-27e8-4ded-b5f4-255bcca9259a
[2017-06-14 16:00:03,605: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ff777729-27e8-4ded-b5f4-255bcca9259a
[2017-06-14 16:00:05,701: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:05,701: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:05,701: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:05,701: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:05,701: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ff777729-27e8-4ded-b5f4-255bcca9259a
[2017-06-14 16:00:05,701: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ff777729-27e8-4ded-b5f4-255bcca9259a
[2017-06-14 16:00:05,727: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ff777729-27e8-4ded-b5f4-255bcca9259a
[2017-06-14 16:00:05,727: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ff777729-27e8-4ded-b5f4-255bcca9259a
[2017-06-14 16:00:05,727: INFO/job_scheduler] Job 32 finished 4 steps successfully
[2017-06-14 16:00:05,775: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 140623806381392
[2017-06-14 16:00:05,779: INFO/job] Job 32: StateChangeJob complete, setting state packages_installed on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-14 16:00:05,779: INFO/job] Job 32: not_deleted=True
[2017-06-14 16:00:05,779: INFO/job_scheduler] Job 32: complete_job: Updating cache
[2017-06-14 16:00:05,779: INFO/job_scheduler] Job 32: locked_item 140623202468368 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-14 16:00:05,782: INFO/job_scheduler] Job 32 completing (errored=False, cancelled=False)
[2017-06-14 16:00:05,805: INFO/job_scheduler] run_next: 4 runnable jobs of (34 pending, 3 tasked)
[2017-06-14 16:00:05,859: INFO/job_scheduler] Job 33: RunJobThread.run
[2017-06-14 16:00:05,859: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-06-14 16:00:05,880: INFO/job_scheduler] Job 34: RunJobThread.run
[2017-06-14 16:00:05,889: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-06-14 16:00:05,904: INFO/job_scheduler] Job 35: RunJobThread.run
[2017-06-14 16:00:05,905: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-06-14 16:00:05,905: INFO/job_scheduler] Job 36: open DB connection during completion
[2017-06-14 16:00:05,911: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5a983260-7dc1-4da2-89c1-4907c3f49b25
[2017-06-14 16:00:05,911: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5a983260-7dc1-4da2-89c1-4907c3f49b25
[2017-06-14 16:00:05,950: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 140624277303376
[2017-06-14 16:00:05,963: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1ff694ea-63c9-43b5-a36d-686026362060
[2017-06-14 16:00:05,963: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1ff694ea-63c9-43b5-a36d-686026362060
[2017-06-14 16:00:05,973: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-06-14 16:00:05,986: INFO/job] Job 36: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-14 16:00:05,986: INFO/job] Job 36: not_deleted=True
[2017-06-14 16:00:05,986: INFO/job_scheduler] Job 36: complete_job: Updating cache
[2017-06-14 16:00:05,986: INFO/job_scheduler] Job 36: locked_item 140624277303376 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-14 16:00:05,989: INFO/job_scheduler] Job 36 completing (errored=False, cancelled=False)
[2017-06-14 16:00:06,017: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8bbcda4d-ea10-4e5a-9716-89040143e63b
[2017-06-14 16:00:06,017: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8bbcda4d-ea10-4e5a-9716-89040143e63b
[2017-06-14 16:00:06,058: INFO/job_scheduler] run_next: 1 runnable jobs of (30 pending, 6 tasked)
[2017-06-14 16:00:06,080: INFO/job_scheduler] Job 37: RunJobThread.run
[2017-06-14 16:00:06,081: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com load_lnet {}
[2017-06-14 16:00:06,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d276a926-3135-480d-89dc-05f35f58f894
[2017-06-14 16:00:06,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d276a926-3135-480d-89dc-05f35f58f894
[2017-06-14 16:00:07,210: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:07,210: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:07,210: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:07,210: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:07,211: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d276a926-3135-480d-89dc-05f35f58f894
[2017-06-14 16:00:07,211: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d276a926-3135-480d-89dc-05f35f58f894
[2017-06-14 16:00:07,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d276a926-3135-480d-89dc-05f35f58f894
[2017-06-14 16:00:07,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d276a926-3135-480d-89dc-05f35f58f894
[2017-06-14 16:00:07,244: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-14 16:00:07,272: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9926b9e8-fd1e-439f-8a4f-93fe71b9605c
[2017-06-14 16:00:07,272: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9926b9e8-fd1e-439f-8a4f-93fe71b9605c
[2017-06-14 16:00:08,329: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:08,329: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:08,329: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:08,329: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:08,329: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9926b9e8-fd1e-439f-8a4f-93fe71b9605c
[2017-06-14 16:00:08,329: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9926b9e8-fd1e-439f-8a4f-93fe71b9605c
[2017-06-14 16:00:08,341: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9926b9e8-fd1e-439f-8a4f-93fe71b9605c
[2017-06-14 16:00:08,341: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9926b9e8-fd1e-439f-8a4f-93fe71b9605c
[2017-06-14 16:00:08,618: INFO/job_scheduler] Job 37 finished 2 steps successfully
[2017-06-14 16:00:08,639: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 140624277303376
[2017-06-14 16:00:08,652: INFO/job] Job 37: StateChangeJob complete, setting state lnet_down on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-14 16:00:08,653: INFO/job] Job 37: not_deleted=True
[2017-06-14 16:00:08,653: INFO/job_scheduler] Job 37: complete_job: Updating cache
[2017-06-14 16:00:08,653: INFO/job_scheduler] Job 37: locked_item 140624277303376 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-14 16:00:08,657: INFO/job_scheduler] Job 37 completing (errored=False, cancelled=False)
[2017-06-14 16:00:08,692: INFO/job_scheduler] run_next: 1 runnable jobs of (29 pending, 6 tasked)
[2017-06-14 16:00:08,708: INFO/job_scheduler] Job 40: RunJobThread.run
[2017-06-14 16:00:08,709: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_lnet {}
[2017-06-14 16:00:08,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3baea97a-97d4-485c-99a9-5367403f9823
[2017-06-14 16:00:08,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3baea97a-97d4-485c-99a9-5367403f9823
[2017-06-14 16:00:10,835: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:10,835: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:10,835: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:10,835: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:10,835: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3baea97a-97d4-485c-99a9-5367403f9823
[2017-06-14 16:00:10,835: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3baea97a-97d4-485c-99a9-5367403f9823
[2017-06-14 16:00:10,850: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3baea97a-97d4-485c-99a9-5367403f9823
[2017-06-14 16:00:10,850: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3baea97a-97d4-485c-99a9-5367403f9823
[2017-06-14 16:00:10,851: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-14 16:00:10,864: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 13d06063-9d9a-4624-b32d-05a01a111ef9
[2017-06-14 16:00:10,864: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 13d06063-9d9a-4624-b32d-05a01a111ef9
[2017-06-14 16:00:10,912: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:10,912: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:10,912: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:10,912: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:10,912: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 13d06063-9d9a-4624-b32d-05a01a111ef9
[2017-06-14 16:00:10,912: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 13d06063-9d9a-4624-b32d-05a01a111ef9
[2017-06-14 16:00:10,932: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 13d06063-9d9a-4624-b32d-05a01a111ef9
[2017-06-14 16:00:10,932: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 13d06063-9d9a-4624-b32d-05a01a111ef9
[2017-06-14 16:00:11,501: INFO/job_scheduler] Job 40 finished 2 steps successfully
[2017-06-14 16:00:11,550: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 140624277303376
[2017-06-14 16:00:11,554: INFO/job] Job 40: StateChangeJob complete, setting state lnet_up on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-14 16:00:11,555: INFO/job] Job 40: not_deleted=True
[2017-06-14 16:00:11,555: INFO/job_scheduler] Job 40: complete_job: Updating cache
[2017-06-14 16:00:11,555: INFO/job_scheduler] Job 40: locked_item 140624277303376 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-14 16:00:11,558: INFO/job_scheduler] Job 40 completing (errored=False, cancelled=False)
[2017-06-14 16:00:11,591: INFO/job_scheduler] run_next: 0 runnable jobs of (28 pending, 6 tasked)
[2017-06-14 16:00:12,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:12,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:12,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:12,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:12,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1ff694ea-63c9-43b5-a36d-686026362060
[2017-06-14 16:00:12,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1ff694ea-63c9-43b5-a36d-686026362060
[2017-06-14 16:00:12,091: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1ff694ea-63c9-43b5-a36d-686026362060
[2017-06-14 16:00:12,091: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1ff694ea-63c9-43b5-a36d-686026362060
[2017-06-14 16:00:12,092: INFO/job_scheduler] Job 34 finished 1 steps successfully
[2017-06-14 16:00:12,230: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 140624277304464
[2017-06-14 16:00:12,233: INFO/job] Job 34: StateChangeJob complete, setting state configured on lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration
[2017-06-14 16:00:12,233: INFO/job] Job 34: not_deleted=True
[2017-06-14 16:00:12,233: INFO/job_scheduler] Job 34: complete_job: Updating cache
[2017-06-14 16:00:12,233: INFO/job_scheduler] Job 34: locked_item 140624277304464 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-06-14 16:00:12,236: INFO/job_scheduler] Job 34 completing (errored=False, cancelled=False)
[2017-06-14 16:00:12,286: INFO/job_scheduler] run_next: 0 runnable jobs of (28 pending, 5 tasked)
[2017-06-14 16:00:13,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:13,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:13,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:13,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:13,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5a983260-7dc1-4da2-89c1-4907c3f49b25
[2017-06-14 16:00:13,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5a983260-7dc1-4da2-89c1-4907c3f49b25
[2017-06-14 16:00:13,086: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5a983260-7dc1-4da2-89c1-4907c3f49b25
[2017-06-14 16:00:13,086: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5a983260-7dc1-4da2-89c1-4907c3f49b25
[2017-06-14 16:00:13,088: INFO/job_scheduler] Job 33 finished 1 steps successfully
[2017-06-14 16:00:13,229: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 140624277303568
[2017-06-14 16:00:13,232: INFO/job] Job 33: StateChangeJob complete, setting state configured on lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-06-14 16:00:13,233: INFO/job] Job 33: not_deleted=True
[2017-06-14 16:00:13,233: INFO/job_scheduler] Job 33: complete_job: Updating cache
[2017-06-14 16:00:13,233: INFO/job_scheduler] Job 33: locked_item 140624277303568 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-06-14 16:00:13,236: INFO/job_scheduler] Job 33 completing (errored=False, cancelled=False)
[2017-06-14 16:00:13,265: INFO/job_scheduler] run_next: 0 runnable jobs of (28 pending, 4 tasked)
[2017-06-14 16:00:17,142: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a
[2017-06-14 16:00:17,142: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a
[2017-06-14 16:00:17,143: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a
[2017-06-14 16:00:17,143: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a
[2017-06-14 16:00:17,143: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 36ac2c91-887a-4c1a-81b1-6b4d66f8358b
[2017-06-14 16:00:17,143: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 36ac2c91-887a-4c1a-81b1-6b4d66f8358b
[2017-06-14 16:00:17,172: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 36ac2c91-887a-4c1a-81b1-6b4d66f8358b
[2017-06-14 16:00:17,172: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 36ac2c91-887a-4c1a-81b1-6b4d66f8358b
[2017-06-14 16:00:17,227: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com kernel_status {}
[2017-06-14 16:00:17,246: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 84011976-af21-459d-89fa-fb4a0e74ff3f
[2017-06-14 16:00:17,246: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 84011976-af21-459d-89fa-fb4a0e74ff3f
[2017-06-14 16:00:19,337: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a
[2017-06-14 16:00:19,337: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a
[2017-06-14 16:00:19,338: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a
[2017-06-14 16:00:19,338: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a
[2017-06-14 16:00:19,338: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 84011976-af21-459d-89fa-fb4a0e74ff3f
[2017-06-14 16:00:19,338: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 84011976-af21-459d-89fa-fb4a0e74ff3f
[2017-06-14 16:00:19,368: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 84011976-af21-459d-89fa-fb4a0e74ff3f
[2017-06-14 16:00:19,368: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 84011976-af21-459d-89fa-fb4a0e74ff3f
[2017-06-14 16:00:19,368: INFO/job] Job 54 RebootIfNeededStep: Reboot of lotus-32vm8.lotus.hpdd.lab.intel.com required to switch from running kernel kernel-3.10.0-514.10.2.el7_lustre.x86_64 to required kernel-3.10.0-514.2.2.el7_lustre.x86_64
[2017-06-14 16:00:19,369: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com reboot_server {}
[2017-06-14 16:00:19,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 19b17299-e290-4b08-aa4b-9aa49ab12de4
[2017-06-14 16:00:19,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 19b17299-e290-4b08-aa4b-9aa49ab12de4
[2017-06-14 16:00:19,457: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a
[2017-06-14 16:00:19,457: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a
[2017-06-14 16:00:19,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a
[2017-06-14 16:00:19,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a
[2017-06-14 16:00:19,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 19b17299-e290-4b08-aa4b-9aa49ab12de4
[2017-06-14 16:00:19,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 19b17299-e290-4b08-aa4b-9aa49ab12de4
[2017-06-14 16:00:19,499: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 19b17299-e290-4b08-aa4b-9aa49ab12de4
[2017-06-14 16:00:19,499: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 19b17299-e290-4b08-aa4b-9aa49ab12de4
[2017-06-14 16:00:19,500: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm8.lotus.hpdd.lab.intel.com (old e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a)
[2017-06-14 16:00:19,500: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm8.lotus.hpdd.lab.intel.com (old e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a)
[2017-06-14 16:00:34,802: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:00:34,802: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:00:34,802: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:00:34,802: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:00:34,802: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8dac0c29-a7e3-45c0-9726-c5085cd2d93b
[2017-06-14 16:00:34,802: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8dac0c29-a7e3-45c0-9726-c5085cd2d93b
[2017-06-14 16:00:34,845: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8dac0c29-a7e3-45c0-9726-c5085cd2d93b
[2017-06-14 16:00:34,845: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8dac0c29-a7e3-45c0-9726-c5085cd2d93b
[2017-06-14 16:00:34,846: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.226', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-06-14 16:00:34,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3bd37b33-c59c-4673-823e-7f7ff21a10ea
[2017-06-14 16:00:34,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3bd37b33-c59c-4673-823e-7f7ff21a10ea
[2017-06-14 16:00:34,921: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:00:34,921: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:00:34,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:00:34,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:00:34,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3bd37b33-c59c-4673-823e-7f7ff21a10ea
[2017-06-14 16:00:34,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3bd37b33-c59c-4673-823e-7f7ff21a10ea
[2017-06-14 16:00:34,925: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3bd37b33-c59c-4673-823e-7f7ff21a10ea
[2017-06-14 16:00:34,925: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3bd37b33-c59c-4673-823e-7f7ff21a10ea
[2017-06-14 16:00:34,938: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 49429, 'pcs_password': 'yqwcl617EOio5UzpY2hF'}
[2017-06-14 16:00:34,956: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f81c7774-6750-41de-ac1a-9015d7041a68
[2017-06-14 16:00:34,956: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f81c7774-6750-41de-ac1a-9015d7041a68
[2017-06-14 16:00:47,057: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:00:47,057: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:00:47,057: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:00:47,057: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:00:47,057: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f81c7774-6750-41de-ac1a-9015d7041a68
[2017-06-14 16:00:47,057: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f81c7774-6750-41de-ac1a-9015d7041a68
[2017-06-14 16:00:47,080: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f81c7774-6750-41de-ac1a-9015d7041a68
[2017-06-14 16:00:47,080: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f81c7774-6750-41de-ac1a-9015d7041a68
[2017-06-14 16:00:47,081: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': True, 'mcast_port': 49429, 'new_node_fqdn': u'lotus-32vm7.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'yqwcl617EOio5UzpY2hF'}
[2017-06-14 16:00:47,095: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d3f5dfb9-154f-42d9-946e-d1f4b53c0aca
[2017-06-14 16:00:47,095: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d3f5dfb9-154f-42d9-946e-d1f4b53c0aca
[2017-06-14 16:00:53,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:00:53,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:00:53,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:00:53,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:00:53,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d3f5dfb9-154f-42d9-946e-d1f4b53c0aca
[2017-06-14 16:00:53,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d3f5dfb9-154f-42d9-946e-d1f4b53c0aca
[2017-06-14 16:00:53,217: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d3f5dfb9-154f-42d9-946e-d1f4b53c0aca
[2017-06-14 16:00:53,217: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d3f5dfb9-154f-42d9-946e-d1f4b53c0aca
[2017-06-14 16:00:53,218: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-06-14 16:00:53.218012+00:00:
[2017-06-14 16:00:53,218: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-06-14 16:00:53.218012+00:00:
[2017-06-14 16:00:53,218: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->49429
[2017-06-14 16:00:53,218: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->49429
[2017-06-14 16:00:53,226: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-06-14 16:00:53,226: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-06-14 16:00:53,249: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:00:53,249: INFO/job_scheduler]   Job 46 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-06-14 16:00:53,249: INFO/job_scheduler]   Job 49 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-06-14 16:00:53,249: INFO/job_scheduler]   Job 50 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:00:53,249: INFO/job_scheduler]   Job 52 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:00:53,250: INFO/job_scheduler] run_next: 0 runnable jobs of (28 pending, 4 tasked)
[2017-06-14 16:00:53,251: INFO/job_scheduler] Job 46 finished 1 steps successfully
[2017-06-14 16:00:53,276: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 140623806328656
[2017-06-14 16:00:53,288: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-06-14 16:00:53,306: INFO/job] Job 46: StateChangeJob complete, setting state stopped on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:00:53,306: INFO/job] Job 46: not_deleted=True
[2017-06-14 16:00:53,306: INFO/job_scheduler] Job 46: complete_job: Updating cache
[2017-06-14 16:00:53,306: INFO/job_scheduler] Job 46: locked_item 140623806328656 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-14 16:00:53,315: INFO/job_scheduler] Job 46 completing (errored=False, cancelled=False)
[2017-06-14 16:00:53,351: INFO/job_scheduler] run_next: 1 runnable jobs of (28 pending, 3 tasked)
[2017-06-14 16:00:53,362: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:53,362: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:53,362: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:53,362: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:53,363: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8bbcda4d-ea10-4e5a-9716-89040143e63b
[2017-06-14 16:00:53,363: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8bbcda4d-ea10-4e5a-9716-89040143e63b
[2017-06-14 16:00:53,371: INFO/job_scheduler] Job 49: RunJobThread.run
[2017-06-14 16:00:53,372: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-06-14 16:00:53,379: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8bbcda4d-ea10-4e5a-9716-89040143e63b
[2017-06-14 16:00:53,379: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8bbcda4d-ea10-4e5a-9716-89040143e63b
[2017-06-14 16:00:53,380: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.225', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-06-14 16:00:53,455: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d8a19a13-8de1-4dae-aeb1-9cf3f79d0190
[2017-06-14 16:00:53,455: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d8a19a13-8de1-4dae-aeb1-9cf3f79d0190
[2017-06-14 16:00:53,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc eb645add-86c5-44c9-9fa5-2bd079892e0d
[2017-06-14 16:00:53,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc eb645add-86c5-44c9-9fa5-2bd079892e0d
[2017-06-14 16:00:53,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:53,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:53,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:53,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:00:53,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc eb645add-86c5-44c9-9fa5-2bd079892e0d
[2017-06-14 16:00:53,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc eb645add-86c5-44c9-9fa5-2bd079892e0d
[2017-06-14 16:00:53,622: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc eb645add-86c5-44c9-9fa5-2bd079892e0d
[2017-06-14 16:00:53,622: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc eb645add-86c5-44c9-9fa5-2bd079892e0d
[2017-06-14 16:00:53,659: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 65087, 'pcs_password': 'e6Z83F2HRoKkLrUicD2h'}
[2017-06-14 16:00:53,683: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4d872c2d-982c-48a0-ad2b-1224268ec57c
[2017-06-14 16:00:53,683: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4d872c2d-982c-48a0-ad2b-1224268ec57c
[2017-06-14 16:00:58,884: INFO/job_scheduler] run_next: 0 runnable jobs of (27 pending, 4 tasked)
[2017-06-14 16:00:58,903: INFO/job_scheduler] run_next: 0 runnable jobs of (27 pending, 4 tasked)
[2017-06-14 16:01:01,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:01:01,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:01:01,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:01:01,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:01:01,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d8a19a13-8de1-4dae-aeb1-9cf3f79d0190
[2017-06-14 16:01:01,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d8a19a13-8de1-4dae-aeb1-9cf3f79d0190
[2017-06-14 16:01:01,627: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d8a19a13-8de1-4dae-aeb1-9cf3f79d0190
[2017-06-14 16:01:01,627: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d8a19a13-8de1-4dae-aeb1-9cf3f79d0190
[2017-06-14 16:01:01,627: INFO/job_scheduler] Job 49 finished 1 steps successfully
[2017-06-14 16:01:01,684: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 140623806328656
[2017-06-14 16:01:01,712: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 30
[2017-06-14 16:01:01,722: INFO/job] Job 49: StateChangeJob complete, setting state started on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:01:01,722: INFO/job] Job 49: not_deleted=True
[2017-06-14 16:01:01,722: INFO/job_scheduler] Job 49: complete_job: Updating cache
[2017-06-14 16:01:01,722: INFO/job_scheduler] Job 49: locked_item 140623806328656 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-14 16:01:01,726: INFO/job_scheduler] Job 49 completing (errored=False, cancelled=False)
[2017-06-14 16:01:01,749: INFO/job_scheduler] run_next: 1 runnable jobs of (27 pending, 3 tasked)
[2017-06-14 16:01:01,777: INFO/job_scheduler] Job 50: RunJobThread.run
[2017-06-14 16:01:01,778: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-06-14 16:01:01,793: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6865b169-96ce-43e7-a768-5e9b24e5dc0e
[2017-06-14 16:01:01,793: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6865b169-96ce-43e7-a768-5e9b24e5dc0e
[2017-06-14 16:01:06,892: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:01:06,892: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:01:06,892: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:01:06,892: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:01:06,893: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4d872c2d-982c-48a0-ad2b-1224268ec57c
[2017-06-14 16:01:06,893: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4d872c2d-982c-48a0-ad2b-1224268ec57c
[2017-06-14 16:01:06,911: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/28bd4119-43c0-46cc-95d1-202fa8408bad
[2017-06-14 16:01:06,911: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/28bd4119-43c0-46cc-95d1-202fa8408bad
[2017-06-14 16:01:06,911: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 28bd4119-43c0-46cc-95d1-202fa8408bad because session terminated
[2017-06-14 16:01:06,911: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 28bd4119-43c0-46cc-95d1-202fa8408bad because session terminated
[2017-06-14 16:01:06,914: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4d872c2d-982c-48a0-ad2b-1224268ec57c
[2017-06-14 16:01:06,914: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4d872c2d-982c-48a0-ad2b-1224268ec57c
[2017-06-14 16:01:06,914: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': True, 'mcast_port': 65087, 'new_node_fqdn': u'lotus-32vm6.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'e6Z83F2HRoKkLrUicD2h'}
[2017-06-14 16:01:06,931: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 37f2a3c9-6336-4eb7-9b3e-4a76e294b81a
[2017-06-14 16:01:06,931: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 37f2a3c9-6336-4eb7-9b3e-4a76e294b81a
[2017-06-14 16:01:08,927: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:01:08,927: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:01:08,927: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:01:08,927: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:01:08,928: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6865b169-96ce-43e7-a768-5e9b24e5dc0e
[2017-06-14 16:01:08,928: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6865b169-96ce-43e7-a768-5e9b24e5dc0e
[2017-06-14 16:01:08,944: INFO/job_scheduler] run_next: 0 runnable jobs of (26 pending, 4 tasked)
[2017-06-14 16:01:08,961: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6865b169-96ce-43e7-a768-5e9b24e5dc0e
[2017-06-14 16:01:08,961: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6865b169-96ce-43e7-a768-5e9b24e5dc0e
[2017-06-14 16:01:08,962: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-06-14 16:01:08,971: INFO/job_scheduler] run_next: 0 runnable jobs of (26 pending, 4 tasked)
[2017-06-14 16:01:08,977: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c105c65e-47e9-4bc6-abad-aa7776d3c9c3
[2017-06-14 16:01:08,977: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c105c65e-47e9-4bc6-abad-aa7776d3c9c3
[2017-06-14 16:01:13,001: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:01:13,001: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:01:13,001: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:01:13,001: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:01:13,001: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 37f2a3c9-6336-4eb7-9b3e-4a76e294b81a
[2017-06-14 16:01:13,001: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 37f2a3c9-6336-4eb7-9b3e-4a76e294b81a
[2017-06-14 16:01:13,004: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 37f2a3c9-6336-4eb7-9b3e-4a76e294b81a
[2017-06-14 16:01:13,004: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 37f2a3c9-6336-4eb7-9b3e-4a76e294b81a
[2017-06-14 16:01:13,004: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-06-14 16:01:13.004891+00:00:
[2017-06-14 16:01:13,004: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-06-14 16:01:13.004891+00:00:
[2017-06-14 16:01:13,005: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->65087
[2017-06-14 16:01:13,005: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->65087
[2017-06-14 16:01:13,006: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-06-14 16:01:13,006: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-06-14 16:01:13,020: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:01:13,020: INFO/job_scheduler]   Job 35 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-06-14 16:01:13,020: INFO/job_scheduler]   Job 38 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-06-14 16:01:13,020: INFO/job_scheduler]   Job 39 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:01:13,020: INFO/job_scheduler]   Job 41 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:01:13,021: INFO/job_scheduler] run_next: 0 runnable jobs of (26 pending, 4 tasked)
[2017-06-14 16:01:13,028: INFO/job_scheduler] Job 35 finished 1 steps successfully
[2017-06-14 16:01:13,089: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 140623806405456
[2017-06-14 16:01:13,096: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-06-14 16:01:13,111: INFO/job] Job 35: StateChangeJob complete, setting state stopped on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:01:13,111: INFO/job] Job 35: not_deleted=True
[2017-06-14 16:01:13,111: INFO/job_scheduler] Job 35: complete_job: Updating cache
[2017-06-14 16:01:13,111: INFO/job_scheduler] Job 35: locked_item 140623806405456 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-14 16:01:13,115: INFO/job_scheduler] Job 35 completing (errored=False, cancelled=False)
[2017-06-14 16:01:13,173: INFO/job_scheduler] run_next: 1 runnable jobs of (26 pending, 3 tasked)
[2017-06-14 16:01:13,203: INFO/job_scheduler] Job 38: RunJobThread.run
[2017-06-14 16:01:13,204: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-06-14 16:01:13,234: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 126d709d-147a-4f09-a550-e7c7a44fb435
[2017-06-14 16:01:13,234: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 126d709d-147a-4f09-a550-e7c7a44fb435
[2017-06-14 16:01:18,498: INFO/job_scheduler] run_next: 0 runnable jobs of (25 pending, 4 tasked)
[2017-06-14 16:01:18,531: INFO/job_scheduler] run_next: 0 runnable jobs of (25 pending, 4 tasked)
[2017-06-14 16:01:18,971: INFO/job_scheduler] run_next: 0 runnable jobs of (25 pending, 4 tasked)
[2017-06-14 16:01:18,986: INFO/job_scheduler] run_next: 0 runnable jobs of (25 pending, 4 tasked)
[2017-06-14 16:01:21,339: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:01:21,339: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:01:21,340: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:01:21,340: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:01:21,340: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 126d709d-147a-4f09-a550-e7c7a44fb435
[2017-06-14 16:01:21,340: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 126d709d-147a-4f09-a550-e7c7a44fb435
[2017-06-14 16:01:21,360: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 126d709d-147a-4f09-a550-e7c7a44fb435
[2017-06-14 16:01:21,360: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 126d709d-147a-4f09-a550-e7c7a44fb435
[2017-06-14 16:01:21,360: INFO/job_scheduler] Job 38 finished 1 steps successfully
[2017-06-14 16:01:21,414: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 140623806405456
[2017-06-14 16:01:21,456: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm6.lotus.hpdd.lab.intel.com at severity 30
[2017-06-14 16:01:21,479: INFO/job] Job 38: StateChangeJob complete, setting state started on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:01:21,479: INFO/job] Job 38: not_deleted=True
[2017-06-14 16:01:21,480: INFO/job_scheduler] Job 38: complete_job: Updating cache
[2017-06-14 16:01:21,480: INFO/job_scheduler] Job 38: locked_item 140623806405456 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-14 16:01:21,491: INFO/job_scheduler] Job 38 completing (errored=False, cancelled=False)
[2017-06-14 16:01:21,553: INFO/job_scheduler] run_next: 1 runnable jobs of (25 pending, 3 tasked)
[2017-06-14 16:01:21,603: INFO/job_scheduler] Job 39: RunJobThread.run
[2017-06-14 16:01:21,604: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-06-14 16:01:21,656: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c4b0cbf2-9fba-4187-a8c4-7aa8a149c507
[2017-06-14 16:01:21,656: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c4b0cbf2-9fba-4187-a8c4-7aa8a149c507
[2017-06-14 16:01:26,963: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a
[2017-06-14 16:01:26,963: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a
[2017-06-14 16:01:26,963: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a because session terminated
[2017-06-14 16:01:26,963: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session e61b9fb0-0b0a-44fd-bcda-6dd8df5ecd4a because session terminated
[2017-06-14 16:01:28,527: INFO/job_scheduler] run_next: 0 runnable jobs of (24 pending, 4 tasked)
[2017-06-14 16:01:28,543: INFO/job_scheduler] run_next: 0 runnable jobs of (24 pending, 4 tasked)
[2017-06-14 16:01:28,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:01:28,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:01:28,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:01:28,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:01:28,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c4b0cbf2-9fba-4187-a8c4-7aa8a149c507
[2017-06-14 16:01:28,784: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c4b0cbf2-9fba-4187-a8c4-7aa8a149c507
[2017-06-14 16:01:28,832: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c4b0cbf2-9fba-4187-a8c4-7aa8a149c507
[2017-06-14 16:01:28,832: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c4b0cbf2-9fba-4187-a8c4-7aa8a149c507
[2017-06-14 16:01:28,833: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-06-14 16:01:28,864: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2af827b0-26d8-45fd-9f5e-64bd9a4b56e7
[2017-06-14 16:01:28,864: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2af827b0-26d8-45fd-9f5e-64bd9a4b56e7
[2017-06-14 16:01:32,539: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:01:32,539: INFO/job_scheduler]   Job 21 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-14 16:01:32,539: INFO/job_scheduler]   Job 22 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 16:01:32,539: INFO/job_scheduler]   Job 23 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 16:01:32,539: INFO/job_scheduler]   Job 24 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 16:01:32,540: INFO/job_scheduler]   Job 25 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 16:01:32,540: INFO/job_scheduler]   Job 28 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 16:01:32,540: INFO/job_scheduler]   Job 31 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:01:32,540: INFO/job_scheduler] run_next: 0 runnable jobs of (24 pending, 4 tasked)
[2017-06-14 16:01:32,624: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:32,624: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:32,963: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:32,963: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:32,964: INFO/job_scheduler] Job 21 finished 4 steps successfully
[2017-06-14 16:01:33,027: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 140623806294736
[2017-06-14 16:01:33,032: INFO/job] Job 21: StateChangeJob complete, setting state packages_installed on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 16:01:33,032: INFO/job] Job 21: not_deleted=True
[2017-06-14 16:01:33,032: INFO/job_scheduler] Job 21: complete_job: Updating cache
[2017-06-14 16:01:33,032: INFO/job_scheduler] Job 21: locked_item 140623806293456 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-14 16:01:33,035: INFO/job_scheduler] Job 21 completing (errored=False, cancelled=False)
[2017-06-14 16:01:33,077: INFO/job_scheduler] run_next: 4 runnable jobs of (24 pending, 3 tasked)
[2017-06-14 16:01:33,168: INFO/job_scheduler] Job 22: RunJobThread.run
[2017-06-14 16:01:33,169: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-06-14 16:01:33,203: INFO/job_scheduler] Job 23: RunJobThread.run
[2017-06-14 16:01:33,204: INFO/job_scheduler] Job 24: open DB connection during completion
[2017-06-14 16:01:33,206: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-06-14 16:01:33,214: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cf11eaf7-a8bd-4da0-8332-6d7a1de3a916
[2017-06-14 16:01:33,214: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cf11eaf7-a8bd-4da0-8332-6d7a1de3a916
[2017-06-14 16:01:33,237: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f71f36a9-f265-4f89-8234-50684bcb327f
[2017-06-14 16:01:33,237: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f71f36a9-f265-4f89-8234-50684bcb327f
[2017-06-14 16:01:33,241: INFO/job_scheduler] Job 25: RunJobThread.run
[2017-06-14 16:01:33,242: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-06-14 16:01:33,277: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6fb08f03-5a1c-4061-9431-5b1596c9bc95
[2017-06-14 16:01:33,277: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6fb08f03-5a1c-4061-9431-5b1596c9bc95
[2017-06-14 16:01:33,327: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 140624277298704
[2017-06-14 16:01:33,335: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-06-14 16:01:33,354: INFO/job] Job 24: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-14 16:01:33,355: INFO/job] Job 24: not_deleted=True
[2017-06-14 16:01:33,355: INFO/job_scheduler] Job 24: complete_job: Updating cache
[2017-06-14 16:01:33,355: INFO/job_scheduler] Job 24: locked_item 140624277298704 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-14 16:01:33,361: INFO/job_scheduler] Job 24 completing (errored=False, cancelled=False)
[2017-06-14 16:01:33,535: INFO/job_scheduler] run_next: 1 runnable jobs of (20 pending, 6 tasked)
[2017-06-14 16:01:33,574: INFO/job_scheduler] Job 26: RunJobThread.run
[2017-06-14 16:01:33,575: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com load_lnet {}
[2017-06-14 16:01:33,619: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4a57719c-5dea-4504-a715-cb73abca087e
[2017-06-14 16:01:33,619: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4a57719c-5dea-4504-a715-cb73abca087e
[2017-06-14 16:01:33,716: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:33,716: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:33,716: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:33,716: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:33,716: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4a57719c-5dea-4504-a715-cb73abca087e
[2017-06-14 16:01:33,716: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4a57719c-5dea-4504-a715-cb73abca087e
[2017-06-14 16:01:33,740: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4a57719c-5dea-4504-a715-cb73abca087e
[2017-06-14 16:01:33,740: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4a57719c-5dea-4504-a715-cb73abca087e
[2017-06-14 16:01:33,741: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-14 16:01:33,772: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3a4025e9-2e92-44bf-b648-0e4081fe17bb
[2017-06-14 16:01:33,772: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3a4025e9-2e92-44bf-b648-0e4081fe17bb
[2017-06-14 16:01:34,893: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:34,893: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:34,893: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:34,893: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:34,893: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3a4025e9-2e92-44bf-b648-0e4081fe17bb
[2017-06-14 16:01:34,893: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3a4025e9-2e92-44bf-b648-0e4081fe17bb
[2017-06-14 16:01:34,941: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3a4025e9-2e92-44bf-b648-0e4081fe17bb
[2017-06-14 16:01:34,941: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3a4025e9-2e92-44bf-b648-0e4081fe17bb
[2017-06-14 16:01:34,989: INFO/job_scheduler] Job 26 finished 2 steps successfully
[2017-06-14 16:01:35,020: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 140624277298704
[2017-06-14 16:01:35,030: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-06-14 16:01:35,052: INFO/job] Job 26: StateChangeJob complete, setting state lnet_down on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-14 16:01:35,053: INFO/job] Job 26: not_deleted=True
[2017-06-14 16:01:35,053: INFO/job_scheduler] Job 26: complete_job: Updating cache
[2017-06-14 16:01:35,053: INFO/job_scheduler] Job 26: locked_item 140624277298704 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-14 16:01:35,058: INFO/job_scheduler] Job 26 completing (errored=False, cancelled=False)
[2017-06-14 16:01:35,086: INFO/job_scheduler] run_next: 1 runnable jobs of (19 pending, 6 tasked)
[2017-06-14 16:01:35,120: INFO/job_scheduler] Job 29: RunJobThread.run
[2017-06-14 16:01:35,120: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_lnet {}
[2017-06-14 16:01:35,173: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4726ed52-19ba-46ec-a906-0efb5dbb33da
[2017-06-14 16:01:35,173: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4726ed52-19ba-46ec-a906-0efb5dbb33da
[2017-06-14 16:01:35,967: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:01:35,967: INFO/job_scheduler]   Job 22 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 16:01:35,968: INFO/job_scheduler]   Job 23 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 16:01:35,968: INFO/job_scheduler]   Job 25 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 16:01:35,968: INFO/job_scheduler]   Job 28 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 16:01:35,968: INFO/job_scheduler]   Job 31 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:01:35,968: INFO/job_scheduler] run_next: 0 runnable jobs of (18 pending, 7 tasked)
[2017-06-14 16:01:37,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:37,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:37,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:37,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:37,468: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4726ed52-19ba-46ec-a906-0efb5dbb33da
[2017-06-14 16:01:37,468: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4726ed52-19ba-46ec-a906-0efb5dbb33da
[2017-06-14 16:01:37,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4726ed52-19ba-46ec-a906-0efb5dbb33da
[2017-06-14 16:01:37,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4726ed52-19ba-46ec-a906-0efb5dbb33da
[2017-06-14 16:01:37,495: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-14 16:01:37,535: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 723d85ec-f091-4c39-97d5-a2708fda07b3
[2017-06-14 16:01:37,535: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 723d85ec-f091-4c39-97d5-a2708fda07b3
[2017-06-14 16:01:38,567: INFO/job_scheduler] run_next: 0 runnable jobs of (18 pending, 7 tasked)
[2017-06-14 16:01:38,582: INFO/job_scheduler] run_next: 0 runnable jobs of (18 pending, 7 tasked)
[2017-06-14 16:01:38,616: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:38,616: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:38,616: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:38,616: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:38,616: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 723d85ec-f091-4c39-97d5-a2708fda07b3
[2017-06-14 16:01:38,616: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 723d85ec-f091-4c39-97d5-a2708fda07b3
[2017-06-14 16:01:38,650: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 723d85ec-f091-4c39-97d5-a2708fda07b3
[2017-06-14 16:01:38,650: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 723d85ec-f091-4c39-97d5-a2708fda07b3
[2017-06-14 16:01:38,772: INFO/job_scheduler] Job 29 finished 2 steps successfully
[2017-06-14 16:01:38,799: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 140624277298704
[2017-06-14 16:01:38,804: INFO/job] Job 29: StateChangeJob complete, setting state lnet_up on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-14 16:01:38,804: INFO/job] Job 29: not_deleted=True
[2017-06-14 16:01:38,804: INFO/job_scheduler] Job 29: complete_job: Updating cache
[2017-06-14 16:01:38,804: INFO/job_scheduler] Job 29: locked_item 140624277298704 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-14 16:01:38,808: INFO/job_scheduler] Job 29 completing (errored=False, cancelled=False)
[2017-06-14 16:01:38,851: INFO/job_scheduler] run_next: 0 runnable jobs of (18 pending, 6 tasked)
[2017-06-14 16:01:39,020: INFO/job_scheduler] run_next: 0 runnable jobs of (18 pending, 6 tasked)
[2017-06-14 16:01:39,058: INFO/job_scheduler] run_next: 0 runnable jobs of (18 pending, 6 tasked)
[2017-06-14 16:01:39,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:39,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:39,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:39,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:39,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f71f36a9-f265-4f89-8234-50684bcb327f
[2017-06-14 16:01:39,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f71f36a9-f265-4f89-8234-50684bcb327f
[2017-06-14 16:01:39,418: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f71f36a9-f265-4f89-8234-50684bcb327f
[2017-06-14 16:01:39,418: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f71f36a9-f265-4f89-8234-50684bcb327f
[2017-06-14 16:01:39,419: INFO/job_scheduler] Job 23 finished 1 steps successfully
[2017-06-14 16:01:39,511: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 140623806318928
[2017-06-14 16:01:39,531: INFO/job] Job 23: StateChangeJob complete, setting state configured on lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration
[2017-06-14 16:01:39,531: INFO/job] Job 23: not_deleted=True
[2017-06-14 16:01:39,531: INFO/job_scheduler] Job 23: complete_job: Updating cache
[2017-06-14 16:01:39,531: INFO/job_scheduler] Job 23: locked_item 140623806318928 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-06-14 16:01:39,539: INFO/job_scheduler] Job 23 completing (errored=False, cancelled=False)
[2017-06-14 16:01:39,581: INFO/job_scheduler] run_next: 0 runnable jobs of (18 pending, 5 tasked)
[2017-06-14 16:01:40,394: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:40,394: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:40,394: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:40,394: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:40,394: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6fb08f03-5a1c-4061-9431-5b1596c9bc95
[2017-06-14 16:01:40,394: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6fb08f03-5a1c-4061-9431-5b1596c9bc95
[2017-06-14 16:01:40,404: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6fb08f03-5a1c-4061-9431-5b1596c9bc95
[2017-06-14 16:01:40,404: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6fb08f03-5a1c-4061-9431-5b1596c9bc95
[2017-06-14 16:01:40,404: INFO/job_scheduler] Job 25 finished 1 steps successfully
[2017-06-14 16:01:40,558: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 140623806318288
[2017-06-14 16:01:40,561: INFO/job] Job 25: StateChangeJob complete, setting state configured on lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-06-14 16:01:40,561: INFO/job] Job 25: not_deleted=True
[2017-06-14 16:01:40,562: INFO/job_scheduler] Job 25: complete_job: Updating cache
[2017-06-14 16:01:40,562: INFO/job_scheduler] Job 25: locked_item 140623806318288 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-06-14 16:01:40,564: INFO/job_scheduler] Job 25 completing (errored=False, cancelled=False)
[2017-06-14 16:01:40,603: INFO/job_scheduler] run_next: 0 runnable jobs of (18 pending, 4 tasked)
[2017-06-14 16:01:51,704: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:51,704: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:51,704: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:51,704: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:51,704: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cf11eaf7-a8bd-4da0-8332-6d7a1de3a916
[2017-06-14 16:01:51,704: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cf11eaf7-a8bd-4da0-8332-6d7a1de3a916
[2017-06-14 16:01:51,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cf11eaf7-a8bd-4da0-8332-6d7a1de3a916
[2017-06-14 16:01:51,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cf11eaf7-a8bd-4da0-8332-6d7a1de3a916
[2017-06-14 16:01:51,748: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.224', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-06-14 16:01:51,761: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f63c5e4b-2294-4009-9e47-3a75bfef9f7d
[2017-06-14 16:01:51,761: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f63c5e4b-2294-4009-9e47-3a75bfef9f7d
[2017-06-14 16:01:51,826: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:51,826: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:51,826: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:51,826: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:01:51,827: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f63c5e4b-2294-4009-9e47-3a75bfef9f7d
[2017-06-14 16:01:51,827: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f63c5e4b-2294-4009-9e47-3a75bfef9f7d
[2017-06-14 16:01:51,827: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f63c5e4b-2294-4009-9e47-3a75bfef9f7d
[2017-06-14 16:01:51,827: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f63c5e4b-2294-4009-9e47-3a75bfef9f7d
[2017-06-14 16:01:51,843: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 65087, 'pcs_password': 'plbbBczr0hYeY3r9nfow'}
[2017-06-14 16:01:51,855: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1e2fa764-4352-4b2d-81cb-6911b150fb85
[2017-06-14 16:01:51,855: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1e2fa764-4352-4b2d-81cb-6911b150fb85
[2017-06-14 16:01:52,779: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:01:52,779: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:01:52,817: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:01:52,817: INFO/job_scheduler]   Job 54 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-14 16:01:52,817: INFO/job_scheduler]   Job 55 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 16:01:52,818: INFO/job_scheduler]   Job 56 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 16:01:52,818: INFO/job_scheduler]   Job 57 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 16:01:52,818: INFO/job_scheduler]   Job 58 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 16:01:52,818: INFO/job_scheduler]   Job 62 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 16:01:52,818: INFO/job_scheduler]   Job 64 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:01:52,818: INFO/job_scheduler] run_next: 0 runnable jobs of (18 pending, 4 tasked)
[2017-06-14 16:01:53,649: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm8.lotus.hpdd.lab.intel.com new 201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:01:53,649: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm8.lotus.hpdd.lab.intel.com new 201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:01:53,650: INFO/job_scheduler] Job 54 finished 4 steps successfully
[2017-06-14 16:01:53,672: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 140623806425168
[2017-06-14 16:01:53,676: INFO/job] Job 54: StateChangeJob complete, setting state packages_installed on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 16:01:53,676: INFO/job] Job 54: not_deleted=True
[2017-06-14 16:01:53,676: INFO/job_scheduler] Job 54: complete_job: Updating cache
[2017-06-14 16:01:53,676: INFO/job_scheduler] Job 54: locked_item 140623806319376 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-14 16:01:53,679: INFO/job_scheduler] Job 54 completing (errored=False, cancelled=False)
[2017-06-14 16:01:53,731: INFO/job_scheduler] run_next: 4 runnable jobs of (18 pending, 3 tasked)
[2017-06-14 16:01:53,818: INFO/job_scheduler] Job 55: RunJobThread.run
[2017-06-14 16:01:53,818: INFO/job_scheduler] Job 56: open DB connection during completion
[2017-06-14 16:01:53,824: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-06-14 16:01:53,855: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dd26abe4-b4c4-4451-a8a2-065a8d7a3e58
[2017-06-14 16:01:53,855: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dd26abe4-b4c4-4451-a8a2-065a8d7a3e58
[2017-06-14 16:01:53,863: INFO/job_scheduler] Job 57: RunJobThread.run
[2017-06-14 16:01:53,864: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-06-14 16:01:53,905: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 99060d80-ef03-41eb-a6af-bee9d5518455
[2017-06-14 16:01:53,905: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 99060d80-ef03-41eb-a6af-bee9d5518455
[2017-06-14 16:01:53,916: INFO/job_scheduler] Job 58: RunJobThread.run
[2017-06-14 16:01:53,916: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-06-14 16:01:53,974: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0f84b603-5842-4ac8-aa42-87b035e9451f
[2017-06-14 16:01:53,974: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0f84b603-5842-4ac8-aa42-87b035e9451f
[2017-06-14 16:01:54,022: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 140623806413840
[2017-06-14 16:01:54,030: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-06-14 16:01:54,050: INFO/job] Job 56: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-14 16:01:54,050: INFO/job] Job 56: not_deleted=True
[2017-06-14 16:01:54,050: INFO/job_scheduler] Job 56: complete_job: Updating cache
[2017-06-14 16:01:54,050: INFO/job_scheduler] Job 56: locked_item 140623806413840 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-14 16:01:54,060: INFO/job_scheduler] Job 56 completing (errored=False, cancelled=False)
[2017-06-14 16:01:54,113: INFO/job_scheduler] run_next: 1 runnable jobs of (14 pending, 6 tasked)
[2017-06-14 16:01:54,154: INFO/job_scheduler] Job 60: RunJobThread.run
[2017-06-14 16:01:54,155: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com load_lnet {}
[2017-06-14 16:01:54,196: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5e836d2b-1b0e-4251-97e4-93f7aed134c8
[2017-06-14 16:01:54,196: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5e836d2b-1b0e-4251-97e4-93f7aed134c8
[2017-06-14 16:01:54,388: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:01:54,388: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:01:54,388: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:01:54,388: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:01:54,388: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5e836d2b-1b0e-4251-97e4-93f7aed134c8
[2017-06-14 16:01:54,388: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5e836d2b-1b0e-4251-97e4-93f7aed134c8
[2017-06-14 16:01:54,425: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5e836d2b-1b0e-4251-97e4-93f7aed134c8
[2017-06-14 16:01:54,425: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5e836d2b-1b0e-4251-97e4-93f7aed134c8
[2017-06-14 16:01:54,425: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-14 16:01:54,454: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0e6468b2-c07c-45ef-aa21-017f57b40f62
[2017-06-14 16:01:54,454: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0e6468b2-c07c-45ef-aa21-017f57b40f62
[2017-06-14 16:01:54,561: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:01:54,561: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:01:54,561: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:01:54,561: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:01:54,562: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0e6468b2-c07c-45ef-aa21-017f57b40f62
[2017-06-14 16:01:54,562: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0e6468b2-c07c-45ef-aa21-017f57b40f62
[2017-06-14 16:01:54,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0e6468b2-c07c-45ef-aa21-017f57b40f62
[2017-06-14 16:01:54,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0e6468b2-c07c-45ef-aa21-017f57b40f62
[2017-06-14 16:01:55,464: INFO/job_scheduler] Job 60 finished 2 steps successfully
[2017-06-14 16:01:55,496: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 140623806413840
[2017-06-14 16:01:55,502: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-06-14 16:01:55,511: INFO/job] Job 60: StateChangeJob complete, setting state lnet_down on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-14 16:01:55,511: INFO/job] Job 60: not_deleted=True
[2017-06-14 16:01:55,511: INFO/job_scheduler] Job 60: complete_job: Updating cache
[2017-06-14 16:01:55,512: INFO/job_scheduler] Job 60: locked_item 140623806413840 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-14 16:01:55,528: INFO/job_scheduler] Job 60 completing (errored=False, cancelled=False)
[2017-06-14 16:01:55,616: INFO/job_scheduler] run_next: 1 runnable jobs of (13 pending, 6 tasked)
[2017-06-14 16:01:55,645: INFO/job_scheduler] Job 61: RunJobThread.run
[2017-06-14 16:01:55,645: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_lnet {}
[2017-06-14 16:01:55,676: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 259eaebe-490d-4a2a-9d36-39ff020e9fe1
[2017-06-14 16:01:55,676: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 259eaebe-490d-4a2a-9d36-39ff020e9fe1
[2017-06-14 16:01:57,113: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:01:57,114: INFO/job_scheduler]   Job 55 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 16:01:57,114: INFO/job_scheduler]   Job 57 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 16:01:57,114: INFO/job_scheduler]   Job 58 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 16:01:57,114: INFO/job_scheduler]   Job 62 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 16:01:57,114: INFO/job_scheduler]   Job 64 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:01:57,114: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 7 tasked)
[2017-06-14 16:01:57,805: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:01:57,805: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:01:57,805: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:01:57,805: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:01:57,805: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 259eaebe-490d-4a2a-9d36-39ff020e9fe1
[2017-06-14 16:01:57,805: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 259eaebe-490d-4a2a-9d36-39ff020e9fe1
[2017-06-14 16:01:57,850: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 259eaebe-490d-4a2a-9d36-39ff020e9fe1
[2017-06-14 16:01:57,850: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 259eaebe-490d-4a2a-9d36-39ff020e9fe1
[2017-06-14 16:01:57,851: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-14 16:01:57,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 09616162-e6ef-407a-95b6-b1996b13c042
[2017-06-14 16:01:57,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 09616162-e6ef-407a-95b6-b1996b13c042
[2017-06-14 16:01:58,607: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 7 tasked)
[2017-06-14 16:01:58,624: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 7 tasked)
[2017-06-14 16:01:58,926: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:01:58,926: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:01:58,926: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:01:58,926: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:01:58,926: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 09616162-e6ef-407a-95b6-b1996b13c042
[2017-06-14 16:01:58,926: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 09616162-e6ef-407a-95b6-b1996b13c042
[2017-06-14 16:01:58,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 09616162-e6ef-407a-95b6-b1996b13c042
[2017-06-14 16:01:58,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 09616162-e6ef-407a-95b6-b1996b13c042
[2017-06-14 16:01:59,149: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:01:59,150: INFO/job_scheduler]   Job 50 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:01:59,150: INFO/job_scheduler]   Job 52 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:01:59,150: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 7 tasked)
[2017-06-14 16:01:59,193: INFO/job_scheduler] Job 61 finished 2 steps successfully
[2017-06-14 16:01:59,212: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 140623806413840
[2017-06-14 16:01:59,217: INFO/job] Job 61: StateChangeJob complete, setting state lnet_up on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-14 16:01:59,217: INFO/job] Job 61: not_deleted=True
[2017-06-14 16:01:59,217: INFO/job_scheduler] Job 61: complete_job: Updating cache
[2017-06-14 16:01:59,217: INFO/job_scheduler] Job 61: locked_item 140623806413840 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-14 16:01:59,220: INFO/job_scheduler] Job 61 completing (errored=False, cancelled=False)
[2017-06-14 16:01:59,257: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 6 tasked)
[2017-06-14 16:02:00,144: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:02:00,144: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:02:00,144: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:02:00,144: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:02:00,145: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dd26abe4-b4c4-4451-a8a2-065a8d7a3e58
[2017-06-14 16:02:00,145: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dd26abe4-b4c4-4451-a8a2-065a8d7a3e58
[2017-06-14 16:02:00,154: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:02:00,154: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:02:00,154: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:02:00,154: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:02:00,154: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0f84b603-5842-4ac8-aa42-87b035e9451f
[2017-06-14 16:02:00,154: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0f84b603-5842-4ac8-aa42-87b035e9451f
[2017-06-14 16:02:00,186: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dd26abe4-b4c4-4451-a8a2-065a8d7a3e58
[2017-06-14 16:02:00,186: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dd26abe4-b4c4-4451-a8a2-065a8d7a3e58
[2017-06-14 16:02:00,187: INFO/job_scheduler] Job 55 finished 1 steps successfully
[2017-06-14 16:02:00,204: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0f84b603-5842-4ac8-aa42-87b035e9451f
[2017-06-14 16:02:00,204: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0f84b603-5842-4ac8-aa42-87b035e9451f
[2017-06-14 16:02:00,205: INFO/job_scheduler] Job 58 finished 1 steps successfully
[2017-06-14 16:02:00,253: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 140623806369232
[2017-06-14 16:02:00,257: INFO/job] Job 55: StateChangeJob complete, setting state configured on lotus-32vm8.lotus.hpdd.lab.intel.com NTP configuration
[2017-06-14 16:02:00,257: INFO/job] Job 55: not_deleted=True
[2017-06-14 16:02:00,257: INFO/job_scheduler] Job 55: complete_job: Updating cache
[2017-06-14 16:02:00,257: INFO/job_scheduler] Job 55: locked_item 140623806369232 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-06-14 16:02:00,263: INFO/job_scheduler] Job 55 completing (errored=False, cancelled=False)
[2017-06-14 16:02:00,300: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 5 tasked)
[2017-06-14 16:02:00,383: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 140623806367760
[2017-06-14 16:02:00,389: INFO/job] Job 58: StateChangeJob complete, setting state configured on lotus-32vm8.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-06-14 16:02:00,390: INFO/job] Job 58: not_deleted=True
[2017-06-14 16:02:00,390: INFO/job_scheduler] Job 58: complete_job: Updating cache
[2017-06-14 16:02:00,390: INFO/job_scheduler] Job 58: locked_item 140623806367760 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-06-14 16:02:00,392: INFO/job_scheduler] Job 58 completing (errored=False, cancelled=False)
[2017-06-14 16:02:00,427: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-06-14 16:02:04,988: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:02:04,988: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:02:04,989: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:02:04,989: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:02:04,989: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1e2fa764-4352-4b2d-81cb-6911b150fb85
[2017-06-14 16:02:04,989: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1e2fa764-4352-4b2d-81cb-6911b150fb85
[2017-06-14 16:02:05,037: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1e2fa764-4352-4b2d-81cb-6911b150fb85
[2017-06-14 16:02:05,037: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1e2fa764-4352-4b2d-81cb-6911b150fb85
[2017-06-14 16:02:05,038: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': False, 'mcast_port': 65087, 'new_node_fqdn': u'lotus-32vm5.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'plbbBczr0hYeY3r9nfow'}
[2017-06-14 16:02:05,093: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4a94dc12-7209-4f17-b7f8-22fb9409cd20
[2017-06-14 16:02:05,093: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4a94dc12-7209-4f17-b7f8-22fb9409cd20
[2017-06-14 16:02:09,088: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:02:09,089: INFO/job_scheduler]   Job 50 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:09,089: INFO/job_scheduler]   Job 52 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:09,089: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-06-14 16:02:09,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:02:09,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:02:09,857: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:02:09,857: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:02:09,857: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 99060d80-ef03-41eb-a6af-bee9d5518455
[2017-06-14 16:02:09,857: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 99060d80-ef03-41eb-a6af-bee9d5518455
[2017-06-14 16:02:09,902: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 99060d80-ef03-41eb-a6af-bee9d5518455
[2017-06-14 16:02:09,902: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 99060d80-ef03-41eb-a6af-bee9d5518455
[2017-06-14 16:02:09,903: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.227', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-06-14 16:02:09,947: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 294783c8-7e15-4fc3-b373-a29362ab73ea
[2017-06-14 16:02:09,947: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 294783c8-7e15-4fc3-b373-a29362ab73ea
[2017-06-14 16:02:10,031: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:02:10,031: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:02:10,031: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:02:10,031: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:02:10,031: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 294783c8-7e15-4fc3-b373-a29362ab73ea
[2017-06-14 16:02:10,031: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 294783c8-7e15-4fc3-b373-a29362ab73ea
[2017-06-14 16:02:10,066: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 294783c8-7e15-4fc3-b373-a29362ab73ea
[2017-06-14 16:02:10,066: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 294783c8-7e15-4fc3-b373-a29362ab73ea
[2017-06-14 16:02:10,095: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 49429, 'pcs_password': 'QwkPhKcOapUJigOInOdw'}
[2017-06-14 16:02:10,122: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8ed6eec4-c660-4fe5-93c8-85196946f29c
[2017-06-14 16:02:10,122: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8ed6eec4-c660-4fe5-93c8-85196946f29c
[2017-06-14 16:02:11,203: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:02:11,203: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:02:11,203: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:02:11,203: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:02:11,203: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4a94dc12-7209-4f17-b7f8-22fb9409cd20
[2017-06-14 16:02:11,203: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4a94dc12-7209-4f17-b7f8-22fb9409cd20
[2017-06-14 16:02:11,208: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4a94dc12-7209-4f17-b7f8-22fb9409cd20
[2017-06-14 16:02:11,208: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4a94dc12-7209-4f17-b7f8-22fb9409cd20
[2017-06-14 16:02:11,209: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-06-14 16:02:11.209111+00:00:
[2017-06-14 16:02:11,209: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-06-14 16:02:11.209111+00:00:
[2017-06-14 16:02:11,209: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->65087
[2017-06-14 16:02:11,209: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->65087
[2017-06-14 16:02:11,211: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-06-14 16:02:11,211: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-06-14 16:02:11,229: INFO/job_scheduler] Job 22 finished 1 steps successfully
[2017-06-14 16:02:11,232: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:02:11,232: INFO/job_scheduler]   Job 22 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-06-14 16:02:11,232: INFO/job_scheduler]   Job 27 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-06-14 16:02:11,232: INFO/job_scheduler]   Job 28 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:11,232: INFO/job_scheduler]   Job 30 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:11,233: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-06-14 16:02:11,271: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 140623806240784
[2017-06-14 16:02:11,278: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-06-14 16:02:11,282: INFO/job] Job 22: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:11,282: INFO/job] Job 22: not_deleted=True
[2017-06-14 16:02:11,282: INFO/job_scheduler] Job 22: complete_job: Updating cache
[2017-06-14 16:02:11,282: INFO/job_scheduler] Job 22: locked_item 140623806240784 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-14 16:02:11,298: INFO/job_scheduler] Job 22 completing (errored=False, cancelled=False)
[2017-06-14 16:02:11,344: INFO/job_scheduler] run_next: 1 runnable jobs of (12 pending, 3 tasked)
[2017-06-14 16:02:11,372: INFO/job_scheduler] Job 27: RunJobThread.run
[2017-06-14 16:02:11,373: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-06-14 16:02:11,398: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d3c21f6c-b1fb-4f02-8d4e-7239e7105412
[2017-06-14 16:02:11,398: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d3c21f6c-b1fb-4f02-8d4e-7239e7105412
[2017-06-14 16:02:16,684: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 4 tasked)
[2017-06-14 16:02:16,700: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 4 tasked)
[2017-06-14 16:02:18,696: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:02:18,697: INFO/job_scheduler]   Job 27 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-06-14 16:02:18,697: INFO/job_scheduler]   Job 28 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:18,697: INFO/job_scheduler]   Job 30 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:18,697: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 4 tasked)
[2017-06-14 16:02:18,745: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:02:18,745: INFO/job_scheduler]   Job 28 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 16:02:18,745: INFO/job_scheduler]   Job 31 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:02:18,745: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 4 tasked)
[2017-06-14 16:02:18,826: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:02:18,826: INFO/job_scheduler]   Job 39 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:18,826: INFO/job_scheduler]   Job 41 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:18,827: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 4 tasked)
[2017-06-14 16:02:18,855: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:02:18,855: INFO/job_scheduler]   Job 39 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-14 16:02:18,856: INFO/job_scheduler]   Job 42 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:02:18,856: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 4 tasked)
[2017-06-14 16:02:19,167: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:02:19,167: INFO/job_scheduler]   Job 50 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:19,167: INFO/job_scheduler]   Job 52 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:19,167: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 4 tasked)
[2017-06-14 16:02:19,500: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:02:19,500: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:02:19,500: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:02:19,500: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:02:19,500: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d3c21f6c-b1fb-4f02-8d4e-7239e7105412
[2017-06-14 16:02:19,500: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d3c21f6c-b1fb-4f02-8d4e-7239e7105412
[2017-06-14 16:02:19,522: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d3c21f6c-b1fb-4f02-8d4e-7239e7105412
[2017-06-14 16:02:19,522: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d3c21f6c-b1fb-4f02-8d4e-7239e7105412
[2017-06-14 16:02:19,522: INFO/job_scheduler] Job 27 finished 1 steps successfully
[2017-06-14 16:02:19,600: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 140623806240784
[2017-06-14 16:02:19,616: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 30
[2017-06-14 16:02:19,623: INFO/job] Job 27: StateChangeJob complete, setting state started on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:19,624: INFO/job] Job 27: not_deleted=True
[2017-06-14 16:02:19,624: INFO/job_scheduler] Job 27: complete_job: Updating cache
[2017-06-14 16:02:19,624: INFO/job_scheduler] Job 27: locked_item 140623806240784 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-14 16:02:19,638: INFO/job_scheduler] Job 27 completing (errored=False, cancelled=False)
[2017-06-14 16:02:19,674: INFO/job_scheduler] run_next: 1 runnable jobs of (11 pending, 3 tasked)
[2017-06-14 16:02:19,687: INFO/job_scheduler] Job 28: RunJobThread.run
[2017-06-14 16:02:19,688: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-06-14 16:02:19,704: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e2a39d34-9216-4963-a20d-214cea8a84c9
[2017-06-14 16:02:19,704: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e2a39d34-9216-4963-a20d-214cea8a84c9
[2017-06-14 16:02:23,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:02:23,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:02:23,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:02:23,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:02:23,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8ed6eec4-c660-4fe5-93c8-85196946f29c
[2017-06-14 16:02:23,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8ed6eec4-c660-4fe5-93c8-85196946f29c
[2017-06-14 16:02:23,348: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8ed6eec4-c660-4fe5-93c8-85196946f29c
[2017-06-14 16:02:23,348: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8ed6eec4-c660-4fe5-93c8-85196946f29c
[2017-06-14 16:02:23,349: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': False, 'mcast_port': 49429, 'new_node_fqdn': u'lotus-32vm8.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'QwkPhKcOapUJigOInOdw'}
[2017-06-14 16:02:23,364: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9e2fc350-ddd3-475b-9aee-f25f9a530615
[2017-06-14 16:02:23,364: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9e2fc350-ddd3-475b-9aee-f25f9a530615
[2017-06-14 16:02:26,717: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 4 tasked)
[2017-06-14 16:02:26,738: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 4 tasked)
[2017-06-14 16:02:26,771: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:02:26,771: INFO/job_scheduler]   Job 28 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:26,771: INFO/job_scheduler]   Job 30 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:26,772: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 4 tasked)
[2017-06-14 16:02:26,776: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:02:26,776: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:02:26,776: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:02:26,776: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:02:26,776: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e2a39d34-9216-4963-a20d-214cea8a84c9
[2017-06-14 16:02:26,776: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e2a39d34-9216-4963-a20d-214cea8a84c9
[2017-06-14 16:02:26,788: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:02:26,788: INFO/job_scheduler]   Job 28 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 16:02:26,789: INFO/job_scheduler]   Job 31 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:02:26,789: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 4 tasked)
[2017-06-14 16:02:26,807: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:02:26,807: INFO/job_scheduler]   Job 39 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:26,807: INFO/job_scheduler]   Job 41 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:26,808: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 4 tasked)
[2017-06-14 16:02:26,828: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e2a39d34-9216-4963-a20d-214cea8a84c9
[2017-06-14 16:02:26,828: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e2a39d34-9216-4963-a20d-214cea8a84c9
[2017-06-14 16:02:26,829: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-06-14 16:02:26,840: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:02:26,842: INFO/job_scheduler]   Job 39 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-14 16:02:26,842: INFO/job_scheduler]   Job 42 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:02:26,842: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 4 tasked)
[2017-06-14 16:02:26,849: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b99aa6ec-a40a-4156-881b-ee15729bed65
[2017-06-14 16:02:26,849: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b99aa6ec-a40a-4156-881b-ee15729bed65
[2017-06-14 16:02:28,679: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:02:28,680: INFO/job_scheduler]   Job 28 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:28,680: INFO/job_scheduler]   Job 30 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:28,680: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 4 tasked)
[2017-06-14 16:02:28,696: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:02:28,696: INFO/job_scheduler]   Job 28 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 16:02:28,696: INFO/job_scheduler]   Job 31 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:02:28,696: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 4 tasked)
[2017-06-14 16:02:28,714: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:02:28,714: INFO/job_scheduler]   Job 39 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:28,714: INFO/job_scheduler]   Job 41 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:28,715: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 4 tasked)
[2017-06-14 16:02:28,729: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:02:28,729: INFO/job_scheduler]   Job 39 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-14 16:02:28,729: INFO/job_scheduler]   Job 42 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:02:28,729: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 4 tasked)
[2017-06-14 16:02:29,205: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:02:29,206: INFO/job_scheduler]   Job 50 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:29,206: INFO/job_scheduler]   Job 52 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:29,206: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 4 tasked)
[2017-06-14 16:02:29,473: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:02:29,473: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:02:29,474: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:02:29,474: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:02:29,474: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9e2fc350-ddd3-475b-9aee-f25f9a530615
[2017-06-14 16:02:29,474: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9e2fc350-ddd3-475b-9aee-f25f9a530615
[2017-06-14 16:02:29,488: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9e2fc350-ddd3-475b-9aee-f25f9a530615
[2017-06-14 16:02:29,488: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9e2fc350-ddd3-475b-9aee-f25f9a530615
[2017-06-14 16:02:29,488: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-06-14 16:02:29.488493+00:00:
[2017-06-14 16:02:29,488: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-06-14 16:02:29.488493+00:00:
[2017-06-14 16:02:29,488: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->49429
[2017-06-14 16:02:29,488: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->49429
[2017-06-14 16:02:29,490: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-06-14 16:02:29,490: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-06-14 16:02:29,513: INFO/job_scheduler] Job 57 finished 1 steps successfully
[2017-06-14 16:02:29,514: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:02:29,514: INFO/job_scheduler]   Job 57 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-06-14 16:02:29,514: INFO/job_scheduler]   Job 59 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-06-14 16:02:29,514: INFO/job_scheduler]   Job 62 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:29,515: INFO/job_scheduler]   Job 63 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:29,515: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 4 tasked)
[2017-06-14 16:02:29,550: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 140623806426320
[2017-06-14 16:02:29,557: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-06-14 16:02:29,573: INFO/job] Job 57: StateChangeJob complete, setting state stopped on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:29,573: INFO/job] Job 57: not_deleted=True
[2017-06-14 16:02:29,574: INFO/job_scheduler] Job 57: complete_job: Updating cache
[2017-06-14 16:02:29,574: INFO/job_scheduler] Job 57: locked_item 140623806426320 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-14 16:02:29,580: INFO/job_scheduler] Job 57 completing (errored=False, cancelled=False)
[2017-06-14 16:02:29,645: INFO/job_scheduler] run_next: 1 runnable jobs of (10 pending, 3 tasked)
[2017-06-14 16:02:29,680: INFO/job_scheduler] Job 59: RunJobThread.run
[2017-06-14 16:02:29,681: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-06-14 16:02:29,724: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 505f4110-01fb-4742-90c4-4655bb5ba153
[2017-06-14 16:02:29,724: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 505f4110-01fb-4742-90c4-4655bb5ba153
[2017-06-14 16:02:30,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:02:30,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:02:30,422: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:02:30,422: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:02:30,422: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c105c65e-47e9-4bc6-abad-aa7776d3c9c3
[2017-06-14 16:02:30,422: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c105c65e-47e9-4bc6-abad-aa7776d3c9c3
[2017-06-14 16:02:30,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c105c65e-47e9-4bc6-abad-aa7776d3c9c3
[2017-06-14 16:02:30,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c105c65e-47e9-4bc6-abad-aa7776d3c9c3
[2017-06-14 16:02:30,475: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-06-14 16:02:30,488: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 554003f9-3b49-4318-8ee6-7c8f744e1bc9
[2017-06-14 16:02:30,488: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 554003f9-3b49-4318-8ee6-7c8f744e1bc9
[2017-06-14 16:02:36,731: INFO/job_scheduler] run_next: 0 runnable jobs of (9 pending, 4 tasked)
[2017-06-14 16:02:36,752: INFO/job_scheduler] run_next: 0 runnable jobs of (9 pending, 4 tasked)
[2017-06-14 16:02:37,274: INFO/job_scheduler] run_next: 0 runnable jobs of (9 pending, 4 tasked)
[2017-06-14 16:02:37,290: INFO/job_scheduler] run_next: 0 runnable jobs of (9 pending, 4 tasked)
[2017-06-14 16:02:37,590: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:02:37,590: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:02:37,590: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:02:37,590: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:02:37,590: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 554003f9-3b49-4318-8ee6-7c8f744e1bc9
[2017-06-14 16:02:37,590: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 554003f9-3b49-4318-8ee6-7c8f744e1bc9
[2017-06-14 16:02:37,606: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 554003f9-3b49-4318-8ee6-7c8f744e1bc9
[2017-06-14 16:02:37,606: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 554003f9-3b49-4318-8ee6-7c8f744e1bc9
[2017-06-14 16:02:37,607: INFO/job_scheduler] Job 50 finished 3 steps successfully
[2017-06-14 16:02:37,654: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 140623202469712
[2017-06-14 16:02:37,665: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-06-14 16:02:37,682: INFO/job] Job 50: StateChangeJob complete, setting state stopped on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-14 16:02:37,682: INFO/job] Job 50: not_deleted=True
[2017-06-14 16:02:37,682: INFO/job_scheduler] Job 50: complete_job: Updating cache
[2017-06-14 16:02:37,682: INFO/job_scheduler] Job 50: locked_item 140623202469712 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-14 16:02:37,688: INFO/job_scheduler] Job 50 completing (errored=False, cancelled=False)
[2017-06-14 16:02:37,740: INFO/job_scheduler] run_next: 1 runnable jobs of (9 pending, 3 tasked)
[2017-06-14 16:02:37,792: INFO/job_scheduler] Job 52: RunJobThread.run
[2017-06-14 16:02:37,793: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-06-14 16:02:37,853: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5857c5c7-fae3-4583-b809-80210311fd1c
[2017-06-14 16:02:37,853: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5857c5c7-fae3-4583-b809-80210311fd1c
[2017-06-14 16:02:37,857: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:02:37,857: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:02:37,857: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:02:37,857: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:02:37,857: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 505f4110-01fb-4742-90c4-4655bb5ba153
[2017-06-14 16:02:37,857: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 505f4110-01fb-4742-90c4-4655bb5ba153
[2017-06-14 16:02:37,907: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 505f4110-01fb-4742-90c4-4655bb5ba153
[2017-06-14 16:02:37,907: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 505f4110-01fb-4742-90c4-4655bb5ba153
[2017-06-14 16:02:37,908: INFO/job_scheduler] Job 59 finished 1 steps successfully
[2017-06-14 16:02:37,980: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 140623806426320
[2017-06-14 16:02:38,033: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm8.lotus.hpdd.lab.intel.com at severity 30
[2017-06-14 16:02:38,073: INFO/job] Job 59: StateChangeJob complete, setting state started on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:38,074: INFO/job] Job 59: not_deleted=True
[2017-06-14 16:02:38,074: INFO/job_scheduler] Job 59: complete_job: Updating cache
[2017-06-14 16:02:38,074: INFO/job_scheduler] Job 59: locked_item 140623806426320 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-14 16:02:38,104: INFO/job_scheduler] Job 59 completing (errored=False, cancelled=False)
[2017-06-14 16:02:38,158: INFO/job_scheduler] run_next: 1 runnable jobs of (8 pending, 3 tasked)
[2017-06-14 16:02:38,230: INFO/job_scheduler] Job 62: RunJobThread.run
[2017-06-14 16:02:38,231: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-06-14 16:02:38,305: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8e4051b2-ad1e-4e0a-9e0f-056ed05a8cf6
[2017-06-14 16:02:38,305: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8e4051b2-ad1e-4e0a-9e0f-056ed05a8cf6
[2017-06-14 16:02:38,822: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:02:38,822: INFO/job_scheduler]   Job 28 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:38,822: INFO/job_scheduler]   Job 30 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:38,822: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-06-14 16:02:38,824: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:02:38,824: INFO/job_scheduler]   Job 28 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 16:02:38,824: INFO/job_scheduler]   Job 31 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:02:38,824: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-06-14 16:02:38,832: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:02:38,832: INFO/job_scheduler]   Job 39 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:38,832: INFO/job_scheduler]   Job 41 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:38,832: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-06-14 16:02:38,860: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:02:38,860: INFO/job_scheduler]   Job 39 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-14 16:02:38,860: INFO/job_scheduler]   Job 42 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:02:38,860: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-06-14 16:02:44,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:02:44,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:02:44,940: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:02:44,940: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:02:44,940: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5857c5c7-fae3-4583-b809-80210311fd1c
[2017-06-14 16:02:44,940: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5857c5c7-fae3-4583-b809-80210311fd1c
[2017-06-14 16:02:44,972: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5857c5c7-fae3-4583-b809-80210311fd1c
[2017-06-14 16:02:44,972: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5857c5c7-fae3-4583-b809-80210311fd1c
[2017-06-14 16:02:44,972: INFO/job_scheduler] Job 52 finished 1 steps successfully
[2017-06-14 16:02:45,023: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 140623202469712
[2017-06-14 16:02:45,049: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 30
[2017-06-14 16:02:45,056: INFO/job] Job 52: StateChangeJob complete, setting state started on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-14 16:02:45,057: INFO/job] Job 52: not_deleted=True
[2017-06-14 16:02:45,057: INFO/job_scheduler] Job 52: complete_job: Updating cache
[2017-06-14 16:02:45,057: INFO/job_scheduler] Job 52: locked_item 140623202469712 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-14 16:02:45,062: INFO/job_scheduler] Job 52 completing (errored=False, cancelled=False)
[2017-06-14 16:02:45,092: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-3
[2017-06-14 16:02:45,093: INFO/job_scheduler] _notify: Updating .mcast_port of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 49429
[2017-06-14 16:02:45,097: INFO/job_scheduler] _notify: Set mcast_port=49429 on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-3) and saved
[2017-06-14 16:02:45,098: INFO/job_scheduler] _notify: Updating .network_interfaces of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-06-14 16:02:45,106: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-3) and saved
[2017-06-14 16:02:45,126: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-06-14 16:02:45,134: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-3) and saved
[2017-06-14 16:02:45,228: INFO/job_scheduler] run_next: 1 runnable jobs of (7 pending, 3 tasked)
[2017-06-14 16:02:45,350: INFO/job_scheduler] Job 53: RunJobThread.run
[2017-06-14 16:02:45,353: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-06-14 16:02:45,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 463a3c68-b1ca-4dd5-97fb-f27b37e2cab8
[2017-06-14 16:02:45,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 463a3c68-b1ca-4dd5-97fb-f27b37e2cab8
[2017-06-14 16:02:45,419: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:02:45,419: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:02:45,419: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:02:45,419: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:02:45,419: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8e4051b2-ad1e-4e0a-9e0f-056ed05a8cf6
[2017-06-14 16:02:45,419: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8e4051b2-ad1e-4e0a-9e0f-056ed05a8cf6
[2017-06-14 16:02:45,434: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8e4051b2-ad1e-4e0a-9e0f-056ed05a8cf6
[2017-06-14 16:02:45,434: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8e4051b2-ad1e-4e0a-9e0f-056ed05a8cf6
[2017-06-14 16:02:45,436: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-06-14 16:02:45,468: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 851b7fa2-5708-448d-8d26-12e1512f68f9
[2017-06-14 16:02:45,468: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 851b7fa2-5708-448d-8d26-12e1512f68f9
[2017-06-14 16:02:46,775: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-06-14 16:02:46,793: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-06-14 16:02:46,855: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:02:46,855: INFO/job_scheduler]   Job 28 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:46,855: INFO/job_scheduler]   Job 30 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:46,856: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-06-14 16:02:46,871: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:02:46,871: INFO/job_scheduler]   Job 28 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 16:02:46,871: INFO/job_scheduler]   Job 31 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:02:46,872: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-06-14 16:02:46,894: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:02:46,894: INFO/job_scheduler]   Job 39 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:46,895: INFO/job_scheduler]   Job 41 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:46,895: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-06-14 16:02:46,926: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:02:46,926: INFO/job_scheduler]   Job 39 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-14 16:02:46,926: INFO/job_scheduler]   Job 42 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:02:46,927: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-06-14 16:02:47,012: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:02:47,012: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:02:47,012: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:02:47,012: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:02:47,013: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b99aa6ec-a40a-4156-881b-ee15729bed65
[2017-06-14 16:02:47,013: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b99aa6ec-a40a-4156-881b-ee15729bed65
[2017-06-14 16:02:47,036: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b99aa6ec-a40a-4156-881b-ee15729bed65
[2017-06-14 16:02:47,036: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b99aa6ec-a40a-4156-881b-ee15729bed65
[2017-06-14 16:02:47,038: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-06-14 16:02:47,056: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1f32b625-6a0a-425b-9564-1ba9d55cf9b5
[2017-06-14 16:02:47,056: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1f32b625-6a0a-425b-9564-1ba9d55cf9b5
[2017-06-14 16:02:47,326: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-06-14 16:02:47,371: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-06-14 16:02:47,429: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from True to False
[2017-06-14 16:02:47,435: INFO/job_scheduler] _notify: Set corosync_reported_up=False on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-3) and saved
[2017-06-14 16:02:47,445: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-06-14 16:02:47,463: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:02:47,463: INFO/job_scheduler]   Job 53 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:02:47,463: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-06-14 16:02:47,498: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:02:47,498: INFO/job_scheduler]   Job 62 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 16:02:47,498: INFO/job_scheduler]   Job 64 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:02:47,498: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-06-14 16:02:48,759: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:02:48,759: INFO/job_scheduler]   Job 28 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:48,759: INFO/job_scheduler]   Job 30 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:48,760: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-06-14 16:02:48,776: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:02:48,776: INFO/job_scheduler]   Job 28 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 16:02:48,776: INFO/job_scheduler]   Job 31 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:02:48,776: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-06-14 16:02:48,823: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:02:48,823: INFO/job_scheduler]   Job 39 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:48,823: INFO/job_scheduler]   Job 41 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:02:48,823: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-06-14 16:02:48,851: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:02:48,852: INFO/job_scheduler]   Job 39 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-14 16:02:48,852: INFO/job_scheduler]   Job 42 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:02:48,852: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-06-14 16:02:49,250: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:02:49,250: INFO/job_scheduler]   Job 53 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:02:49,250: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-06-14 16:02:49,271: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:02:49,271: INFO/job_scheduler]   Job 62 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 16:02:49,271: INFO/job_scheduler]   Job 64 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:02:49,272: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-06-14 16:02:50,244: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:02:50,244: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:02:50,245: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:02:50,245: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:02:50,245: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2af827b0-26d8-45fd-9f5e-64bd9a4b56e7
[2017-06-14 16:02:50,245: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2af827b0-26d8-45fd-9f5e-64bd9a4b56e7
[2017-06-14 16:02:50,292: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2af827b0-26d8-45fd-9f5e-64bd9a4b56e7
[2017-06-14 16:02:50,292: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2af827b0-26d8-45fd-9f5e-64bd9a4b56e7
[2017-06-14 16:02:50,294: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-06-14 16:02:50,318: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dd8fd2b3-e668-46eb-9213-d704d8f9d66e
[2017-06-14 16:02:50,318: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dd8fd2b3-e668-46eb-9213-d704d8f9d66e
[2017-06-14 16:02:54,150: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:02:54,150: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:02:54,150: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:02:54,150: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:02:54,150: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1f32b625-6a0a-425b-9564-1ba9d55cf9b5
[2017-06-14 16:02:54,150: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1f32b625-6a0a-425b-9564-1ba9d55cf9b5
[2017-06-14 16:02:54,176: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1f32b625-6a0a-425b-9564-1ba9d55cf9b5
[2017-06-14 16:02:54,176: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1f32b625-6a0a-425b-9564-1ba9d55cf9b5
[2017-06-14 16:02:54,176: INFO/job_scheduler] Job 28 finished 3 steps successfully
[2017-06-14 16:02:54,229: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 140624277296720
[2017-06-14 16:02:54,245: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-06-14 16:02:54,260: INFO/job] Job 28: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-14 16:02:54,260: INFO/job] Job 28: not_deleted=True
[2017-06-14 16:02:54,260: INFO/job_scheduler] Job 28: complete_job: Updating cache
[2017-06-14 16:02:54,260: INFO/job_scheduler] Job 28: locked_item 140624277296720 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-14 16:02:54,270: INFO/job_scheduler] Job 28 completing (errored=False, cancelled=False)
[2017-06-14 16:02:54,334: INFO/job_scheduler] run_next: 1 runnable jobs of (6 pending, 3 tasked)
[2017-06-14 16:02:54,397: INFO/job_scheduler] Job 30: RunJobThread.run
[2017-06-14 16:02:54,398: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-06-14 16:02:54,459: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f78f0733-fbdc-4e31-bae6-79eeae6a2bbf
[2017-06-14 16:02:54,459: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f78f0733-fbdc-4e31-bae6-79eeae6a2bbf
[2017-06-14 16:02:57,322: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-06-14 16:02:57,340: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-06-14 16:02:57,369: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:02:57,370: INFO/job_scheduler]   Job 53 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:02:57,370: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-06-14 16:02:57,390: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:02:57,390: INFO/job_scheduler]   Job 62 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 16:02:57,391: INFO/job_scheduler]   Job 64 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:02:57,391: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-06-14 16:02:57,425: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:02:57,425: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:02:57,426: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:02:57,426: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:02:57,426: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dd8fd2b3-e668-46eb-9213-d704d8f9d66e
[2017-06-14 16:02:57,426: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dd8fd2b3-e668-46eb-9213-d704d8f9d66e
[2017-06-14 16:02:57,443: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dd8fd2b3-e668-46eb-9213-d704d8f9d66e
[2017-06-14 16:02:57,443: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dd8fd2b3-e668-46eb-9213-d704d8f9d66e
[2017-06-14 16:02:57,444: INFO/job_scheduler] Job 39 finished 3 steps successfully
[2017-06-14 16:02:57,486: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 140624277305936
[2017-06-14 16:02:57,493: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-06-14 16:02:57,504: INFO/job] Job 39: StateChangeJob complete, setting state stopped on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-14 16:02:57,504: INFO/job] Job 39: not_deleted=True
[2017-06-14 16:02:57,504: INFO/job_scheduler] Job 39: complete_job: Updating cache
[2017-06-14 16:02:57,504: INFO/job_scheduler] Job 39: locked_item 140624277305936 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-14 16:02:57,510: INFO/job_scheduler] Job 39 completing (errored=False, cancelled=False)
[2017-06-14 16:02:57,547: INFO/job_scheduler] run_next: 1 runnable jobs of (5 pending, 3 tasked)
[2017-06-14 16:02:57,590: INFO/job_scheduler] Job 41: RunJobThread.run
[2017-06-14 16:02:57,591: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-06-14 16:02:57,644: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 78d4daf3-2cf0-41e2-be08-08f0f1008057
[2017-06-14 16:02:57,644: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 78d4daf3-2cf0-41e2-be08-08f0f1008057
[2017-06-14 16:02:59,266: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:02:59,266: INFO/job_scheduler]   Job 53 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:02:59,266: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 4 tasked)
[2017-06-14 16:02:59,285: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:02:59,285: INFO/job_scheduler]   Job 62 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 16:02:59,285: INFO/job_scheduler]   Job 64 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:02:59,285: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 4 tasked)
[2017-06-14 16:03:01,565: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:03:01,565: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:03:01,565: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:03:01,565: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:03:01,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f78f0733-fbdc-4e31-bae6-79eeae6a2bbf
[2017-06-14 16:03:01,566: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f78f0733-fbdc-4e31-bae6-79eeae6a2bbf
[2017-06-14 16:03:01,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f78f0733-fbdc-4e31-bae6-79eeae6a2bbf
[2017-06-14 16:03:01,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f78f0733-fbdc-4e31-bae6-79eeae6a2bbf
[2017-06-14 16:03:01,589: INFO/job_scheduler] Job 30 finished 1 steps successfully
[2017-06-14 16:03:01,627: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 140624277296720
[2017-06-14 16:03:01,658: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 30
[2017-06-14 16:03:01,674: INFO/job] Job 30: StateChangeJob complete, setting state started on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-14 16:03:01,675: INFO/job] Job 30: not_deleted=True
[2017-06-14 16:03:01,675: INFO/job_scheduler] Job 30: complete_job: Updating cache
[2017-06-14 16:03:01,675: INFO/job_scheduler] Job 30: locked_item 140624277296720 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-14 16:03:01,678: INFO/job_scheduler] Job 30 completing (errored=False, cancelled=False)
[2017-06-14 16:03:01,716: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-1
[2017-06-14 16:03:01,716: INFO/job_scheduler] _notify: Updating .mcast_port of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 65087
[2017-06-14 16:03:01,729: INFO/job_scheduler] _notify: Set mcast_port=65087 on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-06-14 16:03:01,738: INFO/job_scheduler] _notify: Updating .network_interfaces of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-06-14 16:03:01,758: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-06-14 16:03:01,764: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-06-14 16:03:01,776: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-06-14 16:03:01,797: INFO/job_scheduler] run_next: 1 runnable jobs of (4 pending, 3 tasked)
[2017-06-14 16:03:01,851: INFO/job_scheduler] Job 31: RunJobThread.run
[2017-06-14 16:03:01,854: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-06-14 16:03:01,889: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5741b53c-12e3-44a3-912f-6e80bbbafed9
[2017-06-14 16:03:01,889: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5741b53c-12e3-44a3-912f-6e80bbbafed9
[2017-06-14 16:03:04,727: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:03:04,727: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:03:04,727: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:03:04,727: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:03:04,727: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 78d4daf3-2cf0-41e2-be08-08f0f1008057
[2017-06-14 16:03:04,727: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 78d4daf3-2cf0-41e2-be08-08f0f1008057
[2017-06-14 16:03:04,767: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 78d4daf3-2cf0-41e2-be08-08f0f1008057
[2017-06-14 16:03:04,767: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 78d4daf3-2cf0-41e2-be08-08f0f1008057
[2017-06-14 16:03:04,768: INFO/job_scheduler] Job 41 finished 1 steps successfully
[2017-06-14 16:03:04,829: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 140624277305936
[2017-06-14 16:03:04,859: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm6.lotus.hpdd.lab.intel.com at severity 30
[2017-06-14 16:03:04,877: INFO/job] Job 41: StateChangeJob complete, setting state started on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-14 16:03:04,877: INFO/job] Job 41: not_deleted=True
[2017-06-14 16:03:04,877: INFO/job_scheduler] Job 41: complete_job: Updating cache
[2017-06-14 16:03:04,877: INFO/job_scheduler] Job 41: locked_item 140624277305936 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-14 16:03:04,881: INFO/job_scheduler] Job 41 completing (errored=False, cancelled=False)
[2017-06-14 16:03:04,923: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-2
[2017-06-14 16:03:04,923: INFO/job_scheduler] _notify: Updating .mcast_port of item 2 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 65087
[2017-06-14 16:03:04,928: INFO/job_scheduler] _notify: Set mcast_port=65087 on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-2) and saved
[2017-06-14 16:03:04,933: INFO/job_scheduler] _notify: Updating .network_interfaces of item 2 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-06-14 16:03:04,965: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-2) and saved
[2017-06-14 16:03:04,983: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 2 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-06-14 16:03:04,988: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-2) and saved
[2017-06-14 16:03:05,044: INFO/job_scheduler] run_next: 1 runnable jobs of (3 pending, 3 tasked)
[2017-06-14 16:03:05,111: INFO/job_scheduler] Job 42: RunJobThread.run
[2017-06-14 16:03:05,118: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-06-14 16:03:05,142: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 85367e92-8019-4370-b223-e82c9de19651
[2017-06-14 16:03:05,142: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 85367e92-8019-4370-b223-e82c9de19651
[2017-06-14 16:03:06,863: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from True to False
[2017-06-14 16:03:06,868: INFO/job_scheduler] _notify: Set corosync_reported_up=False on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-06-14 16:03:06,885: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-06-14 16:03:06,897: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:03:06,897: INFO/job_scheduler]   Job 31 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:03:06,898: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-06-14 16:03:06,920: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 2 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from True to False
[2017-06-14 16:03:06,925: INFO/job_scheduler] _notify: Set corosync_reported_up=False on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-2) and saved
[2017-06-14 16:03:06,943: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-06-14 16:03:06,955: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:03:06,955: INFO/job_scheduler]   Job 42 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:03:06,955: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-06-14 16:03:08,694: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:03:08,694: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:03:08,695: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:03:08,695: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 16:03:08,695: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 463a3c68-b1ca-4dd5-97fb-f27b37e2cab8
[2017-06-14 16:03:08,695: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 463a3c68-b1ca-4dd5-97fb-f27b37e2cab8
[2017-06-14 16:03:08,716: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 463a3c68-b1ca-4dd5-97fb-f27b37e2cab8
[2017-06-14 16:03:08,716: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 463a3c68-b1ca-4dd5-97fb-f27b37e2cab8
[2017-06-14 16:03:08,718: INFO/job_scheduler] Job 53 finished 1 steps successfully
[2017-06-14 16:03:08,843: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:03:08,843: INFO/job_scheduler]   Job 31 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:03:08,843: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-06-14 16:03:08,910: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:03:08,910: INFO/job_scheduler]   Job 42 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:03:08,910: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 4 tasked)
[2017-06-14 16:03:09,312: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 140623806239120
[2017-06-14 16:03:09,322: INFO/job] Job 53: StateChangeJob complete, setting state managed on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-14 16:03:09,322: INFO/job] Job 53: not_deleted=True
[2017-06-14 16:03:09,322: INFO/job_scheduler] Job 53: complete_job: Updating cache
[2017-06-14 16:03:09,322: INFO/job_scheduler] Job 53: locked_item 140623806239120 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-14 16:03:09,325: INFO/job_scheduler] Job 53 completing (errored=False, cancelled=False)
[2017-06-14 16:03:09,385: INFO/job_scheduler] Replaying 4 buffered notifications for ManagedHost-3
[2017-06-14 16:03:09,426: INFO/job_scheduler] _notify: Updating .immutable_state of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com) from True to False
[2017-06-14 16:03:09,435: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-3) and saved
[2017-06-14 16:03:09,460: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7fe588060dd0> to [4]
[2017-06-14 16:03:09,493: INFO/job_scheduler] _notify: Set ha_cluster_peers=[4] on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-3) and saved
[2017-06-14 16:03:09,644: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 3 tasked)
[2017-06-14 16:03:09,646: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-06-14 16:03:09,653: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-3) and saved
[2017-06-14 16:03:09,658: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 3 tasked)
[2017-06-14 16:03:09,673: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 3 (lotus-32vm7.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7fe56c0549d0> to [4]
[2017-06-14 16:03:09,717: INFO/job_scheduler] _notify: Set ha_cluster_peers=[4] on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-3) and saved
[2017-06-14 16:03:09,941: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 3 tasked)
[2017-06-14 16:03:09,983: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:03:09,984: INFO/job_scheduler]   Job 62 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:03:09,984: INFO/job_scheduler]   Job 63 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:03:09,984: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 3 tasked)
[2017-06-14 16:03:09,986: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:03:09,986: INFO/job_scheduler]   Job 62 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 16:03:09,986: INFO/job_scheduler]   Job 64 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:03:09,986: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 3 tasked)
[2017-06-14 16:03:16,858: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:03:16,858: INFO/job_scheduler]   Job 31 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:03:16,858: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 3 tasked)
[2017-06-14 16:03:16,904: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:03:16,912: INFO/job_scheduler]   Job 42 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:03:16,912: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 3 tasked)
[2017-06-14 16:03:17,408: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 3 tasked)
[2017-06-14 16:03:17,437: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 3 tasked)
[2017-06-14 16:03:17,500: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:03:17,500: INFO/job_scheduler]   Job 62 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:03:17,500: INFO/job_scheduler]   Job 63 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:03:17,501: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 3 tasked)
[2017-06-14 16:03:18,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:03:18,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:03:18,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:03:18,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:03:18,613: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 851b7fa2-5708-448d-8d26-12e1512f68f9
[2017-06-14 16:03:18,613: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 851b7fa2-5708-448d-8d26-12e1512f68f9
[2017-06-14 16:03:18,615: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 851b7fa2-5708-448d-8d26-12e1512f68f9
[2017-06-14 16:03:18,615: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 851b7fa2-5708-448d-8d26-12e1512f68f9
[2017-06-14 16:03:18,616: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-06-14 16:03:18,633: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3605a267-6913-4ccc-99eb-ad6dc149f9f3
[2017-06-14 16:03:18,633: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3605a267-6913-4ccc-99eb-ad6dc149f9f3
[2017-06-14 16:03:18,845: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:03:18,845: INFO/job_scheduler]   Job 31 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:03:18,849: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 3 tasked)
[2017-06-14 16:03:18,900: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:03:18,900: INFO/job_scheduler]   Job 42 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:03:18,901: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 3 tasked)
[2017-06-14 16:03:19,312: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:03:19,312: INFO/job_scheduler]   Job 62 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:03:19,312: INFO/job_scheduler]   Job 63 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:03:19,313: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 3 tasked)
[2017-06-14 16:03:26,923: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-06-14 16:03:26,928: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-06-14 16:03:26,941: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 3 tasked)
[2017-06-14 16:03:26,970: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:03:26,970: INFO/job_scheduler]   Job 31 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:03:26,970: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 3 tasked)
[2017-06-14 16:03:26,984: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 2 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-06-14 16:03:26,988: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-2) and saved
[2017-06-14 16:03:26,994: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 3 tasked)
[2017-06-14 16:03:27,002: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-14 16:03:27,003: INFO/job_scheduler]   Job 42 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-14 16:03:27,003: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 3 tasked)
[2017-06-14 16:03:27,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:03:27,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:03:27,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:03:27,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 16:03:27,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5741b53c-12e3-44a3-912f-6e80bbbafed9
[2017-06-14 16:03:27,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5741b53c-12e3-44a3-912f-6e80bbbafed9
[2017-06-14 16:03:27,319: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5741b53c-12e3-44a3-912f-6e80bbbafed9
[2017-06-14 16:03:27,319: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5741b53c-12e3-44a3-912f-6e80bbbafed9
[2017-06-14 16:03:27,320: INFO/job_scheduler] Job 31 finished 1 steps successfully
[2017-06-14 16:03:27,486: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 3 tasked)
[2017-06-14 16:03:27,514: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 3 tasked)
[2017-06-14 16:03:27,584: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:03:27,585: INFO/job_scheduler]   Job 62 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:03:27,585: INFO/job_scheduler]   Job 63 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:03:27,585: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 3 tasked)
[2017-06-14 16:03:27,973: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 140623806293456
[2017-06-14 16:03:27,978: INFO/job] Job 31: StateChangeJob complete, setting state managed on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-14 16:03:27,978: INFO/job] Job 31: not_deleted=True
[2017-06-14 16:03:27,979: INFO/job_scheduler] Job 31: complete_job: Updating cache
[2017-06-14 16:03:27,979: INFO/job_scheduler] Job 31: locked_item 140623806293456 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-14 16:03:27,988: INFO/job_scheduler] Job 31 completing (errored=False, cancelled=False)
[2017-06-14 16:03:28,051: INFO/job_scheduler] Replaying 5 buffered notifications for ManagedHost-1
[2017-06-14 16:03:28,069: INFO/job_scheduler] _notify: Updating .immutable_state of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com) from True to False
[2017-06-14 16:03:28,093: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-1) and saved
[2017-06-14 16:03:28,100: INFO/job_scheduler] _notify: Updating .boot_time of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com) from 2017-05-10 06:56:09+00:00 to 2017-06-14 09:00:19+00:00
[2017-06-14 16:03:28,117: INFO/job_scheduler] _notify: Set boot_time=2017-06-14 09:00:19+00:00 on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-1) and saved
[2017-06-14 16:03:28,176: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7fe59c0c1210> to [2]
[2017-06-14 16:03:28,190: INFO/job_scheduler] _notify: Set ha_cluster_peers=[2] on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-1) and saved
[2017-06-14 16:03:28,365: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 2 tasked)
[2017-06-14 16:03:29,316: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:03:29,316: INFO/job_scheduler]   Job 62 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:03:29,316: INFO/job_scheduler]   Job 63 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:03:29,317: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 2 tasked)
[2017-06-14 16:03:31,454: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:03:31,454: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:03:31,455: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:03:31,455: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 16:03:31,455: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 85367e92-8019-4370-b223-e82c9de19651
[2017-06-14 16:03:31,455: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 85367e92-8019-4370-b223-e82c9de19651
[2017-06-14 16:03:31,481: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 85367e92-8019-4370-b223-e82c9de19651
[2017-06-14 16:03:31,481: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 85367e92-8019-4370-b223-e82c9de19651
[2017-06-14 16:03:31,482: INFO/job_scheduler] Job 42 finished 1 steps successfully
[2017-06-14 16:03:32,013: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 140623202468368
[2017-06-14 16:03:32,018: INFO/job] Job 42: StateChangeJob complete, setting state managed on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-14 16:03:32,018: INFO/job] Job 42: not_deleted=True
[2017-06-14 16:03:32,018: INFO/job_scheduler] Job 42: complete_job: Updating cache
[2017-06-14 16:03:32,019: INFO/job_scheduler] Job 42: locked_item 140623202468368 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-14 16:03:32,021: INFO/job_scheduler] Job 42 completing (errored=False, cancelled=False)
[2017-06-14 16:03:32,091: INFO/job_scheduler] Replaying 4 buffered notifications for ManagedHost-2
[2017-06-14 16:03:32,119: INFO/job_scheduler] _notify: Updating .immutable_state of item 2 (lotus-32vm6.lotus.hpdd.lab.intel.com) from True to False
[2017-06-14 16:03:32,149: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-2) and saved
[2017-06-14 16:03:32,180: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 2 (lotus-32vm6.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7fe584081a90> to [1]
[2017-06-14 16:03:32,207: INFO/job_scheduler] _notify: Set ha_cluster_peers=[1] on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-2) and saved
[2017-06-14 16:03:32,371: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-14 16:03:37,517: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:03:37,518: INFO/job_scheduler]   Job 62 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:03:37,518: INFO/job_scheduler]   Job 63 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:03:37,518: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-14 16:03:39,377: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:03:39,377: INFO/job_scheduler]   Job 62 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:03:39,377: INFO/job_scheduler]   Job 63 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:03:39,377: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-14 16:03:47,562: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:03:47,562: INFO/job_scheduler]   Job 62 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:03:47,562: INFO/job_scheduler]   Job 63 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:03:47,562: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-14 16:03:49,418: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:03:49,418: INFO/job_scheduler]   Job 62 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:03:49,419: INFO/job_scheduler]   Job 63 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:03:49,419: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-14 16:03:57,566: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:03:57,566: INFO/job_scheduler]   Job 62 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:03:57,567: INFO/job_scheduler]   Job 63 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:03:57,567: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-14 16:03:59,506: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:03:59,506: INFO/job_scheduler]   Job 62 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:03:59,506: INFO/job_scheduler]   Job 63 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:03:59,507: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-14 16:04:07,636: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:04:07,636: INFO/job_scheduler]   Job 62 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:04:07,636: INFO/job_scheduler]   Job 63 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:04:07,636: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-14 16:04:09,465: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:04:09,466: INFO/job_scheduler]   Job 62 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:04:09,466: INFO/job_scheduler]   Job 63 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:04:09,466: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-14 16:04:17,625: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:04:17,625: INFO/job_scheduler]   Job 62 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:04:17,625: INFO/job_scheduler]   Job 63 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:04:17,625: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-14 16:04:19,547: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:04:19,547: INFO/job_scheduler]   Job 62 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:04:19,547: INFO/job_scheduler]   Job 63 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:04:19,547: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-14 16:04:28,791: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:04:28,791: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:04:28,791: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:04:28,791: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:04:28,791: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3605a267-6913-4ccc-99eb-ad6dc149f9f3
[2017-06-14 16:04:28,791: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3605a267-6913-4ccc-99eb-ad6dc149f9f3
[2017-06-14 16:04:28,794: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3605a267-6913-4ccc-99eb-ad6dc149f9f3
[2017-06-14 16:04:28,794: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3605a267-6913-4ccc-99eb-ad6dc149f9f3
[2017-06-14 16:04:28,794: INFO/job_scheduler] Job 62 finished 3 steps successfully
[2017-06-14 16:04:28,870: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 140623806413328
[2017-06-14 16:04:28,878: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-06-14 16:04:28,880: INFO/job] Job 62: StateChangeJob complete, setting state stopped on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-14 16:04:28,880: INFO/job] Job 62: not_deleted=True
[2017-06-14 16:04:28,880: INFO/job_scheduler] Job 62: complete_job: Updating cache
[2017-06-14 16:04:28,880: INFO/job_scheduler] Job 62: locked_item 140623806413328 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-14 16:04:28,883: INFO/job_scheduler] Job 62 completing (errored=False, cancelled=False)
[2017-06-14 16:04:28,905: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 0 tasked)
[2017-06-14 16:04:28,916: INFO/job_scheduler] Job 63: RunJobThread.run
[2017-06-14 16:04:28,917: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-06-14 16:04:28,931: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6ae13fbe-5511-441c-950f-06eea7c548e9
[2017-06-14 16:04:28,931: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6ae13fbe-5511-441c-950f-06eea7c548e9
[2017-06-14 16:04:29,594: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-14 16:04:29,594: INFO/job_scheduler]   Job 63 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-14 16:04:29,595: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-06-14 16:04:36,028: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:04:36,028: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:04:36,029: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:04:36,029: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:04:36,029: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6ae13fbe-5511-441c-950f-06eea7c548e9
[2017-06-14 16:04:36,029: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6ae13fbe-5511-441c-950f-06eea7c548e9
[2017-06-14 16:04:36,055: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6ae13fbe-5511-441c-950f-06eea7c548e9
[2017-06-14 16:04:36,055: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6ae13fbe-5511-441c-950f-06eea7c548e9
[2017-06-14 16:04:36,055: INFO/job_scheduler] Job 63 finished 1 steps successfully
[2017-06-14 16:04:36,086: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 140623806413328
[2017-06-14 16:04:36,131: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm8.lotus.hpdd.lab.intel.com at severity 30
[2017-06-14 16:04:36,147: INFO/job] Job 63: StateChangeJob complete, setting state started on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-14 16:04:36,147: INFO/job] Job 63: not_deleted=True
[2017-06-14 16:04:36,147: INFO/job_scheduler] Job 63: complete_job: Updating cache
[2017-06-14 16:04:36,148: INFO/job_scheduler] Job 63: locked_item 140623806413328 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-14 16:04:36,158: INFO/job_scheduler] Job 63 completing (errored=False, cancelled=False)
[2017-06-14 16:04:36,196: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-4
[2017-06-14 16:04:36,196: INFO/job_scheduler] _notify: Updating .mcast_port of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 49429
[2017-06-14 16:04:36,201: INFO/job_scheduler] _notify: Set mcast_port=49429 on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-4) and saved
[2017-06-14 16:04:36,217: INFO/job_scheduler] _notify: Updating .network_interfaces of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-06-14 16:04:36,242: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-4) and saved
[2017-06-14 16:04:36,262: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-06-14 16:04:36,271: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-4) and saved
[2017-06-14 16:04:36,285: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-14 16:04:36,362: INFO/job_scheduler] Job 64: RunJobThread.run
[2017-06-14 16:04:36,364: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-06-14 16:04:36,407: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 20a726fd-fa38-48b4-b3a5-d746ee88f552
[2017-06-14 16:04:36,407: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 20a726fd-fa38-48b4-b3a5-d746ee88f552
[2017-06-14 16:04:37,710: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-14 16:04:37,720: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-14 16:05:02,736: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:05:02,736: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:05:02,736: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:05:02,736: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 16:05:02,736: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 20a726fd-fa38-48b4-b3a5-d746ee88f552
[2017-06-14 16:05:02,736: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 20a726fd-fa38-48b4-b3a5-d746ee88f552
[2017-06-14 16:05:02,737: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 20a726fd-fa38-48b4-b3a5-d746ee88f552
[2017-06-14 16:05:02,737: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 20a726fd-fa38-48b4-b3a5-d746ee88f552
[2017-06-14 16:05:02,738: INFO/job_scheduler] Job 64 finished 1 steps successfully
[2017-06-14 16:05:03,190: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 140623806319376
[2017-06-14 16:05:03,195: INFO/job] Job 64: StateChangeJob complete, setting state managed on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-14 16:05:03,195: INFO/job] Job 64: not_deleted=True
[2017-06-14 16:05:03,195: INFO/job_scheduler] Job 64: complete_job: Updating cache
[2017-06-14 16:05:03,195: INFO/job_scheduler] Job 64: locked_item 140623806319376 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-14 16:05:03,198: INFO/job_scheduler] Job 64 completing (errored=False, cancelled=False)
[2017-06-14 16:05:03,242: INFO/job_scheduler] Replaying 5 buffered notifications for ManagedHost-4
[2017-06-14 16:05:03,263: INFO/job_scheduler] _notify: Updating .immutable_state of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com) from True to False
[2017-06-14 16:05:03,273: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-4) and saved
[2017-06-14 16:05:03,304: INFO/job_scheduler] _notify: Updating .boot_time of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com) from 2017-04-25 07:09:56+00:00 to 2017-06-14 09:00:40+00:00
[2017-06-14 16:05:03,331: INFO/job_scheduler] _notify: Set boot_time=2017-06-14 09:00:40+00:00 on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-4) and saved
[2017-06-14 16:05:03,392: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 4 (lotus-32vm8.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7fe5880793d0> to [3]
[2017-06-14 16:05:03,442: INFO/job_scheduler] _notify: Set ha_cluster_peers=[3] on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-4) and saved
[2017-06-14 16:05:03,619: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:08:40,300: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:08:40,320: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:08:50,008: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:08:50,043: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:11:38,340: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:11:38,366: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:11:59,032: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:11:59,053: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:18:52,219: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:18:52,238: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:19:01,608: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:19:01,620: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:21:50,111: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:21:50,130: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:22:10,943: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:22:10,956: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:29:04,149: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:29:04,167: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:29:13,295: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:29:13,310: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:32:01,884: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:32:01,897: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:32:22,862: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:32:22,877: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:39:16,038: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:39:16,053: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:39:24,984: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:39:25,001: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:42:13,709: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:42:13,731: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:42:34,677: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:42:34,693: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:49:27,852: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:49:27,872: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:49:36,694: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:49:36,710: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:52:25,485: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:52:25,505: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:52:46,635: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:52:46,660: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:59:39,521: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:59:39,534: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:59:48,269: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 16:59:48,284: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:02:37,173: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:02:37,187: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:02:58,471: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:02:58,491: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:09:51,164: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:09:51,184: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:09:59,758: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:09:59,773: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:12:48,782: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:12:48,804: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:13:10,265: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:13:10,283: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:20:02,921: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:20:02,940: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:20:11,307: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:20:11,324: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:23:00,415: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:23:00,429: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:23:22,154: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:23:22,168: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:30:14,841: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:30:14,861: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:30:22,890: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:30:22,908: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:33:12,009: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:33:12,022: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:33:34,155: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:33:34,169: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:40:26,648: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:40:26,666: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:40:34,589: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:40:34,607: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:43:23,831: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:43:23,848: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:43:46,047: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:43:46,063: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:50:38,598: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:50:38,612: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:50:46,402: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:50:46,419: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:53:35,584: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:53:35,601: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:53:57,805: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 17:53:57,823: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:00:50,412: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:00:50,432: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:00:58,061: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:00:58,204: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:03:47,194: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:03:47,215: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:04:09,584: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:04:09,603: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:11:02,365: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:11:02,389: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:11:09,765: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:11:09,788: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:13:59,016: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:13:59,045: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:14:21,561: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:14:21,583: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:21:14,229: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:21:14,243: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:21:21,505: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:21:21,524: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:24:10,889: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:24:10,909: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:24:33,521: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:24:33,541: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:31:25,972: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:31:25,989: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:31:32,989: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:31:33,014: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:34:22,568: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:34:22,585: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:34:45,438: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:34:45,460: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:41:37,826: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:41:37,839: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:41:44,515: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:41:44,528: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:44:34,172: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:44:34,197: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:44:57,293: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:44:57,338: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:51:49,629: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:51:49,645: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:51:56,140: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:51:56,157: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:54:45,794: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:54:45,815: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:55:09,128: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 18:55:09,143: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:02:01,525: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:02:01,557: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:02:07,644: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:02:07,659: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:04:57,393: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:04:57,407: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:05:21,037: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:05:21,061: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:12:13,199: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:12:13,212: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:12:19,186: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:12:19,211: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:15:09,263: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:15:09,280: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:15:32,902: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:15:32,920: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:22:24,943: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:22:24,960: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:22:30,748: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:22:30,770: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:25:20,943: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:25:20,965: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:25:44,662: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:25:44,679: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:32:36,783: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:32:36,798: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:32:42,445: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:32:42,462: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:35:32,656: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:35:32,675: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:35:56,479: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:35:56,491: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:42:48,428: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:42:48,444: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:42:54,006: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:42:54,024: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:45:44,447: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:45:44,472: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:46:08,361: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:46:08,382: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:53:00,260: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:53:00,273: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:53:05,536: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:53:05,547: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:55:56,272: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:55:56,293: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:56:20,273: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 19:56:20,289: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:03:11,991: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:03:12,004: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:03:17,186: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:03:17,200: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:06:08,060: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:06:08,073: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:06:32,196: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:06:32,209: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:11:44,650: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-14 20:11:44,653: INFO/command_plan] set_state: <class 'chroma_core.models.filesystem.ManagedFilesystem'>-1 transitioning from unavailable to available
[2017-06-14 20:11:45,153: INFO/command_plan] Created command 26 (Creating filesystem testfs) with 13 jobs
[2017-06-14 20:11:45,184: INFO/job_scheduler] run_next: 3 runnable jobs of (13 pending, 0 tasked)
[2017-06-14 20:11:45,438: INFO/job] create_parameters: host: 'lotus-32vm5.lotus.hpdd.lab.intel.com' active_volume_node: 'TargetVolumeInfo(host=<ManagedHost: lotus-32vm5.lotus.hpdd.lab.intel.com>, path=u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5', device_type='zfs')'
[2017-06-14 20:11:45,654: INFO/job] create_parameters: host: 'lotus-32vm8.lotus.hpdd.lab.intel.com' active_volume_node: 'TargetVolumeInfo(host=<ManagedHost: lotus-32vm8.lotus.hpdd.lab.intel.com>, path=u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3', device_type='zfs')'
[2017-06-14 20:11:45,809: INFO/job] create_parameters: host: 'lotus-32vm7.lotus.hpdd.lab.intel.com' active_volume_node: 'TargetVolumeInfo(host=<ManagedHost: lotus-32vm7.lotus.hpdd.lab.intel.com>, path=u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1', device_type='zfs')'
[2017-06-14 20:11:45,849: INFO/job_scheduler] Job 66: RunJobThread.run
[2017-06-14 20:11:45,858: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5', 'device_type': 'zfs'}
[2017-06-14 20:11:45,867: INFO/job_scheduler] Job 67: RunJobThread.run
[2017-06-14 20:11:45,891: INFO/job_scheduler] Job 68: RunJobThread.run
[2017-06-14 20:11:45,896: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4d5ad05c-c59e-4322-8f64-28bb195c59f4
[2017-06-14 20:11:45,896: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4d5ad05c-c59e-4322-8f64-28bb195c59f4
[2017-06-14 20:11:45,891: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5', 'device_type': 'zfs'}
[2017-06-14 20:11:45,915: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3', 'device_type': 'zfs'}
[2017-06-14 20:11:45,922: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5', 'device_type': 'zfs'}
[2017-06-14 20:11:45,928: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3', 'device_type': 'zfs'}
[2017-06-14 20:11:45,934: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1', 'device_type': 'zfs'}
[2017-06-14 20:11:45,946: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b2b4b9c7-45cf-4b59-8a05-38bae87aad11
[2017-06-14 20:11:45,946: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b2b4b9c7-45cf-4b59-8a05-38bae87aad11
[2017-06-14 20:11:45,964: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c8c87359-f621-4633-a48d-e85a2d5f67c9
[2017-06-14 20:11:45,964: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c8c87359-f621-4633-a48d-e85a2d5f67c9
[2017-06-14 20:11:45,975: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1', 'device_type': 'zfs'}
[2017-06-14 20:11:45,997: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1', 'device_type': 'zfs'}
[2017-06-14 20:11:46,006: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3', 'device_type': 'zfs'}
[2017-06-14 20:11:46,007: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4a57984b-166d-4237-a214-7c65e4499bcb
[2017-06-14 20:11:46,007: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4a57984b-166d-4237-a214-7c65e4499bcb
[2017-06-14 20:11:46,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5a1d1d88-27ec-4dc4-9838-4ef2f24c63d8
[2017-06-14 20:11:46,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5a1d1d88-27ec-4dc4-9838-4ef2f24c63d8
[2017-06-14 20:11:46,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fc01e403-e9d4-438f-a5f2-b364e4ee0192
[2017-06-14 20:11:46,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fc01e403-e9d4-438f-a5f2-b364e4ee0192
[2017-06-14 20:11:46,099: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0226cd8b-3114-4927-b1a7-3257ce7586b8
[2017-06-14 20:11:46,099: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0226cd8b-3114-4927-b1a7-3257ce7586b8
[2017-06-14 20:11:46,112: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 750544e8-eee2-460f-9edc-efac448c4318
[2017-06-14 20:11:46,112: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 750544e8-eee2-460f-9edc-efac448c4318
[2017-06-14 20:11:46,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6bcebfc5-e69a-48b1-bfd5-e113047de25f
[2017-06-14 20:11:46,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6bcebfc5-e69a-48b1-bfd5-e113047de25f
[2017-06-14 20:11:47,021: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:11:47,021: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:11:47,021: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:11:47,021: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:11:47,022: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b2b4b9c7-45cf-4b59-8a05-38bae87aad11
[2017-06-14 20:11:47,022: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b2b4b9c7-45cf-4b59-8a05-38bae87aad11
[2017-06-14 20:11:47,036: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:11:47,036: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:11:47,036: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:11:47,036: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:11:47,036: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c8c87359-f621-4633-a48d-e85a2d5f67c9
[2017-06-14 20:11:47,036: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c8c87359-f621-4633-a48d-e85a2d5f67c9
[2017-06-14 20:11:47,049: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c8c87359-f621-4633-a48d-e85a2d5f67c9
[2017-06-14 20:11:47,049: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c8c87359-f621-4633-a48d-e85a2d5f67c9
[2017-06-14 20:11:47,069: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b2b4b9c7-45cf-4b59-8a05-38bae87aad11
[2017-06-14 20:11:47,069: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b2b4b9c7-45cf-4b59-8a05-38bae87aad11
[2017-06-14 20:11:47,125: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:11:47,125: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:11:47,125: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:11:47,125: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:11:47,125: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5a1d1d88-27ec-4dc4-9838-4ef2f24c63d8
[2017-06-14 20:11:47,125: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5a1d1d88-27ec-4dc4-9838-4ef2f24c63d8
[2017-06-14 20:11:47,156: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:11:47,156: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:11:47,164: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:11:47,164: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:11:47,164: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fc01e403-e9d4-438f-a5f2-b364e4ee0192
[2017-06-14 20:11:47,164: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fc01e403-e9d4-438f-a5f2-b364e4ee0192
[2017-06-14 20:11:47,164: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fc01e403-e9d4-438f-a5f2-b364e4ee0192
[2017-06-14 20:11:47,164: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fc01e403-e9d4-438f-a5f2-b364e4ee0192
[2017-06-14 20:11:47,167: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5a1d1d88-27ec-4dc4-9838-4ef2f24c63d8
[2017-06-14 20:11:47,167: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5a1d1d88-27ec-4dc4-9838-4ef2f24c63d8
[2017-06-14 20:11:47,247: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:11:47,247: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:11:47,248: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:11:47,248: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:11:47,248: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6bcebfc5-e69a-48b1-bfd5-e113047de25f
[2017-06-14 20:11:47,248: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6bcebfc5-e69a-48b1-bfd5-e113047de25f
[2017-06-14 20:11:47,252: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6bcebfc5-e69a-48b1-bfd5-e113047de25f
[2017-06-14 20:11:47,252: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6bcebfc5-e69a-48b1-bfd5-e113047de25f
[2017-06-14 20:11:48,014: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:11:48,014: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:11:48,014: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:11:48,014: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:11:48,015: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4d5ad05c-c59e-4322-8f64-28bb195c59f4
[2017-06-14 20:11:48,015: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4d5ad05c-c59e-4322-8f64-28bb195c59f4
[2017-06-14 20:11:48,020: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4d5ad05c-c59e-4322-8f64-28bb195c59f4
[2017-06-14 20:11:48,020: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4d5ad05c-c59e-4322-8f64-28bb195c59f4
[2017-06-14 20:11:48,022: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com import_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5', 'pacemaker_ha_operation': False, 'device_type': 'zfs', 'validate_importable': False}
[2017-06-14 20:11:48,054: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 343ce938-3ac4-459c-9ae7-53f7a26a2116
[2017-06-14 20:11:48,054: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 343ce938-3ac4-459c-9ae7-53f7a26a2116
[2017-06-14 20:11:48,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:11:48,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:11:48,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:11:48,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:11:48,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4a57984b-166d-4237-a214-7c65e4499bcb
[2017-06-14 20:11:48,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4a57984b-166d-4237-a214-7c65e4499bcb
[2017-06-14 20:11:48,130: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4a57984b-166d-4237-a214-7c65e4499bcb
[2017-06-14 20:11:48,130: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4a57984b-166d-4237-a214-7c65e4499bcb
[2017-06-14 20:11:48,214: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:11:48,214: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:11:48,215: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:11:48,215: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:11:48,215: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0226cd8b-3114-4927-b1a7-3257ce7586b8
[2017-06-14 20:11:48,215: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0226cd8b-3114-4927-b1a7-3257ce7586b8
[2017-06-14 20:11:48,216: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:11:48,216: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:11:48,216: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:11:48,216: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:11:48,216: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 750544e8-eee2-460f-9edc-efac448c4318
[2017-06-14 20:11:48,216: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 750544e8-eee2-460f-9edc-efac448c4318
[2017-06-14 20:11:48,219: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0226cd8b-3114-4927-b1a7-3257ce7586b8
[2017-06-14 20:11:48,219: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0226cd8b-3114-4927-b1a7-3257ce7586b8
[2017-06-14 20:11:48,220: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com import_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3', 'pacemaker_ha_operation': False, 'device_type': 'zfs', 'validate_importable': False}
[2017-06-14 20:11:48,230: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 750544e8-eee2-460f-9edc-efac448c4318
[2017-06-14 20:11:48,230: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 750544e8-eee2-460f-9edc-efac448c4318
[2017-06-14 20:11:48,230: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com import_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1', 'pacemaker_ha_operation': False, 'device_type': 'zfs', 'validate_importable': False}
[2017-06-14 20:11:48,256: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c9f9e4ce-6e89-4852-a8f8-60be5193f0bd
[2017-06-14 20:11:48,256: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c9f9e4ce-6e89-4852-a8f8-60be5193f0bd
[2017-06-14 20:11:48,278: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 79cf9629-5f21-4509-b041-46c6783babad
[2017-06-14 20:11:48,278: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 79cf9629-5f21-4509-b041-46c6783babad
[2017-06-14 20:11:54,137: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:11:54,137: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:11:54,138: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:11:54,138: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:11:54,138: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 343ce938-3ac4-459c-9ae7-53f7a26a2116
[2017-06-14 20:11:54,138: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 343ce938-3ac4-459c-9ae7-53f7a26a2116
[2017-06-14 20:11:54,174: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 343ce938-3ac4-459c-9ae7-53f7a26a2116
[2017-06-14 20:11:54,174: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 343ce938-3ac4-459c-9ae7-53f7a26a2116
[2017-06-14 20:11:54,189: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com check_block_device {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5', 'device_type': 'zfs'}
[2017-06-14 20:11:54,201: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc acacad14-f005-4151-befc-5a0ced1cc839
[2017-06-14 20:11:54,201: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc acacad14-f005-4151-befc-5a0ced1cc839
[2017-06-14 20:11:54,362: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:11:54,362: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:11:54,362: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:11:54,362: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:11:54,363: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c9f9e4ce-6e89-4852-a8f8-60be5193f0bd
[2017-06-14 20:11:54,363: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c9f9e4ce-6e89-4852-a8f8-60be5193f0bd
[2017-06-14 20:11:54,385: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c9f9e4ce-6e89-4852-a8f8-60be5193f0bd
[2017-06-14 20:11:54,385: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c9f9e4ce-6e89-4852-a8f8-60be5193f0bd
[2017-06-14 20:11:54,386: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com check_block_device {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3', 'device_type': 'zfs'}
[2017-06-14 20:11:54,405: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 58cfbdb4-21a9-4078-96fc-efd0cd4c906a
[2017-06-14 20:11:54,405: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 58cfbdb4-21a9-4078-96fc-efd0cd4c906a
[2017-06-14 20:11:55,249: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:11:55,249: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:11:55,249: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:11:55,249: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:11:55,249: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc acacad14-f005-4151-befc-5a0ced1cc839
[2017-06-14 20:11:55,249: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc acacad14-f005-4151-befc-5a0ced1cc839
[2017-06-14 20:11:55,272: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc acacad14-f005-4151-befc-5a0ced1cc839
[2017-06-14 20:11:55,272: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc acacad14-f005-4151-befc-5a0ced1cc839
[2017-06-14 20:11:55,273: INFO/job] testfs-OST0000 passed pre-format check, allowing subsequent reformats
[2017-06-14 20:11:55,300: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com format_target {'index': 0, 'backfstype': 'zfs', 'target_name': u'testfs-OST0000', 'mgsnode': ((u'10.14.81.226@tcp0',), (u'10.14.81.227@tcp0',)), 'target_types': 'ost', 'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5', 'device_type': 'zfs', 'failnode': [u'10.14.81.225@tcp0'], 'fsname': u'testfs'}
[2017-06-14 20:11:55,324: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d64149ce-c4d5-4213-bb81-162177277881
[2017-06-14 20:11:55,324: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d64149ce-c4d5-4213-bb81-162177277881
[2017-06-14 20:11:55,369: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:11:55,369: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:11:55,369: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:11:55,369: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:11:55,369: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 79cf9629-5f21-4509-b041-46c6783babad
[2017-06-14 20:11:55,369: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 79cf9629-5f21-4509-b041-46c6783babad
[2017-06-14 20:11:55,397: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 79cf9629-5f21-4509-b041-46c6783babad
[2017-06-14 20:11:55,397: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 79cf9629-5f21-4509-b041-46c6783babad
[2017-06-14 20:11:55,398: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com check_block_device {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1', 'device_type': 'zfs'}
[2017-06-14 20:11:55,414: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 526182a7-6456-4886-b4e3-4ca33dfe685f
[2017-06-14 20:11:55,414: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 526182a7-6456-4886-b4e3-4ca33dfe685f
[2017-06-14 20:11:55,496: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:11:55,496: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:11:55,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:11:55,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:11:55,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 526182a7-6456-4886-b4e3-4ca33dfe685f
[2017-06-14 20:11:55,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 526182a7-6456-4886-b4e3-4ca33dfe685f
[2017-06-14 20:11:55,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:11:55,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:11:55,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:11:55,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:11:55,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 58cfbdb4-21a9-4078-96fc-efd0cd4c906a
[2017-06-14 20:11:55,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 58cfbdb4-21a9-4078-96fc-efd0cd4c906a
[2017-06-14 20:11:55,535: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 58cfbdb4-21a9-4078-96fc-efd0cd4c906a
[2017-06-14 20:11:55,535: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 58cfbdb4-21a9-4078-96fc-efd0cd4c906a
[2017-06-14 20:11:55,536: INFO/job] testfs-MDT0000 passed pre-format check, allowing subsequent reformats
[2017-06-14 20:11:55,537: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 526182a7-6456-4886-b4e3-4ca33dfe685f
[2017-06-14 20:11:55,537: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 526182a7-6456-4886-b4e3-4ca33dfe685f
[2017-06-14 20:11:55,538: INFO/job] MGS passed pre-format check, allowing subsequent reformats
[2017-06-14 20:11:55,576: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com format_target {'index': 0, 'backfstype': 'zfs', 'target_name': u'testfs-MDT0000', 'mgsnode': ((u'10.14.81.226@tcp0',), (u'10.14.81.227@tcp0',)), 'target_types': 'mdt', 'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3', 'device_type': 'zfs', 'failnode': [u'10.14.81.226@tcp0'], 'fsname': u'testfs'}
[2017-06-14 20:11:55,590: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4ac1488f-40cd-429d-851e-2f8b4a348b64
[2017-06-14 20:11:55,590: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4ac1488f-40cd-429d-851e-2f8b4a348b64
[2017-06-14 20:11:55,608: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com format_target {'backfstype': 'zfs', 'target_name': u'MGS', 'target_types': 'mgs', 'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1', 'device_type': 'zfs', 'failnode': [u'10.14.81.227@tcp0']}
[2017-06-14 20:11:55,625: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6551b334-4041-444f-87b1-09d11248f966
[2017-06-14 20:11:55,625: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6551b334-4041-444f-87b1-09d11248f966
[2017-06-14 20:12:00,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:12:00,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:12:00,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:12:00,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:12:00,473: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d64149ce-c4d5-4213-bb81-162177277881
[2017-06-14 20:12:00,473: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d64149ce-c4d5-4213-bb81-162177277881
[2017-06-14 20:12:00,490: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d64149ce-c4d5-4213-bb81-162177277881
[2017-06-14 20:12:00,490: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d64149ce-c4d5-4213-bb81-162177277881
[2017-06-14 20:12:00,577: INFO/job] Updating mtm volume_nodes for target testfs-OST0000
[2017-06-14 20:12:04,111: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:12:04,111: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:12:04,111: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:12:04,111: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:12:04,112: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4ac1488f-40cd-429d-851e-2f8b4a348b64
[2017-06-14 20:12:04,112: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4ac1488f-40cd-429d-851e-2f8b4a348b64
[2017-06-14 20:12:04,112: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4ac1488f-40cd-429d-851e-2f8b4a348b64
[2017-06-14 20:12:04,112: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4ac1488f-40cd-429d-851e-2f8b4a348b64
[2017-06-14 20:12:04,152: INFO/job] Updating mtm volume_nodes for target testfs-MDT0000
[2017-06-14 20:12:05,280: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:12:05,280: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:12:05,281: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:12:05,281: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:12:05,281: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6551b334-4041-444f-87b1-09d11248f966
[2017-06-14 20:12:05,281: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6551b334-4041-444f-87b1-09d11248f966
[2017-06-14 20:12:05,312: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6551b334-4041-444f-87b1-09d11248f966
[2017-06-14 20:12:05,312: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6551b334-4041-444f-87b1-09d11248f966
[2017-06-14 20:12:05,349: INFO/job] Updating mtm volume_nodes for target MGS
[2017-06-14 20:12:43,996: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-14 20:12:43,996: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-14 20:12:44,011: INFO/command_plan] add_jobs: created Job 79 (Trigger plugin poll for linux plugins)
[2017-06-14 20:12:44,053: INFO/job_scheduler] run_next: 1 runnable jobs of (11 pending, 3 tasked)
[2017-06-14 20:12:44,069: INFO/job_scheduler] Job 79: RunJobThread.run
[2017-06-14 20:12:44,082: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-14 20:12:44,108: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-14 20:12:44,114: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 38fe72fd-061e-4f15-a109-e475bdc8a419
[2017-06-14 20:12:44,114: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 38fe72fd-061e-4f15-a109-e475bdc8a419
[2017-06-14 20:12:44,137: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4c95100b-8424-4b90-a7d7-401ebca3fcd5
[2017-06-14 20:12:44,137: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4c95100b-8424-4b90-a7d7-401ebca3fcd5
[2017-06-14 20:12:44,139: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-14 20:12:44,167: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2298fa4f-acbe-4f67-9619-27c85817ac3b
[2017-06-14 20:12:44,167: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2298fa4f-acbe-4f67-9619-27c85817ac3b
[2017-06-14 20:12:44,188: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:12:44,188: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:12:44,188: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:12:44,188: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:12:44,188: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 38fe72fd-061e-4f15-a109-e475bdc8a419
[2017-06-14 20:12:44,188: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 38fe72fd-061e-4f15-a109-e475bdc8a419
[2017-06-14 20:12:44,195: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 38fe72fd-061e-4f15-a109-e475bdc8a419
[2017-06-14 20:12:44,195: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 38fe72fd-061e-4f15-a109-e475bdc8a419
[2017-06-14 20:12:44,196: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-14 20:12:44,216: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:12:44,216: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:12:44,216: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:12:44,216: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:12:44,216: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2298fa4f-acbe-4f67-9619-27c85817ac3b
[2017-06-14 20:12:44,216: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2298fa4f-acbe-4f67-9619-27c85817ac3b
[2017-06-14 20:12:44,219: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:12:44,219: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:12:44,219: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:12:44,219: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:12:44,219: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4c95100b-8424-4b90-a7d7-401ebca3fcd5
[2017-06-14 20:12:44,219: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4c95100b-8424-4b90-a7d7-401ebca3fcd5
[2017-06-14 20:12:44,230: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4c95100b-8424-4b90-a7d7-401ebca3fcd5
[2017-06-14 20:12:44,230: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4c95100b-8424-4b90-a7d7-401ebca3fcd5
[2017-06-14 20:12:44,231: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-14 20:12:44,231: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2298fa4f-acbe-4f67-9619-27c85817ac3b
[2017-06-14 20:12:44,231: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2298fa4f-acbe-4f67-9619-27c85817ac3b
[2017-06-14 20:12:44,232: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-14 20:12:44,232: INFO/job_scheduler] Job 79 finished 1 steps successfully
[2017-06-14 20:12:44,269: INFO/job_scheduler] Job 79: complete_job: Updating cache
[2017-06-14 20:12:44,270: INFO/job_scheduler] Job 79 completing (errored=False, cancelled=False)
[2017-06-14 20:12:44,331: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 3 tasked)
[2017-06-14 20:12:45,711: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-14 20:12:45,711: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-14 20:12:45,746: INFO/command_plan] add_jobs: created Job 80 (Trigger plugin poll for linux plugins)
[2017-06-14 20:12:45,757: INFO/job_scheduler] run_next: 1 runnable jobs of (11 pending, 3 tasked)
[2017-06-14 20:12:45,776: INFO/job_scheduler] Job 80: RunJobThread.run
[2017-06-14 20:12:45,777: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-14 20:12:45,795: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 375d5c71-1667-4e21-86aa-ec7cc6977586
[2017-06-14 20:12:45,795: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 375d5c71-1667-4e21-86aa-ec7cc6977586
[2017-06-14 20:12:45,797: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-14 20:12:45,821: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-14 20:12:45,829: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1fb16de0-fe33-4776-8ea6-72ab9eab4a39
[2017-06-14 20:12:45,829: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1fb16de0-fe33-4776-8ea6-72ab9eab4a39
[2017-06-14 20:12:45,875: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5c51ff5a-44ca-45c9-a0ad-1af0db605ede
[2017-06-14 20:12:45,875: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5c51ff5a-44ca-45c9-a0ad-1af0db605ede
[2017-06-14 20:12:45,875: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:12:45,875: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:12:45,876: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:12:45,876: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:12:45,876: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 375d5c71-1667-4e21-86aa-ec7cc6977586
[2017-06-14 20:12:45,876: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 375d5c71-1667-4e21-86aa-ec7cc6977586
[2017-06-14 20:12:45,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:12:45,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:12:45,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:12:45,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:12:45,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1fb16de0-fe33-4776-8ea6-72ab9eab4a39
[2017-06-14 20:12:45,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1fb16de0-fe33-4776-8ea6-72ab9eab4a39
[2017-06-14 20:12:45,923: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 375d5c71-1667-4e21-86aa-ec7cc6977586
[2017-06-14 20:12:45,923: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 375d5c71-1667-4e21-86aa-ec7cc6977586
[2017-06-14 20:12:45,927: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-14 20:12:45,954: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1fb16de0-fe33-4776-8ea6-72ab9eab4a39
[2017-06-14 20:12:45,954: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1fb16de0-fe33-4776-8ea6-72ab9eab4a39
[2017-06-14 20:12:45,954: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-14 20:12:45,979: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:12:45,979: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:12:45,979: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:12:45,979: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:12:45,980: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5c51ff5a-44ca-45c9-a0ad-1af0db605ede
[2017-06-14 20:12:45,980: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5c51ff5a-44ca-45c9-a0ad-1af0db605ede
[2017-06-14 20:12:45,997: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5c51ff5a-44ca-45c9-a0ad-1af0db605ede
[2017-06-14 20:12:45,997: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5c51ff5a-44ca-45c9-a0ad-1af0db605ede
[2017-06-14 20:12:45,998: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-14 20:12:45,998: INFO/job_scheduler] Job 80 finished 1 steps successfully
[2017-06-14 20:12:46,120: INFO/job_scheduler] Job 80: complete_job: Updating cache
[2017-06-14 20:12:46,121: INFO/job_scheduler] Job 80 completing (errored=False, cancelled=False)
[2017-06-14 20:12:46,198: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 3 tasked)
[2017-06-14 20:12:47,129: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-14 20:12:47,130: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-14 20:12:47,159: INFO/command_plan] add_jobs: created Job 81 (Trigger plugin poll for linux plugins)
[2017-06-14 20:12:47,183: INFO/job_scheduler] run_next: 1 runnable jobs of (11 pending, 3 tasked)
[2017-06-14 20:12:47,201: INFO/job_scheduler] Job 81: RunJobThread.run
[2017-06-14 20:12:47,238: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-14 20:12:47,278: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d57048e9-b013-4a45-b35a-849640041005
[2017-06-14 20:12:47,278: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d57048e9-b013-4a45-b35a-849640041005
[2017-06-14 20:12:47,279: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-14 20:12:47,304: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2cbae96e-08b6-4398-9b6d-f2b0872cc0d2
[2017-06-14 20:12:47,304: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2cbae96e-08b6-4398-9b6d-f2b0872cc0d2
[2017-06-14 20:12:47,354: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-14 20:12:47,433: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc afca0d7c-0346-438a-869a-23c2f67fe36a
[2017-06-14 20:12:47,433: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc afca0d7c-0346-438a-869a-23c2f67fe36a
[2017-06-14 20:12:47,434: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:12:47,434: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:12:47,434: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:12:47,434: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:12:47,434: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d57048e9-b013-4a45-b35a-849640041005
[2017-06-14 20:12:47,434: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d57048e9-b013-4a45-b35a-849640041005
[2017-06-14 20:12:47,435: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:12:47,435: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:12:47,435: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:12:47,435: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:12:47,435: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2cbae96e-08b6-4398-9b6d-f2b0872cc0d2
[2017-06-14 20:12:47,435: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2cbae96e-08b6-4398-9b6d-f2b0872cc0d2
[2017-06-14 20:12:47,464: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d57048e9-b013-4a45-b35a-849640041005
[2017-06-14 20:12:47,464: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d57048e9-b013-4a45-b35a-849640041005
[2017-06-14 20:12:47,464: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-14 20:12:47,474: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2cbae96e-08b6-4398-9b6d-f2b0872cc0d2
[2017-06-14 20:12:47,474: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2cbae96e-08b6-4398-9b6d-f2b0872cc0d2
[2017-06-14 20:12:47,474: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-14 20:12:47,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:12:47,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:12:47,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:12:47,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:12:47,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc afca0d7c-0346-438a-869a-23c2f67fe36a
[2017-06-14 20:12:47,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc afca0d7c-0346-438a-869a-23c2f67fe36a
[2017-06-14 20:12:47,502: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc afca0d7c-0346-438a-869a-23c2f67fe36a
[2017-06-14 20:12:47,502: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc afca0d7c-0346-438a-869a-23c2f67fe36a
[2017-06-14 20:12:47,502: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-14 20:12:47,504: INFO/job_scheduler] Job 81 finished 1 steps successfully
[2017-06-14 20:12:47,567: INFO/job_scheduler] Job 81: complete_job: Updating cache
[2017-06-14 20:12:47,568: INFO/job_scheduler] Job 81 completing (errored=False, cancelled=False)
[2017-06-14 20:12:47,691: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 3 tasked)
[2017-06-14 20:12:48,451: INFO/job_scheduler] Job 67 finished 5 steps successfully
[2017-06-14 20:12:48,484: INFO/job] StatefulObject.set_state testfs-MDT0000 unformatted->formatted (intentional=True) 140623739419024
[2017-06-14 20:12:48,493: INFO/job] Job 67: StateChangeJob complete, setting state formatted on testfs-MDT0000
[2017-06-14 20:12:48,493: INFO/job] Job 67: not_deleted=True
[2017-06-14 20:12:48,500: INFO/job_scheduler] Job 67: complete_job: Updating cache
[2017-06-14 20:12:48,500: INFO/job_scheduler] Job 67: locked_item 140623739419024 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-14 20:12:48,503: INFO/job_scheduler] Job 67: locked_item 140624142358160 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-06-14 20:12:48,504: INFO/job_scheduler] Job 67: locked_item 140624142358352 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-06-14 20:12:48,506: INFO/job_scheduler] Job 67 completing (errored=False, cancelled=False)
[2017-06-14 20:12:48,534: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 2 tasked)
[2017-06-14 20:12:48,694: INFO/job_scheduler] Job 68 finished 5 steps successfully
[2017-06-14 20:12:48,718: INFO/job] StatefulObject.set_state MGS unformatted->formatted (intentional=True) 140624142344272
[2017-06-14 20:12:48,724: INFO/job] Job 68: StateChangeJob complete, setting state formatted on MGS
[2017-06-14 20:12:48,724: INFO/job] Job 68: not_deleted=True
[2017-06-14 20:12:48,737: INFO/job_scheduler] Job 68: complete_job: Updating cache
[2017-06-14 20:12:48,737: INFO/job_scheduler] Job 68: locked_item 140624142344272 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-14 20:12:48,740: INFO/job_scheduler] Job 68: locked_item 140624142359120 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-06-14 20:12:48,745: INFO/job_scheduler] Job 68: locked_item 140624142359568 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-06-14 20:12:48,749: INFO/job_scheduler] Job 68 completing (errored=False, cancelled=False)
[2017-06-14 20:12:48,810: INFO/job_scheduler] run_next: 1 runnable jobs of (10 pending, 1 tasked)
[2017-06-14 20:12:48,845: INFO/job_scheduler] Job 69: RunJobThread.run
[2017-06-14 20:12:48,847: INFO/job_scheduler] Job 69 finished 1 steps successfully
[2017-06-14 20:12:48,884: INFO/job] StatefulObject.set_state MGS formatted->registered (intentional=True) 140624142344272
[2017-06-14 20:12:48,890: INFO/job] Job 69: StateChangeJob complete, setting state registered on MGS
[2017-06-14 20:12:48,891: INFO/job] Job 69: not_deleted=True
[2017-06-14 20:12:48,891: INFO/job_scheduler] Job 69: complete_job: Updating cache
[2017-06-14 20:12:48,891: INFO/job_scheduler] Job 69: locked_item 140624142344272 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-14 20:12:48,893: INFO/job_scheduler] Job 69 completing (errored=False, cancelled=False)
[2017-06-14 20:12:48,960: INFO/job_scheduler] run_next: 1 runnable jobs of (9 pending, 1 tasked)
[2017-06-14 20:12:49,044: INFO/job_scheduler] Job 70: RunJobThread.run
[2017-06-14 20:12:49,058: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-06-14 20:12:49,091: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c430f85a-1362-4e42-9660-19d4dfc847e4
[2017-06-14 20:12:49,091: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c430f85a-1362-4e42-9660-19d4dfc847e4
[2017-06-14 20:12:50,224: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:12:50,224: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:12:50,224: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:12:50,224: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:12:50,224: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c430f85a-1362-4e42-9660-19d4dfc847e4
[2017-06-14 20:12:50,224: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c430f85a-1362-4e42-9660-19d4dfc847e4
[2017-06-14 20:12:50,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c430f85a-1362-4e42-9660-19d4dfc847e4
[2017-06-14 20:12:50,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c430f85a-1362-4e42-9660-19d4dfc847e4
[2017-06-14 20:12:50,256: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-06-14 20:12:50,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 604862ad-0d9f-4051-97c6-2f6a8a51c5d6
[2017-06-14 20:12:50,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 604862ad-0d9f-4051-97c6-2f6a8a51c5d6
[2017-06-14 20:12:51,325: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:12:51,325: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:12:51,325: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:12:51,325: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:12:51,325: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 604862ad-0d9f-4051-97c6-2f6a8a51c5d6
[2017-06-14 20:12:51,325: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 604862ad-0d9f-4051-97c6-2f6a8a51c5d6
[2017-06-14 20:12:51,337: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 604862ad-0d9f-4051-97c6-2f6a8a51c5d6
[2017-06-14 20:12:51,337: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 604862ad-0d9f-4051-97c6-2f6a8a51c5d6
[2017-06-14 20:12:51,339: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_target_store {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/MGS', 'device_type': 'zfs', 'mount_point': u'/mnt/MGS', 'uuid': u'18405115617293209603', 'backfstype': 'zfs'}
[2017-06-14 20:12:51,351: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2e664ea1-225f-4f87-8086-d02f66b9af3b
[2017-06-14 20:12:51,351: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2e664ea1-225f-4f87-8086-d02f66b9af3b
[2017-06-14 20:12:51,454: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:12:51,454: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:12:51,455: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:12:51,455: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:12:51,455: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2e664ea1-225f-4f87-8086-d02f66b9af3b
[2017-06-14 20:12:51,455: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2e664ea1-225f-4f87-8086-d02f66b9af3b
[2017-06-14 20:12:51,470: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2e664ea1-225f-4f87-8086-d02f66b9af3b
[2017-06-14 20:12:51,470: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2e664ea1-225f-4f87-8086-d02f66b9af3b
[2017-06-14 20:12:51,472: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_target_store {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/MGS', 'device_type': 'zfs', 'mount_point': u'/mnt/MGS', 'uuid': u'18405115617293209603', 'backfstype': 'zfs'}
[2017-06-14 20:12:51,486: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ae952cf4-9f46-4e2a-90e7-4bf3a2044db5
[2017-06-14 20:12:51,486: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ae952cf4-9f46-4e2a-90e7-4bf3a2044db5
[2017-06-14 20:12:51,536: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:12:51,536: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:12:51,537: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:12:51,537: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:12:51,537: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ae952cf4-9f46-4e2a-90e7-4bf3a2044db5
[2017-06-14 20:12:51,537: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ae952cf4-9f46-4e2a-90e7-4bf3a2044db5
[2017-06-14 20:12:51,558: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ae952cf4-9f46-4e2a-90e7-4bf3a2044db5
[2017-06-14 20:12:51,558: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ae952cf4-9f46-4e2a-90e7-4bf3a2044db5
[2017-06-14 20:12:51,559: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/MGS', 'mount_point': u'/mnt/MGS', 'uuid': u'18405115617293209603', 'ha_label': u'MGS_9e39ca', 'primary': True}
[2017-06-14 20:12:51,575: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9f9f63ba-0d7c-4888-8745-742900bfd277
[2017-06-14 20:12:51,575: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9f9f63ba-0d7c-4888-8745-742900bfd277
[2017-06-14 20:12:55,016: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:12:55,016: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:12:55,017: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:12:55,017: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:12:55,017: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9f9f63ba-0d7c-4888-8745-742900bfd277
[2017-06-14 20:12:55,017: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9f9f63ba-0d7c-4888-8745-742900bfd277
[2017-06-14 20:12:55,040: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9f9f63ba-0d7c-4888-8745-742900bfd277
[2017-06-14 20:12:55,040: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9f9f63ba-0d7c-4888-8745-742900bfd277
[2017-06-14 20:12:55,042: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/MGS', 'mount_point': u'/mnt/MGS', 'uuid': u'18405115617293209603', 'ha_label': u'MGS_9e39ca', 'primary': False}
[2017-06-14 20:12:55,055: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cb3c1d57-d87f-49e0-acd3-60e5786b0f7f
[2017-06-14 20:12:55,055: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cb3c1d57-d87f-49e0-acd3-60e5786b0f7f
[2017-06-14 20:12:56,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:12:56,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:12:56,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:12:56,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:12:56,190: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cb3c1d57-d87f-49e0-acd3-60e5786b0f7f
[2017-06-14 20:12:56,190: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cb3c1d57-d87f-49e0-acd3-60e5786b0f7f
[2017-06-14 20:12:56,222: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cb3c1d57-d87f-49e0-acd3-60e5786b0f7f
[2017-06-14 20:12:56,222: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cb3c1d57-d87f-49e0-acd3-60e5786b0f7f
[2017-06-14 20:12:56,223: INFO/job_scheduler] Job 70 finished 6 steps successfully
[2017-06-14 20:12:56,274: INFO/job] StatefulObject.set_state MGS registered->unmounted (intentional=True) 140624142344272
[2017-06-14 20:12:56,279: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on MGS at severity 30
[2017-06-14 20:12:56,292: INFO/job] Job 70: StateChangeJob complete, setting state unmounted on MGS
[2017-06-14 20:12:56,292: INFO/job] Job 70: not_deleted=True
[2017-06-14 20:12:56,297: INFO/job_scheduler] Job 70: complete_job: Updating cache
[2017-06-14 20:12:56,298: INFO/job_scheduler] Job 70: locked_item 140624142344272 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-14 20:12:56,303: INFO/job_scheduler] Job 70 completing (errored=False, cancelled=False)
[2017-06-14 20:12:56,332: INFO/job_scheduler] run_next: 1 runnable jobs of (8 pending, 1 tasked)
[2017-06-14 20:12:56,409: INFO/job] create_parameters: host: 'lotus-32vm7.lotus.hpdd.lab.intel.com' active_volume_node: 'TargetVolumeInfo(host=<ManagedHost: lotus-32vm7.lotus.hpdd.lab.intel.com>, path=u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/MGS', device_type='zfs')'
[2017-06-14 20:12:56,428: INFO/job_scheduler] Job 71: RunJobThread.run
[2017-06-14 20:12:56,441: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/MGS', 'device_type': 'zfs'}
[2017-06-14 20:12:56,465: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/MGS', 'device_type': 'zfs'}
[2017-06-14 20:12:56,470: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc aaa7bb74-25b2-47f1-9d80-e6edc55ad4a2
[2017-06-14 20:12:56,470: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc aaa7bb74-25b2-47f1-9d80-e6edc55ad4a2
[2017-06-14 20:12:56,486: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cb27480d-1730-48d9-a1ef-9dcd883ab7bb
[2017-06-14 20:12:56,486: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cb27480d-1730-48d9-a1ef-9dcd883ab7bb
[2017-06-14 20:12:57,591: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:12:57,591: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:12:57,591: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:12:57,591: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:12:57,592: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cb27480d-1730-48d9-a1ef-9dcd883ab7bb
[2017-06-14 20:12:57,592: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cb27480d-1730-48d9-a1ef-9dcd883ab7bb
[2017-06-14 20:12:57,601: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cb27480d-1730-48d9-a1ef-9dcd883ab7bb
[2017-06-14 20:12:57,601: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cb27480d-1730-48d9-a1ef-9dcd883ab7bb
[2017-06-14 20:12:58,587: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:12:58,587: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:12:58,587: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:12:58,587: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:12:58,587: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc aaa7bb74-25b2-47f1-9d80-e6edc55ad4a2
[2017-06-14 20:12:58,587: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc aaa7bb74-25b2-47f1-9d80-e6edc55ad4a2
[2017-06-14 20:12:58,635: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc aaa7bb74-25b2-47f1-9d80-e6edc55ad4a2
[2017-06-14 20:12:58,635: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc aaa7bb74-25b2-47f1-9d80-e6edc55ad4a2
[2017-06-14 20:12:58,635: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com import_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/MGS', 'pacemaker_ha_operation': False, 'device_type': 'zfs', 'validate_importable': True}
[2017-06-14 20:12:58,690: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0b462bb7-34a2-47dc-a861-90a91a27ee82
[2017-06-14 20:12:58,690: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0b462bb7-34a2-47dc-a861-90a91a27ee82
[2017-06-14 20:12:59,975: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 2 tasked)
[2017-06-14 20:13:02,368: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 2 tasked)
[2017-06-14 20:13:02,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:13:02,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:13:02,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:13:02,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:13:02,782: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0b462bb7-34a2-47dc-a861-90a91a27ee82
[2017-06-14 20:13:02,782: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0b462bb7-34a2-47dc-a861-90a91a27ee82
[2017-06-14 20:13:02,809: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0b462bb7-34a2-47dc-a861-90a91a27ee82
[2017-06-14 20:13:02,809: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0b462bb7-34a2-47dc-a861-90a91a27ee82
[2017-06-14 20:13:02,810: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_target {'ha_label': u'MGS_9e39ca'}
[2017-06-14 20:13:02,833: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2a16ffed-fb10-41e1-a5fc-93d0bd1a303c
[2017-06-14 20:13:02,833: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2a16ffed-fb10-41e1-a5fc-93d0bd1a303c
[2017-06-14 20:13:11,377: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 2 tasked)
[2017-06-14 20:13:16,992: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:13:16,992: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:13:16,992: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:13:16,992: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:13:16,992: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2a16ffed-fb10-41e1-a5fc-93d0bd1a303c
[2017-06-14 20:13:16,992: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2a16ffed-fb10-41e1-a5fc-93d0bd1a303c
[2017-06-14 20:13:17,013: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2a16ffed-fb10-41e1-a5fc-93d0bd1a303c
[2017-06-14 20:13:17,013: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2a16ffed-fb10-41e1-a5fc-93d0bd1a303c
[2017-06-14 20:13:17,019: INFO/job_scheduler] Job 71 finished 1 steps successfully
[2017-06-14 20:13:17,292: INFO/job] StatefulObject.set_state MGS unmounted->mounted (intentional=True) 140624142344272
[2017-06-14 20:13:17,328: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 20
[2017-06-14 20:13:17,336: INFO/job] Job 71: StateChangeJob complete, setting state mounted on MGS
[2017-06-14 20:13:17,336: INFO/job] Job 71: not_deleted=True
[2017-06-14 20:13:17,336: INFO/job_scheduler] Job 71: complete_job: Updating cache
[2017-06-14 20:13:17,336: INFO/job_scheduler] Job 71: locked_item 140624142344272 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-14 20:13:17,339: INFO/job_scheduler] Job 71 completing (errored=False, cancelled=False)
[2017-06-14 20:13:17,376: INFO/job_scheduler] run_next: 1 runnable jobs of (7 pending, 1 tasked)
[2017-06-14 20:13:17,467: INFO/job_scheduler] Job 72: RunJobThread.run
[2017-06-14 20:13:17,469: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com register_target {'device_path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3/testfs-MDT0000', 'mount_point': u'/mnt/testfs-MDT0000', 'backfstype': 'zfs'}
[2017-06-14 20:13:17,507: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5f4b98cf-ee83-437a-afcd-f50170827b92
[2017-06-14 20:13:17,507: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5f4b98cf-ee83-437a-afcd-f50170827b92
[2017-06-14 20:13:20,019: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 2 tasked)
[2017-06-14 20:13:23,395: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 2 tasked)
[2017-06-14 20:13:23,727: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 2 tasked)
[2017-06-14 20:13:23,742: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 2 tasked)
[2017-06-14 20:13:28,729: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 2 tasked)
[2017-06-14 20:13:28,740: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 2 tasked)
[2017-06-14 20:13:29,644: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:13:29,644: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:13:29,645: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:13:29,645: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:13:29,645: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5f4b98cf-ee83-437a-afcd-f50170827b92
[2017-06-14 20:13:29,645: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5f4b98cf-ee83-437a-afcd-f50170827b92
[2017-06-14 20:13:29,684: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5f4b98cf-ee83-437a-afcd-f50170827b92
[2017-06-14 20:13:29,684: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5f4b98cf-ee83-437a-afcd-f50170827b92
[2017-06-14 20:13:29,686: INFO/job_scheduler] Job 72 finished 2 steps successfully
[2017-06-14 20:13:29,856: INFO/job] StatefulObject.set_state testfs-MDT0000 formatted->registered (intentional=True) 140623739419024
[2017-06-14 20:13:29,861: INFO/job] Job 72: StateChangeJob complete, setting state registered on testfs-MDT0000
[2017-06-14 20:13:29,861: INFO/job] Job 72: not_deleted=True
[2017-06-14 20:13:29,861: INFO/job_scheduler] Job 72: complete_job: Updating cache
[2017-06-14 20:13:29,861: INFO/job_scheduler] Job 72: locked_item 140623739419024 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-14 20:13:29,864: INFO/job_scheduler] Job 72 completing (errored=False, cancelled=False)
[2017-06-14 20:13:29,890: INFO/job_scheduler] run_next: 1 runnable jobs of (6 pending, 1 tasked)
[2017-06-14 20:13:30,013: INFO/job_scheduler] Job 73: RunJobThread.run
[2017-06-14 20:13:30,014: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-06-14 20:13:30,059: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 509d9cdc-43b9-4ba8-87f3-6937208f681b
[2017-06-14 20:13:30,059: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 509d9cdc-43b9-4ba8-87f3-6937208f681b
[2017-06-14 20:13:31,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:13:31,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:13:31,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:13:31,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:13:31,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 509d9cdc-43b9-4ba8-87f3-6937208f681b
[2017-06-14 20:13:31,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 509d9cdc-43b9-4ba8-87f3-6937208f681b
[2017-06-14 20:13:31,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 509d9cdc-43b9-4ba8-87f3-6937208f681b
[2017-06-14 20:13:31,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 509d9cdc-43b9-4ba8-87f3-6937208f681b
[2017-06-14 20:13:31,124: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-06-14 20:13:31,138: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2bed7012-d370-476c-a376-071d6d0fb41c
[2017-06-14 20:13:31,138: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2bed7012-d370-476c-a376-071d6d0fb41c
[2017-06-14 20:13:32,225: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:13:32,225: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:13:32,225: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:13:32,225: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:13:32,225: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2bed7012-d370-476c-a376-071d6d0fb41c
[2017-06-14 20:13:32,225: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2bed7012-d370-476c-a376-071d6d0fb41c
[2017-06-14 20:13:32,256: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2bed7012-d370-476c-a376-071d6d0fb41c
[2017-06-14 20:13:32,256: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2bed7012-d370-476c-a376-071d6d0fb41c
[2017-06-14 20:13:32,258: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_target_store {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3/testfs-MDT0000', 'device_type': 'zfs', 'mount_point': u'/mnt/testfs-MDT0000', 'uuid': u'2967121584722494284', 'backfstype': 'zfs'}
[2017-06-14 20:13:32,272: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ec3ad5f2-36ad-4259-b09d-a4a9cbba8ec3
[2017-06-14 20:13:32,272: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ec3ad5f2-36ad-4259-b09d-a4a9cbba8ec3
[2017-06-14 20:13:32,370: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:13:32,370: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:13:32,371: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:13:32,371: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:13:32,371: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ec3ad5f2-36ad-4259-b09d-a4a9cbba8ec3
[2017-06-14 20:13:32,371: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ec3ad5f2-36ad-4259-b09d-a4a9cbba8ec3
[2017-06-14 20:13:32,387: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ec3ad5f2-36ad-4259-b09d-a4a9cbba8ec3
[2017-06-14 20:13:32,387: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ec3ad5f2-36ad-4259-b09d-a4a9cbba8ec3
[2017-06-14 20:13:32,389: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_target_store {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3/testfs-MDT0000', 'device_type': 'zfs', 'mount_point': u'/mnt/testfs-MDT0000', 'uuid': u'2967121584722494284', 'backfstype': 'zfs'}
[2017-06-14 20:13:32,407: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0a115f55-4ddf-4030-8635-edffb43776c7
[2017-06-14 20:13:32,407: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0a115f55-4ddf-4030-8635-edffb43776c7
[2017-06-14 20:13:32,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:13:32,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:13:32,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:13:32,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:13:32,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0a115f55-4ddf-4030-8635-edffb43776c7
[2017-06-14 20:13:32,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0a115f55-4ddf-4030-8635-edffb43776c7
[2017-06-14 20:13:32,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0a115f55-4ddf-4030-8635-edffb43776c7
[2017-06-14 20:13:32,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0a115f55-4ddf-4030-8635-edffb43776c7
[2017-06-14 20:13:32,590: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3/testfs-MDT0000', 'mount_point': u'/mnt/testfs-MDT0000', 'uuid': u'2967121584722494284', 'ha_label': u'testfs-MDT0000_69ad68', 'primary': True}
[2017-06-14 20:13:32,603: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4240b992-55c8-4b48-912b-3fb4cb903fe5
[2017-06-14 20:13:32,603: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4240b992-55c8-4b48-912b-3fb4cb903fe5
[2017-06-14 20:13:35,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:13:35,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:13:35,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:13:35,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:13:35,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4240b992-55c8-4b48-912b-3fb4cb903fe5
[2017-06-14 20:13:35,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4240b992-55c8-4b48-912b-3fb4cb903fe5
[2017-06-14 20:13:35,724: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4240b992-55c8-4b48-912b-3fb4cb903fe5
[2017-06-14 20:13:35,724: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4240b992-55c8-4b48-912b-3fb4cb903fe5
[2017-06-14 20:13:35,726: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3/testfs-MDT0000', 'mount_point': u'/mnt/testfs-MDT0000', 'uuid': u'2967121584722494284', 'ha_label': u'testfs-MDT0000_69ad68', 'primary': False}
[2017-06-14 20:13:35,745: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bbeddb34-1b26-432d-8bba-de90cb67db10
[2017-06-14 20:13:35,745: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bbeddb34-1b26-432d-8bba-de90cb67db10
[2017-06-14 20:13:36,809: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:13:36,809: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:13:36,809: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:13:36,809: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:13:36,810: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bbeddb34-1b26-432d-8bba-de90cb67db10
[2017-06-14 20:13:36,810: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bbeddb34-1b26-432d-8bba-de90cb67db10
[2017-06-14 20:13:36,810: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bbeddb34-1b26-432d-8bba-de90cb67db10
[2017-06-14 20:13:36,810: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bbeddb34-1b26-432d-8bba-de90cb67db10
[2017-06-14 20:13:36,810: INFO/job_scheduler] Job 73 finished 6 steps successfully
[2017-06-14 20:13:36,872: INFO/job] StatefulObject.set_state testfs-MDT0000 registered->unmounted (intentional=True) 140623739419024
[2017-06-14 20:13:36,877: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on testfs-MDT0000 at severity 30
[2017-06-14 20:13:36,879: INFO/job] Job 73: StateChangeJob complete, setting state unmounted on testfs-MDT0000
[2017-06-14 20:13:36,879: INFO/job] Job 73: not_deleted=True
[2017-06-14 20:13:36,879: INFO/job_scheduler] Job 73: complete_job: Updating cache
[2017-06-14 20:13:36,879: INFO/job_scheduler] Job 73: locked_item 140623739419024 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-14 20:13:36,883: INFO/job_scheduler] Job 73 completing (errored=False, cancelled=False)
[2017-06-14 20:13:36,922: INFO/job_scheduler] run_next: 1 runnable jobs of (5 pending, 1 tasked)
[2017-06-14 20:13:37,009: INFO/job] create_parameters: host: 'lotus-32vm8.lotus.hpdd.lab.intel.com' active_volume_node: 'TargetVolumeInfo(host=<ManagedHost: lotus-32vm8.lotus.hpdd.lab.intel.com>, path=u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3/testfs-MDT0000', device_type='zfs')'
[2017-06-14 20:13:37,012: INFO/job_scheduler] Job 74: RunJobThread.run
[2017-06-14 20:13:37,028: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3/testfs-MDT0000', 'device_type': 'zfs'}
[2017-06-14 20:13:37,036: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3/testfs-MDT0000', 'device_type': 'zfs'}
[2017-06-14 20:13:37,046: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7e62f19c-14e1-4866-90f5-12d552ce38a2
[2017-06-14 20:13:37,046: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7e62f19c-14e1-4866-90f5-12d552ce38a2
[2017-06-14 20:13:37,064: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 19f7e5a1-17c7-4f20-adb6-a5ec6b8579d3
[2017-06-14 20:13:37,064: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 19f7e5a1-17c7-4f20-adb6-a5ec6b8579d3
[2017-06-14 20:13:37,136: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:13:37,136: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:13:37,136: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:13:37,136: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:13:37,136: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7e62f19c-14e1-4866-90f5-12d552ce38a2
[2017-06-14 20:13:37,136: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7e62f19c-14e1-4866-90f5-12d552ce38a2
[2017-06-14 20:13:37,173: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7e62f19c-14e1-4866-90f5-12d552ce38a2
[2017-06-14 20:13:37,173: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7e62f19c-14e1-4866-90f5-12d552ce38a2
[2017-06-14 20:13:39,165: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:13:39,165: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:13:39,166: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:13:39,166: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:13:39,166: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 19f7e5a1-17c7-4f20-adb6-a5ec6b8579d3
[2017-06-14 20:13:39,166: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 19f7e5a1-17c7-4f20-adb6-a5ec6b8579d3
[2017-06-14 20:13:39,182: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 19f7e5a1-17c7-4f20-adb6-a5ec6b8579d3
[2017-06-14 20:13:39,182: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 19f7e5a1-17c7-4f20-adb6-a5ec6b8579d3
[2017-06-14 20:13:39,183: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com import_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk3/testfs-MDT0000', 'pacemaker_ha_operation': False, 'device_type': 'zfs', 'validate_importable': True}
[2017-06-14 20:13:39,195: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b95f5d02-0c84-41d8-a255-b7192607cc7f
[2017-06-14 20:13:39,195: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b95f5d02-0c84-41d8-a255-b7192607cc7f
[2017-06-14 20:13:40,044: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-14 20:13:40,044: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-14 20:13:40,074: INFO/command_plan] add_jobs: created Job 82 (Trigger plugin poll for linux plugins)
[2017-06-14 20:13:40,109: INFO/job_scheduler] run_next: 1 runnable jobs of (5 pending, 2 tasked)
[2017-06-14 20:13:40,165: INFO/job_scheduler] Job 82: RunJobThread.run
[2017-06-14 20:13:40,166: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 3 tasked)
[2017-06-14 20:13:40,217: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-14 20:13:40,241: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 3 tasked)
[2017-06-14 20:13:40,258: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 56568e25-7ac8-40c1-a914-e7cb92b4b79e
[2017-06-14 20:13:40,258: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 56568e25-7ac8-40c1-a914-e7cb92b4b79e
[2017-06-14 20:13:40,249: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-14 20:13:40,286: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-14 20:13:40,304: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1b37fc49-25c5-43e7-9be6-8dbd1dae0d21
[2017-06-14 20:13:40,304: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1b37fc49-25c5-43e7-9be6-8dbd1dae0d21
[2017-06-14 20:13:40,335: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 419a1f3c-d803-4924-a552-a9b02091a5d7
[2017-06-14 20:13:40,335: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 419a1f3c-d803-4924-a552-a9b02091a5d7
[2017-06-14 20:13:40,352: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:13:40,352: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:13:40,352: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:13:40,352: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:13:40,352: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 56568e25-7ac8-40c1-a914-e7cb92b4b79e
[2017-06-14 20:13:40,352: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 56568e25-7ac8-40c1-a914-e7cb92b4b79e
[2017-06-14 20:13:40,365: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:13:40,365: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:13:40,365: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:13:40,365: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:13:40,365: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1b37fc49-25c5-43e7-9be6-8dbd1dae0d21
[2017-06-14 20:13:40,365: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1b37fc49-25c5-43e7-9be6-8dbd1dae0d21
[2017-06-14 20:13:40,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1b37fc49-25c5-43e7-9be6-8dbd1dae0d21
[2017-06-14 20:13:40,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1b37fc49-25c5-43e7-9be6-8dbd1dae0d21
[2017-06-14 20:13:40,400: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-14 20:13:40,403: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 56568e25-7ac8-40c1-a914-e7cb92b4b79e
[2017-06-14 20:13:40,403: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 56568e25-7ac8-40c1-a914-e7cb92b4b79e
[2017-06-14 20:13:40,404: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-14 20:13:40,434: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:13:40,434: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:13:40,434: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:13:40,434: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:13:40,434: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 419a1f3c-d803-4924-a552-a9b02091a5d7
[2017-06-14 20:13:40,434: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 419a1f3c-d803-4924-a552-a9b02091a5d7
[2017-06-14 20:13:40,459: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 419a1f3c-d803-4924-a552-a9b02091a5d7
[2017-06-14 20:13:40,459: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 419a1f3c-d803-4924-a552-a9b02091a5d7
[2017-06-14 20:13:40,459: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-14 20:13:40,460: INFO/job_scheduler] Job 82 finished 1 steps successfully
[2017-06-14 20:13:40,491: INFO/job_scheduler] Job 82: complete_job: Updating cache
[2017-06-14 20:13:40,491: INFO/job_scheduler] Job 82 completing (errored=False, cancelled=False)
[2017-06-14 20:13:40,533: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-14 20:13:41,083: INFO/job_scheduler] Job 66 finished 5 steps successfully
[2017-06-14 20:13:41,141: INFO/job] StatefulObject.set_state testfs-OST0000 unformatted->formatted (intentional=True) 140624142274768
[2017-06-14 20:13:41,154: INFO/job] Job 66: StateChangeJob complete, setting state formatted on testfs-OST0000
[2017-06-14 20:13:41,154: INFO/job] Job 66: not_deleted=True
[2017-06-14 20:13:41,181: INFO/job_scheduler] Job 66: complete_job: Updating cache
[2017-06-14 20:13:41,181: INFO/job_scheduler] Job 66: locked_item 140624142274768 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-14 20:13:41,192: INFO/job_scheduler] Job 66: locked_item 140623739505744 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-06-14 20:13:41,195: INFO/job_scheduler] Job 66: locked_item 140623739505872 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-06-14 20:13:41,199: INFO/job_scheduler] Job 66 completing (errored=False, cancelled=False)
[2017-06-14 20:13:41,249: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 1 tasked)
[2017-06-14 20:13:41,494: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 1 tasked)
[2017-06-14 20:13:41,541: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 1 tasked)
[2017-06-14 20:13:45,317: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:13:45,317: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:13:45,317: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:13:45,317: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:13:45,317: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b95f5d02-0c84-41d8-a255-b7192607cc7f
[2017-06-14 20:13:45,317: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b95f5d02-0c84-41d8-a255-b7192607cc7f
[2017-06-14 20:13:45,318: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b95f5d02-0c84-41d8-a255-b7192607cc7f
[2017-06-14 20:13:45,318: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b95f5d02-0c84-41d8-a255-b7192607cc7f
[2017-06-14 20:13:45,319: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_target {'ha_label': u'testfs-MDT0000_69ad68'}
[2017-06-14 20:13:45,339: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 594d2861-19e3-4c76-af91-c198d12f375a
[2017-06-14 20:13:45,339: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 594d2861-19e3-4c76-af91-c198d12f375a
[2017-06-14 20:13:55,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:13:55,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:13:55,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:13:55,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:13:55,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 594d2861-19e3-4c76-af91-c198d12f375a
[2017-06-14 20:13:55,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 594d2861-19e3-4c76-af91-c198d12f375a
[2017-06-14 20:13:55,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 594d2861-19e3-4c76-af91-c198d12f375a
[2017-06-14 20:13:55,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 594d2861-19e3-4c76-af91-c198d12f375a
[2017-06-14 20:13:55,613: INFO/job_scheduler] Job 74 finished 1 steps successfully
[2017-06-14 20:13:55,735: INFO/job] StatefulObject.set_state testfs-MDT0000 unmounted->mounted (intentional=True) 140623739419024
[2017-06-14 20:13:55,788: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm8.lotus.hpdd.lab.intel.com at severity 20
[2017-06-14 20:13:55,806: INFO/job] Job 74: StateChangeJob complete, setting state mounted on testfs-MDT0000
[2017-06-14 20:13:55,806: INFO/job] Job 74: not_deleted=True
[2017-06-14 20:13:55,806: INFO/job_scheduler] Job 74: complete_job: Updating cache
[2017-06-14 20:13:55,806: INFO/job_scheduler] Job 74: locked_item 140623739419024 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-14 20:13:55,812: INFO/job_scheduler] Job 74 completing (errored=False, cancelled=False)
[2017-06-14 20:13:55,861: INFO/job_scheduler] run_next: 1 runnable jobs of (4 pending, 0 tasked)
[2017-06-14 20:13:55,965: INFO/job_scheduler] Job 75: RunJobThread.run
[2017-06-14 20:13:55,966: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com register_target {'device_path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/testfs-OST0000', 'mount_point': u'/mnt/testfs-OST0000', 'backfstype': 'zfs'}
[2017-06-14 20:13:55,980: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9111498d-8889-4876-a237-77e4104cd9de
[2017-06-14 20:13:55,980: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9111498d-8889-4876-a237-77e4104cd9de
[2017-06-14 20:14:01,528: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-14 20:14:01,545: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-14 20:14:02,150: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-14 20:14:02,179: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-14 20:14:04,140: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:14:04,140: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:14:04,140: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:14:04,140: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:14:04,140: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9111498d-8889-4876-a237-77e4104cd9de
[2017-06-14 20:14:04,140: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9111498d-8889-4876-a237-77e4104cd9de
[2017-06-14 20:14:04,146: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9111498d-8889-4876-a237-77e4104cd9de
[2017-06-14 20:14:04,146: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9111498d-8889-4876-a237-77e4104cd9de
[2017-06-14 20:14:04,148: INFO/job_scheduler] Job 75 finished 2 steps successfully
[2017-06-14 20:14:04,268: INFO/job] StatefulObject.set_state testfs-OST0000 formatted->registered (intentional=True) 140624142274768
[2017-06-14 20:14:04,273: INFO/job] Job 75: StateChangeJob complete, setting state registered on testfs-OST0000
[2017-06-14 20:14:04,273: INFO/job] Job 75: not_deleted=True
[2017-06-14 20:14:04,273: INFO/job_scheduler] Job 75: complete_job: Updating cache
[2017-06-14 20:14:04,273: INFO/job_scheduler] Job 75: locked_item 140624142274768 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-14 20:14:04,277: INFO/job_scheduler] Job 75 completing (errored=False, cancelled=False)
[2017-06-14 20:14:04,302: INFO/job_scheduler] run_next: 1 runnable jobs of (3 pending, 0 tasked)
[2017-06-14 20:14:04,375: INFO/job_scheduler] Job 76: RunJobThread.run
[2017-06-14 20:14:04,375: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-06-14 20:14:04,401: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 47e6afa0-eb04-433f-8c97-816a6f4f810a
[2017-06-14 20:14:04,401: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 47e6afa0-eb04-433f-8c97-816a6f4f810a
[2017-06-14 20:14:05,512: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:14:05,512: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:14:05,512: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:14:05,512: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:14:05,513: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 47e6afa0-eb04-433f-8c97-816a6f4f810a
[2017-06-14 20:14:05,513: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 47e6afa0-eb04-433f-8c97-816a6f4f810a
[2017-06-14 20:14:05,516: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 47e6afa0-eb04-433f-8c97-816a6f4f810a
[2017-06-14 20:14:05,516: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 47e6afa0-eb04-433f-8c97-816a6f4f810a
[2017-06-14 20:14:05,517: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-06-14 20:14:05,529: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3073f277-793a-4db7-b1f6-f26f83a55da8
[2017-06-14 20:14:05,529: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3073f277-793a-4db7-b1f6-f26f83a55da8
[2017-06-14 20:14:06,632: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:14:06,632: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:14:06,632: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:14:06,632: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:14:06,632: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3073f277-793a-4db7-b1f6-f26f83a55da8
[2017-06-14 20:14:06,632: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3073f277-793a-4db7-b1f6-f26f83a55da8
[2017-06-14 20:14:06,643: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3073f277-793a-4db7-b1f6-f26f83a55da8
[2017-06-14 20:14:06,643: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3073f277-793a-4db7-b1f6-f26f83a55da8
[2017-06-14 20:14:06,645: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_target_store {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/testfs-OST0000', 'device_type': 'zfs', 'mount_point': u'/mnt/testfs-OST0000', 'uuid': u'14116028923112638812', 'backfstype': 'zfs'}
[2017-06-14 20:14:06,657: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a37532f2-f199-4f4e-aeda-687d0701b6ba
[2017-06-14 20:14:06,657: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a37532f2-f199-4f4e-aeda-687d0701b6ba
[2017-06-14 20:14:06,754: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:14:06,754: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:14:06,754: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:14:06,754: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:14:06,754: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a37532f2-f199-4f4e-aeda-687d0701b6ba
[2017-06-14 20:14:06,754: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a37532f2-f199-4f4e-aeda-687d0701b6ba
[2017-06-14 20:14:06,772: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a37532f2-f199-4f4e-aeda-687d0701b6ba
[2017-06-14 20:14:06,772: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a37532f2-f199-4f4e-aeda-687d0701b6ba
[2017-06-14 20:14:06,773: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_target_store {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/testfs-OST0000', 'device_type': 'zfs', 'mount_point': u'/mnt/testfs-OST0000', 'uuid': u'14116028923112638812', 'backfstype': 'zfs'}
[2017-06-14 20:14:06,786: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e9bc4117-2175-404d-9623-9fb7518fcc62
[2017-06-14 20:14:06,786: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e9bc4117-2175-404d-9623-9fb7518fcc62
[2017-06-14 20:14:06,925: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:14:06,925: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:14:06,925: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:14:06,925: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:14:06,925: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e9bc4117-2175-404d-9623-9fb7518fcc62
[2017-06-14 20:14:06,925: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e9bc4117-2175-404d-9623-9fb7518fcc62
[2017-06-14 20:14:06,952: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e9bc4117-2175-404d-9623-9fb7518fcc62
[2017-06-14 20:14:06,952: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e9bc4117-2175-404d-9623-9fb7518fcc62
[2017-06-14 20:14:06,953: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/testfs-OST0000', 'mount_point': u'/mnt/testfs-OST0000', 'uuid': u'14116028923112638812', 'ha_label': u'testfs-OST0000_181072', 'primary': True}
[2017-06-14 20:14:06,965: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 460709ec-00fc-4413-86ae-c5f39e712585
[2017-06-14 20:14:06,965: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 460709ec-00fc-4413-86ae-c5f39e712585
[2017-06-14 20:14:10,092: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:14:10,092: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:14:10,093: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:14:10,093: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:14:10,093: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 460709ec-00fc-4413-86ae-c5f39e712585
[2017-06-14 20:14:10,093: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 460709ec-00fc-4413-86ae-c5f39e712585
[2017-06-14 20:14:10,131: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 460709ec-00fc-4413-86ae-c5f39e712585
[2017-06-14 20:14:10,131: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 460709ec-00fc-4413-86ae-c5f39e712585
[2017-06-14 20:14:10,133: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/testfs-OST0000', 'mount_point': u'/mnt/testfs-OST0000', 'uuid': u'14116028923112638812', 'ha_label': u'testfs-OST0000_181072', 'primary': False}
[2017-06-14 20:14:10,150: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a304c893-4db2-46c9-b6f7-a8fc8cfb9efd
[2017-06-14 20:14:10,150: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a304c893-4db2-46c9-b6f7-a8fc8cfb9efd
[2017-06-14 20:14:11,337: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:14:11,337: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:14:11,337: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:14:11,337: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:14:11,337: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a304c893-4db2-46c9-b6f7-a8fc8cfb9efd
[2017-06-14 20:14:11,337: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a304c893-4db2-46c9-b6f7-a8fc8cfb9efd
[2017-06-14 20:14:11,375: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a304c893-4db2-46c9-b6f7-a8fc8cfb9efd
[2017-06-14 20:14:11,375: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a304c893-4db2-46c9-b6f7-a8fc8cfb9efd
[2017-06-14 20:14:11,375: INFO/job_scheduler] Job 76 finished 6 steps successfully
[2017-06-14 20:14:11,432: INFO/job] StatefulObject.set_state testfs-OST0000 registered->unmounted (intentional=True) 140624142274768
[2017-06-14 20:14:11,438: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on testfs-OST0000 at severity 30
[2017-06-14 20:14:11,449: INFO/job] Job 76: StateChangeJob complete, setting state unmounted on testfs-OST0000
[2017-06-14 20:14:11,450: INFO/job] Job 76: not_deleted=True
[2017-06-14 20:14:11,452: INFO/job_scheduler] Job 76: complete_job: Updating cache
[2017-06-14 20:14:11,452: INFO/job_scheduler] Job 76: locked_item 140624142274768 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-14 20:14:11,468: INFO/job_scheduler] Job 76 completing (errored=False, cancelled=False)
[2017-06-14 20:14:11,492: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 0 tasked)
[2017-06-14 20:14:11,631: INFO/job] create_parameters: host: 'lotus-32vm5.lotus.hpdd.lab.intel.com' active_volume_node: 'TargetVolumeInfo(host=<ManagedHost: lotus-32vm5.lotus.hpdd.lab.intel.com>, path=u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/testfs-OST0000', device_type='zfs')'
[2017-06-14 20:14:11,633: INFO/job_scheduler] Job 77: RunJobThread.run
[2017-06-14 20:14:11,635: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/testfs-OST0000', 'device_type': 'zfs'}
[2017-06-14 20:14:11,686: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/testfs-OST0000', 'device_type': 'zfs'}
[2017-06-14 20:14:11,686: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com export_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/testfs-OST0000', 'device_type': 'zfs'}
[2017-06-14 20:14:11,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bb8646ae-9fb8-452f-883f-358956262f58
[2017-06-14 20:14:11,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bb8646ae-9fb8-452f-883f-358956262f58
[2017-06-14 20:14:11,708: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 94c06349-a5ba-479d-80e0-ecc22e17f037
[2017-06-14 20:14:11,708: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 94c06349-a5ba-479d-80e0-ecc22e17f037
[2017-06-14 20:14:11,728: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ffd491ba-ce8c-414c-b5c2-ae00f36eb523
[2017-06-14 20:14:11,728: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ffd491ba-ce8c-414c-b5c2-ae00f36eb523
[2017-06-14 20:14:13,839: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:14:13,839: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:14:13,840: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:14:13,840: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-14 20:14:13,840: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 94c06349-a5ba-479d-80e0-ecc22e17f037
[2017-06-14 20:14:13,840: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 94c06349-a5ba-479d-80e0-ecc22e17f037
[2017-06-14 20:14:13,843: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:14:13,843: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:14:13,843: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:14:13,843: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-14 20:14:13,843: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ffd491ba-ce8c-414c-b5c2-ae00f36eb523
[2017-06-14 20:14:13,843: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ffd491ba-ce8c-414c-b5c2-ae00f36eb523
[2017-06-14 20:14:13,845: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ffd491ba-ce8c-414c-b5c2-ae00f36eb523
[2017-06-14 20:14:13,845: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ffd491ba-ce8c-414c-b5c2-ae00f36eb523
[2017-06-14 20:14:13,872: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 94c06349-a5ba-479d-80e0-ecc22e17f037
[2017-06-14 20:14:13,872: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 94c06349-a5ba-479d-80e0-ecc22e17f037
[2017-06-14 20:14:14,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:14:14,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:14:14,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:14:14,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-14 20:14:14,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bb8646ae-9fb8-452f-883f-358956262f58
[2017-06-14 20:14:14,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bb8646ae-9fb8-452f-883f-358956262f58
[2017-06-14 20:14:14,885: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bb8646ae-9fb8-452f-883f-358956262f58
[2017-06-14 20:14:14,885: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bb8646ae-9fb8-452f-883f-358956262f58
[2017-06-14 20:14:14,886: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com import_target {'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk5/testfs-OST0000', 'pacemaker_ha_operation': False, 'device_type': 'zfs', 'validate_importable': True}
[2017-06-14 20:14:14,899: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 467d86e8-0879-47dd-8b59-cf19eabd5ab8
[2017-06-14 20:14:14,899: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 467d86e8-0879-47dd-8b59-cf19eabd5ab8
[2017-06-14 20:14:17,795: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-06-14 20:14:22,034: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:14:22,034: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:14:22,034: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:14:22,034: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:14:22,035: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 467d86e8-0879-47dd-8b59-cf19eabd5ab8
[2017-06-14 20:14:22,035: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 467d86e8-0879-47dd-8b59-cf19eabd5ab8
[2017-06-14 20:14:22,074: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 467d86e8-0879-47dd-8b59-cf19eabd5ab8
[2017-06-14 20:14:22,074: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 467d86e8-0879-47dd-8b59-cf19eabd5ab8
[2017-06-14 20:14:22,075: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_target {'ha_label': u'testfs-OST0000_181072'}
[2017-06-14 20:14:22,095: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 82e0b60f-2f2d-4f9f-912f-bd0ba08fd9b4
[2017-06-14 20:14:22,095: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 82e0b60f-2f2d-4f9f-912f-bd0ba08fd9b4
[2017-06-14 20:14:34,264: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:14:34,264: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:14:34,264: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:14:34,264: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-14 20:14:34,264: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 82e0b60f-2f2d-4f9f-912f-bd0ba08fd9b4
[2017-06-14 20:14:34,264: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 82e0b60f-2f2d-4f9f-912f-bd0ba08fd9b4
[2017-06-14 20:14:34,277: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 82e0b60f-2f2d-4f9f-912f-bd0ba08fd9b4
[2017-06-14 20:14:34,277: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 82e0b60f-2f2d-4f9f-912f-bd0ba08fd9b4
[2017-06-14 20:14:34,278: INFO/job_scheduler] Job 77 finished 1 steps successfully
[2017-06-14 20:14:34,402: INFO/job] StatefulObject.set_state testfs-OST0000 unmounted->mounted (intentional=True) 140624142274768
[2017-06-14 20:14:34,436: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 20
[2017-06-14 20:14:34,443: INFO/job] Job 77: StateChangeJob complete, setting state mounted on testfs-OST0000
[2017-06-14 20:14:34,443: INFO/job] Job 77: not_deleted=True
[2017-06-14 20:14:34,443: INFO/job_scheduler] Job 77: complete_job: Updating cache
[2017-06-14 20:14:34,443: INFO/job_scheduler] Job 77: locked_item 140624142274768 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-14 20:14:34,446: INFO/job_scheduler] Job 77 completing (errored=False, cancelled=False)
[2017-06-14 20:14:34,468: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-14 20:14:34,588: INFO/job_scheduler] Job 78: open DB connection during completion
[2017-06-14 20:14:34,593: INFO/job] StatefulObject.set_state testfs unavailable->available (intentional=True) 140623739455824
[2017-06-14 20:14:34,597: INFO/job] Job 78: StateChangeJob complete, setting state available on testfs
[2017-06-14 20:14:34,597: INFO/job] Job 78: not_deleted=True
[2017-06-14 20:14:34,597: INFO/job_scheduler] Job 78: complete_job: Updating cache
[2017-06-14 20:14:34,598: INFO/job_scheduler] Job 78: locked_item 140623739455824 <class 'chroma_core.models.filesystem.ManagedFilesystem'> False True
[2017-06-14 20:14:34,601: INFO/job_scheduler] Job 78 completing (errored=False, cancelled=False)
[2017-06-14 20:14:34,678: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:14:37,965: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:14:39,863: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:16:10,068: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:16:19,832: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:16:19,848: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:16:31,524: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:16:31,586: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:16:44,137: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:16:44,148: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:23:22,980: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:23:23,053: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:23:29,039: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:23:35,662: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:23:35,678: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:23:40,267: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:23:40,284: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:26:21,443: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:26:31,670: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:26:31,687: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:26:43,060: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:26:43,081: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:26:56,060: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:26:56,080: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:33:34,346: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:33:34,372: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:33:41,324: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:33:47,628: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:33:47,645: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:33:51,869: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:33:51,885: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:36:32,836: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:36:43,405: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:36:43,426: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:36:53,588: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:36:53,616: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:37:07,979: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:37:07,997: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:43:45,755: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:43:45,788: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:43:51,522: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:43:59,395: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:43:59,412: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:44:03,527: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:44:03,629: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:46:45,352: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:46:55,124: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:46:55,145: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:47:06,107: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:47:06,135: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:47:19,856: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:47:19,872: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:53:57,216: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:53:57,354: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:54:02,723: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:54:11,137: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:54:11,157: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:54:15,096: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:54:15,112: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:56:55,715: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:57:06,854: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:57:06,870: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:57:16,696: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:57:16,719: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:57:31,746: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 20:57:31,769: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:04:08,749: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:04:08,780: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:04:13,940: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:04:23,150: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:04:23,166: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:04:26,625: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:04:26,645: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:07:08,129: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:07:18,661: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:07:18,677: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:07:29,145: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:07:29,168: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:07:43,699: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:07:43,713: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:14:20,215: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:14:20,256: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:14:25,138: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:14:35,058: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:14:35,087: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:14:38,203: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:14:38,221: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:17:19,550: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:17:30,456: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:17:30,469: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:17:39,647: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:17:39,670: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:17:55,583: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:17:55,597: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:24:32,004: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:24:32,025: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:24:36,323: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:24:47,006: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:24:47,041: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:24:49,728: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:24:49,742: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:27:29,908: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:27:42,080: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:27:42,097: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:27:51,423: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:27:51,444: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:28:07,638: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:28:07,656: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:34:43,788: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:34:43,821: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:34:47,642: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:34:58,978: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:34:58,993: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:35:01,354: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:35:01,371: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:37:41,269: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:37:53,676: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:37:53,688: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:38:02,822: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:38:02,847: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:38:19,484: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:38:19,498: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:44:56,152: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:44:56,186: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:44:58,934: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:45:10,801: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:45:10,816: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:45:13,054: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:45:13,074: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:47:52,623: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:48:05,299: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:48:05,352: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:48:14,411: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:48:14,437: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:48:31,314: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:48:31,330: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:55:06,640: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:55:06,670: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:55:11,217: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:55:22,709: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:55:22,724: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:55:24,806: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:55:24,824: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:58:03,921: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:58:16,963: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:58:16,977: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:58:27,020: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:58:27,043: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:58:43,284: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 21:58:43,304: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:05:18,164: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:05:18,193: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:05:21,484: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:05:34,624: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:05:34,651: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:05:36,424: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:05:36,438: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:08:15,260: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:08:28,672: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:08:28,707: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:08:37,503: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:08:37,533: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:08:55,321: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:08:55,335: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:15:29,678: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:15:29,717: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:15:32,854: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:15:46,435: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:15:46,454: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:15:48,040: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:15:48,054: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:18:26,617: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:18:40,341: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:18:40,362: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:18:50,077: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:18:50,105: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:19:07,235: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:19:07,249: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:25:41,199: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:25:41,231: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:25:44,042: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:25:58,385: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:25:58,402: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:25:59,636: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:25:59,655: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:28:38,110: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:28:52,155: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:28:52,171: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:29:01,608: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:29:01,635: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:29:19,201: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:29:19,218: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:35:52,609: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:35:52,643: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:35:55,273: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:36:10,239: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:36:10,257: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:36:11,378: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:36:11,390: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:38:49,437: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:39:04,012: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:39:04,034: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:39:12,100: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:39:12,126: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:39:31,086: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:39:31,103: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:46:04,065: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:46:04,097: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:46:06,558: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:46:22,090: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:46:22,109: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:46:23,130: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:46:23,144: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:49:00,776: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:49:15,752: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:49:15,767: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:49:23,491: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:49:23,519: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:49:42,952: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:49:42,970: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:56:15,471: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:56:15,502: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:56:17,832: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:56:33,929: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:56:33,944: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:56:34,794: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:56:34,821: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:59:12,276: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:59:27,639: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:59:27,656: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:59:34,867: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:59:34,902: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:59:54,862: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 22:59:54,883: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:06:26,903: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:06:26,943: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:06:29,412: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:06:45,798: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:06:45,811: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:06:46,411: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:06:46,428: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:09:23,677: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:09:39,391: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:09:39,408: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:09:47,405: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:09:47,436: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:10:06,643: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:10:06,660: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:16:38,420: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:16:38,446: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:16:40,963: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:16:57,581: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:16:57,600: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:16:57,941: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:16:57,954: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:19:35,162: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:19:51,037: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:19:51,056: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:19:57,879: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:19:57,910: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:20:18,509: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:20:18,527: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:26:50,861: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:26:50,895: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:26:52,593: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:27:09,376: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:27:09,394: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:27:09,885: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:27:09,935: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:29:47,765: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:30:02,686: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:30:02,709: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:30:10,383: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:30:10,410: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:30:30,410: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:30:30,424: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:37:01,266: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:37:01,285: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:37:03,780: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:37:21,219: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:37:21,236: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:37:21,489: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:37:21,505: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:39:58,247: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:40:14,400: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:40:14,413: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:40:20,815: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:40:20,838: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:40:42,261: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:40:42,278: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:47:13,174: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:47:13,205: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:47:15,058: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:47:33,212: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:47:33,231: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:47:33,269: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:47:33,286: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:50:10,614: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:50:26,148: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:50:26,185: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:50:32,198: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:50:32,233: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:50:54,117: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:50:54,136: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:57:24,601: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:57:24,632: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:57:27,365: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:57:44,901: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:57:44,918: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:57:45,174: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-14 23:57:45,186: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:00:21,056: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:00:38,004: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:00:38,020: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:00:43,650: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:00:43,681: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:01:05,889: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:01:05,919: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:07:36,050: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:07:36,080: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:07:38,574: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:07:56,609: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:07:56,627: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:07:56,982: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:07:57,003: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:10:32,474: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:10:49,857: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:10:49,874: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:10:55,151: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:10:55,183: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:11:17,821: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:11:17,837: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:17:47,555: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:17:47,602: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:17:48,813: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:18:08,372: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:18:08,391: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:18:08,849: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:18:08,866: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:20:43,874: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:21:01,704: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:21:01,719: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:21:06,518: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:21:06,556: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:21:29,622: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:21:29,635: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:27:58,992: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:27:59,029: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:28:01,105: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:28:20,103: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:28:20,121: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:28:20,741: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:28:20,756: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:30:55,233: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:31:13,533: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:31:13,551: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:31:18,051: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:31:18,087: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:31:41,605: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:31:41,620: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:38:10,385: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:38:10,422: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:38:11,293: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:38:31,692: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:38:31,710: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:38:32,569: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:38:32,585: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:41:06,612: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:41:25,193: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:41:25,209: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:41:29,513: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:41:29,536: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:41:53,460: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:41:53,476: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:48:21,744: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:48:21,781: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:48:22,542: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:48:43,375: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:48:43,391: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:48:44,278: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:48:44,292: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:51:18,040: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:51:36,903: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:51:36,925: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:51:41,061: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:51:41,086: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:52:05,415: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:52:05,432: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:58:33,089: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:58:33,112: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:58:34,816: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:58:54,885: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:58:54,899: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:58:56,107: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 00:58:56,121: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:01:29,393: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:01:48,613: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:01:48,631: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:01:52,554: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:01:52,577: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:02:17,317: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:02:17,338: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:08:45,077: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:08:45,552: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:08:45,602: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:09:06,407: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:09:06,422: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:09:07,934: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:09:07,965: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:11:40,749: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:12:00,299: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:12:00,318: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:12:04,138: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:12:04,162: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:12:29,432: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:12:29,451: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:18:55,967: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:18:56,005: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:18:56,243: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:19:18,019: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:19:18,036: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:19:19,814: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:19:19,828: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:21:52,170: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:22:12,076: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:22:12,104: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:22:15,642: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:22:15,669: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:22:41,390: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:22:41,419: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:29:07,434: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:29:07,488: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:29:07,543: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:29:29,625: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:29:29,641: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:29:31,703: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:29:31,725: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:32:03,835: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:32:23,857: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:32:23,877: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:32:27,262: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:32:27,286: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:32:53,519: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:32:53,563: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:39:18,708: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:39:19,030: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:39:19,067: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:39:41,134: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:39:41,152: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:39:43,646: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:39:43,668: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:42:15,477: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:42:35,631: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:42:35,645: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:42:38,821: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:42:38,855: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:43:05,385: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:43:05,400: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:49:29,914: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:49:30,494: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:49:30,515: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:49:52,838: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:49:52,854: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:49:55,476: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:49:55,494: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:52:27,042: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:52:47,363: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:52:47,379: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:52:50,339: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:52:50,362: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:53:17,288: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:53:17,305: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:59:41,107: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:59:42,024: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 01:59:42,056: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:00:04,483: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:00:04,494: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:00:07,428: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:00:07,444: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:02:39,434: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:02:59,319: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:02:59,340: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:03:02,009: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:03:02,039: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:03:29,178: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:03:29,206: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:09:52,377: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:09:54,495: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:09:54,537: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:10:16,517: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:10:16,535: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:10:19,393: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:10:19,414: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:12:50,131: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:13:11,156: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:13:11,174: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:13:13,487: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:13:13,510: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:13:41,269: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:13:41,287: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:20:04,613: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:20:05,094: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:20:05,133: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:20:28,106: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:20:28,122: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:20:31,088: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:20:31,107: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:23:01,519: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:23:22,844: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:23:22,860: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:23:25,093: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:23:25,138: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:23:53,225: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:23:53,242: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:30:14,895: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:30:17,648: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:30:17,682: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:30:39,660: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:30:39,678: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:30:43,081: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:30:43,101: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:33:12,898: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:33:34,509: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:33:34,524: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:33:36,694: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:33:36,720: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:34:05,270: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:34:05,285: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:40:26,201: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:40:29,177: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:40:29,210: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:40:51,219: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:40:51,240: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:40:54,961: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:40:54,984: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:43:25,326: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:43:46,278: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:43:46,295: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:43:48,232: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:43:48,256: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:44:17,343: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:44:17,357: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:50:37,573: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:50:39,635: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:50:39,659: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:51:02,904: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:51:02,922: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:51:06,931: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:51:06,948: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:53:35,660: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:53:58,148: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:53:58,164: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:53:59,685: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:53:59,716: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:54:29,274: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 02:54:29,291: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:00:48,727: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:00:51,088: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:00:51,124: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:01:14,543: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:01:14,564: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:01:18,769: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:01:18,784: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:03:47,072: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:04:09,905: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:04:09,921: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:04:11,132: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:04:11,157: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:04:41,188: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:04:41,206: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:11:00,146: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:11:02,590: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:11:02,611: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:11:26,074: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:11:26,088: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:11:30,534: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:11:30,554: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:13:58,515: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:14:21,679: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:14:21,692: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:14:22,675: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:14:22,704: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:14:53,023: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:14:53,041: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:21:11,692: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:21:13,992: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:21:14,023: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:21:37,622: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:21:37,639: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:21:42,427: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:21:42,441: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:24:09,985: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:24:33,499: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:24:33,515: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:24:35,137: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:24:35,164: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:25:04,955: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:25:04,970: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:31:23,974: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:31:25,321: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:31:25,356: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:31:49,198: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:31:49,231: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:31:54,244: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:31:54,257: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:34:21,368: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:34:45,211: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:34:45,228: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:34:45,500: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:34:45,524: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:35:16,818: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:35:16,837: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:41:34,243: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:41:36,773: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:41:36,807: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:42:00,903: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:42:00,918: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:42:06,189: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:42:06,208: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:44:32,844: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:44:56,964: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:44:56,987: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:44:57,018: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:44:57,041: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:45:28,693: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:45:28,712: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:51:45,542: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:51:48,226: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:51:48,256: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:52:12,436: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:52:12,453: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:52:17,956: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:52:17,977: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:54:44,257: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:55:08,551: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:55:08,572: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:55:08,712: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:55:08,728: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:55:40,669: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 03:55:40,699: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:01:56,840: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:01:59,783: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:01:59,811: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:02:24,068: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:02:24,085: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:02:29,855: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:02:29,873: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:04:55,645: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:05:20,115: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:05:20,142: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:05:20,467: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:05:20,484: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:05:52,574: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:05:52,592: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:12:08,081: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:12:11,229: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:12:11,253: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:12:35,860: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:12:35,882: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:12:41,697: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:12:41,715: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:15:08,188: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:15:31,543: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:15:31,576: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:15:32,248: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:15:32,270: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:16:04,594: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:16:04,607: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:22:19,348: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:22:22,696: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:22:22,727: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:22:47,396: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:22:47,413: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:22:53,580: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:22:53,598: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:25:18,573: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:25:43,055: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:25:43,091: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:25:44,019: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:25:44,032: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:26:16,490: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:26:16,506: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:32:30,741: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:32:34,126: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:32:34,161: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:32:58,964: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:32:58,982: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:33:05,427: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:33:05,443: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:35:29,928: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:35:54,562: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:35:54,597: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:35:55,871: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:35:55,893: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:36:28,383: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:36:28,396: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:42:41,982: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:42:45,634: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:42:45,661: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:43:10,622: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:43:10,644: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:43:17,388: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:43:17,409: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:45:41,412: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:46:06,093: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:46:06,122: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:46:07,773: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:46:07,796: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:46:40,368: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:46:40,383: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:52:53,291: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:52:57,052: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:52:57,078: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:53:22,185: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:53:22,204: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:53:29,189: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:53:29,246: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:55:53,869: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:56:17,617: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:56:17,646: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:56:19,531: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:56:19,551: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:56:52,315: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 04:56:52,329: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:03:04,513: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:03:08,478: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:03:08,512: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:03:33,763: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:03:33,786: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:03:40,981: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:03:40,998: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:06:04,408: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:06:29,149: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:06:29,176: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:06:31,258: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:06:31,273: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:07:04,206: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:07:04,224: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:13:15,752: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:13:19,803: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:13:19,843: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:13:45,490: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:13:45,507: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:13:52,750: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:13:52,767: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:16:16,866: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:16:40,771: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:16:40,798: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:16:43,079: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:16:43,093: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:17:16,336: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:17:16,354: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:23:27,892: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:23:31,213: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:23:31,248: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:23:57,101: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:23:57,117: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:24:04,624: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:24:04,644: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:26:27,291: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:26:52,356: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:26:52,377: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:26:54,850: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:26:54,866: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:27:28,229: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:27:28,242: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:33:39,074: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:33:42,605: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:33:42,638: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:34:08,706: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:34:08,725: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:34:16,625: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:34:16,641: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:36:39,752: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:37:04,980: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:37:05,007: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:37:06,694: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:37:06,714: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:37:40,264: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:37:40,280: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:43:49,224: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:43:54,141: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:43:54,177: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:44:20,327: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:44:20,342: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:44:28,586: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:44:28,604: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:46:50,336: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:47:15,398: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:47:15,429: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:47:18,334: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:47:18,348: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:47:52,223: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:47:52,239: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:54:00,531: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:54:05,584: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:54:05,603: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:54:31,900: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:54:31,917: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:54:40,485: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:54:40,499: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:57:02,126: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:57:27,209: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:57:27,237: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:57:30,316: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:57:30,335: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:58:04,481: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 05:58:04,496: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:04:11,683: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:04:17,070: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:04:17,105: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:04:43,658: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:04:43,679: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:04:52,431: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:04:52,448: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:07:13,607: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:07:38,702: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:07:38,726: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:07:42,116: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:07:42,130: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:08:16,469: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:08:16,482: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:14:22,908: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:14:28,527: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:14:28,567: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:14:55,341: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:14:55,355: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:15:04,237: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:15:04,263: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:17:25,102: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:17:50,229: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:17:50,266: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:17:54,022: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:17:54,041: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:18:28,319: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:18:28,333: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:24:35,049: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:24:40,048: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:24:40,119: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:25:07,117: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:25:07,135: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:25:16,006: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:25:16,028: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:27:36,524: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:28:01,882: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:28:01,921: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:28:05,860: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:28:05,881: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:28:40,382: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:28:40,407: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:34:45,281: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:34:51,778: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:34:51,810: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:35:18,739: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:35:18,756: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:35:28,163: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:35:28,179: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:37:48,010: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:38:13,314: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:38:13,341: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:38:17,686: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:38:17,704: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:38:52,294: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:38:52,316: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:44:56,589: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:45:03,242: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:45:03,311: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:45:30,468: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:45:30,487: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:45:39,963: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:45:39,979: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:47:59,460: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:48:24,937: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:48:24,971: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:48:29,482: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:48:29,497: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:49:04,230: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:49:04,245: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:55:07,797: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:55:14,921: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:55:14,955: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:55:42,122: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:55:42,140: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:55:51,973: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:55:51,997: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:58:11,999: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:58:36,431: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:58:36,459: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:58:41,101: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:58:41,114: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:59:16,182: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 06:59:16,200: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:05:19,162: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:05:26,496: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:05:26,525: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:05:53,786: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:05:53,805: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:06:03,899: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:06:03,922: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:08:22,444: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:08:47,880: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:08:47,908: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:08:52,780: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:08:52,798: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:09:28,271: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:09:28,284: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:15:30,419: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:15:37,919: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:15:37,938: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:16:05,470: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:16:05,487: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:16:15,910: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:16:15,927: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:18:33,983: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:18:59,257: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:18:59,283: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:19:04,600: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:19:04,620: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:19:40,207: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:19:40,224: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:25:41,732: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:25:49,461: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:25:49,493: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:26:17,115: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:26:17,132: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:26:27,862: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:26:27,875: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:28:45,409: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:29:11,753: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:29:11,773: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:29:16,327: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:29:16,346: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:29:52,078: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:29:52,094: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:35:52,890: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:36:00,895: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:36:00,921: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:36:28,706: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:36:28,731: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:36:39,760: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:36:39,786: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:38:56,876: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:39:23,216: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:39:23,246: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:39:28,009: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:39:28,024: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:40:03,982: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:40:03,995: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:46:04,270: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:46:12,343: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:46:12,381: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:46:40,271: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:46:40,287: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:46:51,587: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:46:51,610: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:49:08,339: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:49:33,654: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:49:33,675: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:49:39,682: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:49:39,697: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:50:15,882: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:50:15,900: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:56:15,627: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:56:23,812: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:56:23,845: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:56:52,067: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:56:52,085: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:57:03,477: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:57:03,492: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:59:19,806: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:59:45,135: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:59:45,161: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:59:51,416: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 07:59:51,438: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:00:27,839: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:00:27,866: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:06:26,929: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:06:36,296: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:06:36,336: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:07:03,719: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:07:03,740: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:07:15,373: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:07:15,393: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:09:32,151: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:09:56,597: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:09:56,623: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:10:03,172: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:10:03,186: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:10:39,739: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:10:39,753: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:16:39,320: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:16:46,896: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:16:46,928: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:17:15,361: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:17:15,375: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:17:27,153: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:17:27,196: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:19:42,629: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:20:08,032: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:20:08,059: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:20:14,877: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:20:14,887: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:20:51,606: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:20:51,621: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:26:49,854: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:26:58,387: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:26:58,422: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:27:27,204: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:27:27,216: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:27:38,975: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:27:38,990: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:29:53,956: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:30:20,472: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:30:20,501: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:30:26,540: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:30:26,554: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:31:03,529: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:31:03,542: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:37:02,121: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:37:09,736: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:37:09,768: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:37:38,815: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:37:38,889: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:37:50,972: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:37:50,987: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:40:05,273: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:40:30,977: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:40:31,033: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:40:38,324: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:40:38,344: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:41:15,433: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:41:15,454: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:47:12,413: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:47:21,153: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:47:21,189: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:47:50,535: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:47:50,550: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:48:02,850: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:48:02,865: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:50:16,701: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:50:42,486: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:50:42,514: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:50:50,133: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:50:50,149: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:51:27,369: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:51:27,390: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:57:23,602: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:57:32,577: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:57:32,598: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:58:02,150: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:58:02,167: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:58:14,701: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 08:58:14,718: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:00:28,200: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:00:53,978: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:00:54,005: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:01:01,813: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:01:01,833: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:01:39,311: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:01:39,327: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:07:34,835: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:07:44,045: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:07:44,078: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:08:13,708: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:08:13,723: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:08:26,502: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:08:26,526: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:10:40,627: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:11:05,457: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:11:05,482: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:11:13,592: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:11:13,615: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:11:51,217: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:11:51,234: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:17:46,077: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:17:55,431: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:17:55,465: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:18:25,219: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:18:25,235: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:18:38,381: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:18:38,400: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:20:51,067: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:21:17,067: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:21:17,098: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:21:25,366: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:21:25,412: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:22:03,080: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:22:03,094: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:27:57,312: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:28:06,804: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:28:06,835: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:28:36,770: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:28:36,786: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:28:50,194: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:28:50,208: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:31:02,539: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:31:28,582: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:31:28,609: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:31:37,124: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:31:37,144: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:32:15,101: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:32:15,116: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:38:08,491: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:38:18,273: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:38:18,303: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:38:48,343: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:38:48,357: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:39:02,070: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:39:02,085: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:41:13,985: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:41:40,085: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:41:40,114: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:41:48,820: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:41:48,846: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:42:27,092: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:42:27,110: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:48:19,876: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:48:29,695: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:48:29,730: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:49:00,017: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:49:00,041: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:49:13,953: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:49:13,971: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:51:25,460: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:51:51,813: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:51:51,835: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:52:00,509: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:52:00,528: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:52:39,209: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:52:39,237: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:58:31,139: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:58:41,191: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:58:41,224: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:59:11,657: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:59:11,684: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:59:25,751: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 09:59:25,765: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:01:36,927: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:02:03,261: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:02:03,289: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:02:12,327: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:02:12,339: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:02:51,059: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:02:51,076: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:08:42,416: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:08:52,717: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:08:52,751: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:09:23,268: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:09:23,285: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:09:37,608: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:09:37,627: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:11:49,389: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:12:15,819: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:12:15,846: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:12:24,024: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:12:24,037: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:13:02,919: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:13:02,931: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:18:54,706: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:19:04,517: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:19:04,551: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:19:34,798: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:19:34,813: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:19:49,658: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:19:49,670: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:21:59,787: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:22:26,324: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:22:26,354: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:22:35,733: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:22:35,745: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:23:14,885: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:23:14,903: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:29:04,862: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:29:15,997: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:29:16,039: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:29:46,399: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:29:46,413: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:30:01,556: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:30:01,573: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:32:11,268: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:32:38,799: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:32:38,826: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:32:47,508: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:32:47,525: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:33:26,781: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:33:26,800: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:39:16,057: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:39:27,493: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:39:27,530: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:39:58,043: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:39:58,061: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:40:13,360: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:40:13,375: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:42:22,732: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:42:49,306: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:42:49,330: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:42:59,317: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:42:59,337: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:43:38,743: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:43:38,760: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:49:27,299: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:49:40,080: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:49:40,116: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:50:09,606: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:50:09,627: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:50:25,285: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:50:25,310: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:52:34,220: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:53:00,812: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:53:00,847: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:53:11,144: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:53:11,161: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:53:50,692: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:53:50,711: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:59:38,570: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:59:50,521: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 10:59:50,554: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:00:21,141: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:00:21,156: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:00:37,249: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:00:37,266: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:02:45,880: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:03:12,373: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:03:12,400: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:03:22,965: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:03:22,977: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:04:02,635: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:04:02,650: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:09:49,777: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:10:01,986: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:10:02,014: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:10:32,798: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:10:32,815: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:10:49,028: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:10:49,047: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:12:57,423: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:13:23,863: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:13:23,891: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:13:34,800: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:13:34,819: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:14:14,608: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:14:14,622: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:20:02,070: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:20:14,710: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:20:14,745: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:20:44,295: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:20:44,314: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:21:01,041: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:21:01,055: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:23:08,850: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:23:35,276: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:23:35,305: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:23:46,612: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:23:46,627: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:24:26,628: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:24:26,649: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:30:12,295: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:30:25,137: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:30:25,175: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:30:55,973: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:30:55,988: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:31:12,885: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:31:12,897: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:33:20,330: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:33:46,737: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:33:46,756: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:33:58,447: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:33:58,461: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:34:38,474: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:34:38,496: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:40:23,533: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:40:36,603: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:40:36,638: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:41:07,605: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:41:07,618: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:41:24,807: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:41:24,820: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:43:31,781: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:43:58,322: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:43:58,356: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:44:10,242: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:44:10,258: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:44:50,516: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:44:50,537: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:50:34,768: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:50:47,979: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:50:48,012: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:51:19,327: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:51:19,339: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:51:36,689: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:51:36,735: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:53:43,145: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:54:11,024: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:54:11,053: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:54:22,112: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:54:22,129: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:55:02,379: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 11:55:02,394: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:00:47,026: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:01:00,426: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:01:00,460: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:01:30,948: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:01:30,967: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:01:48,513: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:01:48,529: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:03:54,535: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:04:21,414: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:04:21,442: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:04:33,799: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:04:33,814: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:05:14,267: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:05:14,285: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:10:57,357: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:11:10,880: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:11:10,912: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:11:42,780: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:11:42,802: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:12:00,404: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:12:00,425: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:14:05,953: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:14:32,938: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:14:32,980: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:14:45,672: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:14:45,687: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:15:26,158: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:15:26,173: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:21:08,595: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:21:22,438: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:21:22,471: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:21:54,448: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:21:54,462: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:22:12,276: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:22:12,295: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:23:01,627: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 12:23:01,630: INFO/command_plan] set_state: <class 'chroma_core.models.filesystem.ManagedFilesystem'>-1 transitioning from available to removed
[2017-06-15 12:23:01,817: INFO/command_plan] Created command 31 (Remove file system testfs from configuration) with 7 jobs
[2017-06-15 12:23:01,862: INFO/job_scheduler] run_next: 1 runnable jobs of (7 pending, 0 tasked)
[2017-06-15 12:23:01,946: INFO/job_scheduler] Job 83: open DB connection during completion
[2017-06-15 12:23:02,039: INFO/job] StatefulObject.set_state testfs available->unavailable (intentional=True) 140623538072144
[2017-06-15 12:23:02,046: INFO/job] Job 83: StateChangeJob complete, setting state unavailable on testfs
[2017-06-15 12:23:02,046: INFO/job] Job 83: not_deleted=True
[2017-06-15 12:23:02,047: INFO/job_scheduler] Job 83: complete_job: Updating cache
[2017-06-15 12:23:02,047: INFO/job_scheduler] Job 83: locked_item 140623538072144 <class 'chroma_core.models.filesystem.ManagedFilesystem'> False True
[2017-06-15 12:23:02,049: INFO/job_scheduler] Job 83 completing (errored=False, cancelled=False)
[2017-06-15 12:23:02,061: INFO/job_scheduler] run_next: 2 runnable jobs of (6 pending, 0 tasked)
[2017-06-15 12:23:02,149: INFO/job_scheduler] Job 84: RunJobThread.run
[2017-06-15 12:23:02,150: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com stop_target {'ha_label': u'testfs-MDT0000_69ad68'}
[2017-06-15 12:23:02,164: INFO/job_scheduler] Job 85: RunJobThread.run
[2017-06-15 12:23:02,165: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_target {'ha_label': u'testfs-OST0000_181072'}
[2017-06-15 12:23:02,188: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0ca9117a-c1f7-45c1-abeb-6f76ed394d1f
[2017-06-15 12:23:02,188: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0ca9117a-c1f7-45c1-abeb-6f76ed394d1f
[2017-06-15 12:23:02,211: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d76359bb-bdba-4291-93b4-3c3fa49df2ce
[2017-06-15 12:23:02,211: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d76359bb-bdba-4291-93b4-3c3fa49df2ce
[2017-06-15 12:23:12,334: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:23:12,334: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:23:12,334: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:23:12,334: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:23:12,334: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0ca9117a-c1f7-45c1-abeb-6f76ed394d1f
[2017-06-15 12:23:12,334: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0ca9117a-c1f7-45c1-abeb-6f76ed394d1f
[2017-06-15 12:23:12,357: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0ca9117a-c1f7-45c1-abeb-6f76ed394d1f
[2017-06-15 12:23:12,357: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0ca9117a-c1f7-45c1-abeb-6f76ed394d1f
[2017-06-15 12:23:12,358: INFO/job_scheduler] Job 84 finished 1 steps successfully
[2017-06-15 12:23:12,483: INFO/job] StatefulObject.set_state testfs-MDT0000 mounted->unmounted (intentional=True) 140623336525008
[2017-06-15 12:23:12,488: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on testfs-MDT0000 at severity 30
[2017-06-15 12:23:12,504: INFO/job] Job 84: StateChangeJob complete, setting state unmounted on testfs-MDT0000
[2017-06-15 12:23:12,504: INFO/job] Job 84: not_deleted=True
[2017-06-15 12:23:12,504: INFO/job_scheduler] Job 84: complete_job: Updating cache
[2017-06-15 12:23:12,504: INFO/job_scheduler] Job 84: locked_item 140623336525008 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-15 12:23:12,507: INFO/job_scheduler] Job 84 completing (errored=False, cancelled=False)
[2017-06-15 12:23:12,533: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 1 tasked)
[2017-06-15 12:23:12,758: INFO/job_scheduler] notify: Dropping update of 2 (testfs-MDT0000) because it has been updated since
[2017-06-15 12:23:12,758: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 1 tasked)
[2017-06-15 12:23:12,803: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 1 tasked)
[2017-06-15 12:23:14,138: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 1 tasked)
[2017-06-15 12:23:14,161: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 1 tasked)
[2017-06-15 12:23:14,532: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:23:14,532: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:23:14,532: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:23:14,532: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:23:14,532: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d76359bb-bdba-4291-93b4-3c3fa49df2ce
[2017-06-15 12:23:14,532: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d76359bb-bdba-4291-93b4-3c3fa49df2ce
[2017-06-15 12:23:14,542: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d76359bb-bdba-4291-93b4-3c3fa49df2ce
[2017-06-15 12:23:14,542: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d76359bb-bdba-4291-93b4-3c3fa49df2ce
[2017-06-15 12:23:14,543: INFO/job_scheduler] Job 85 finished 1 steps successfully
[2017-06-15 12:23:14,641: INFO/job] StatefulObject.set_state testfs-OST0000 mounted->unmounted (intentional=True) 140623739456784
[2017-06-15 12:23:14,646: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on testfs-OST0000 at severity 30
[2017-06-15 12:23:14,659: INFO/job] Job 85: StateChangeJob complete, setting state unmounted on testfs-OST0000
[2017-06-15 12:23:14,660: INFO/job] Job 85: not_deleted=True
[2017-06-15 12:23:14,660: INFO/job_scheduler] Job 85: complete_job: Updating cache
[2017-06-15 12:23:14,660: INFO/job_scheduler] Job 85: locked_item 140623739456784 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-15 12:23:14,676: INFO/job_scheduler] Job 85 completing (errored=False, cancelled=False)
[2017-06-15 12:23:14,730: INFO/job_scheduler] run_next: 1 runnable jobs of (4 pending, 0 tasked)
[2017-06-15 12:23:14,771: INFO/job_scheduler] Job 86: RunJobThread.run
[2017-06-15 12:23:14,772: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com unconfigure_target_ha {'uuid': u'14116028923112638812', 'ha_label': u'testfs-OST0000_181072', 'primary': False}
[2017-06-15 12:23:14,792: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f3dc1a77-5ac2-42aa-b1ae-a850c193136e
[2017-06-15 12:23:14,792: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f3dc1a77-5ac2-42aa-b1ae-a850c193136e
[2017-06-15 12:23:17,137: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-15 12:23:18,880: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-15 12:23:32,953: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:23:32,953: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:23:32,953: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:23:32,953: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:23:32,953: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f3dc1a77-5ac2-42aa-b1ae-a850c193136e
[2017-06-15 12:23:32,953: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f3dc1a77-5ac2-42aa-b1ae-a850c193136e
[2017-06-15 12:23:32,964: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f3dc1a77-5ac2-42aa-b1ae-a850c193136e
[2017-06-15 12:23:32,964: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f3dc1a77-5ac2-42aa-b1ae-a850c193136e
[2017-06-15 12:23:32,965: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unconfigure_target_ha {'uuid': u'14116028923112638812', 'ha_label': u'testfs-OST0000_181072', 'primary': True}
[2017-06-15 12:23:32,987: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9dbbe26c-08f9-49fe-b33b-e1157da9bfc8
[2017-06-15 12:23:32,987: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9dbbe26c-08f9-49fe-b33b-e1157da9bfc8
[2017-06-15 12:23:52,151: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:23:52,151: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:23:52,151: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:23:52,151: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:23:52,152: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9dbbe26c-08f9-49fe-b33b-e1157da9bfc8
[2017-06-15 12:23:52,152: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9dbbe26c-08f9-49fe-b33b-e1157da9bfc8
[2017-06-15 12:23:52,160: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9dbbe26c-08f9-49fe-b33b-e1157da9bfc8
[2017-06-15 12:23:52,160: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9dbbe26c-08f9-49fe-b33b-e1157da9bfc8
[2017-06-15 12:23:52,161: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com unconfigure_target_store {'uuid': u'14116028923112638812'}
[2017-06-15 12:23:52,174: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 118bdcf7-3d99-47d1-9f7d-5e2c554c254a
[2017-06-15 12:23:52,174: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 118bdcf7-3d99-47d1-9f7d-5e2c554c254a
[2017-06-15 12:23:52,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:23:52,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:23:52,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:23:52,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:23:52,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 118bdcf7-3d99-47d1-9f7d-5e2c554c254a
[2017-06-15 12:23:52,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 118bdcf7-3d99-47d1-9f7d-5e2c554c254a
[2017-06-15 12:23:52,289: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 118bdcf7-3d99-47d1-9f7d-5e2c554c254a
[2017-06-15 12:23:52,289: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 118bdcf7-3d99-47d1-9f7d-5e2c554c254a
[2017-06-15 12:23:52,290: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unconfigure_target_store {'uuid': u'14116028923112638812'}
[2017-06-15 12:23:52,303: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b0fc8570-f164-435e-9dc2-6dfc18ea8b4e
[2017-06-15 12:23:52,303: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b0fc8570-f164-435e-9dc2-6dfc18ea8b4e
[2017-06-15 12:23:52,446: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:23:52,446: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:23:52,447: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:23:52,447: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:23:52,447: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b0fc8570-f164-435e-9dc2-6dfc18ea8b4e
[2017-06-15 12:23:52,447: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b0fc8570-f164-435e-9dc2-6dfc18ea8b4e
[2017-06-15 12:23:52,475: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b0fc8570-f164-435e-9dc2-6dfc18ea8b4e
[2017-06-15 12:23:52,475: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b0fc8570-f164-435e-9dc2-6dfc18ea8b4e
[2017-06-15 12:23:52,477: INFO/job_scheduler] Job 86 finished 4 steps successfully
[2017-06-15 12:23:52,493: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.target.ManagedTarget'> 3
[2017-06-15 12:23:52,493: INFO/job] StatefulObject.set_state testfs-OST0000 unmounted->removed (intentional=True) 140623739456784
[2017-06-15 12:23:52,534: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 20
[2017-06-15 12:23:52,542: INFO/job] Job 86: StateChangeJob complete, setting state removed on testfs-OST0000
[2017-06-15 12:23:52,542: INFO/job] Job 86: not_deleted=None
[2017-06-15 12:23:52,542: INFO/job_scheduler] Job 86: complete_job: Updating cache
[2017-06-15 12:23:52,542: INFO/job_scheduler] Job 86: locked_item 140623739456784 <class 'chroma_core.models.target.ManagedTarget'> False None
[2017-06-15 12:23:52,543: INFO/job_scheduler] Job 86 completing (errored=False, cancelled=False)
[2017-06-15 12:23:52,563: INFO/job_scheduler] run_next: 1 runnable jobs of (3 pending, 0 tasked)
[2017-06-15 12:23:52,676: INFO/job_scheduler] Job 87: open DB connection during completion
[2017-06-15 12:23:52,680: INFO/job] StatefulObject.set_state testfs unavailable->stopped (intentional=True) 140623538072144
[2017-06-15 12:23:52,692: INFO/job] Job 87: StateChangeJob complete, setting state stopped on testfs
[2017-06-15 12:23:52,692: INFO/job] Job 87: not_deleted=True
[2017-06-15 12:23:52,692: INFO/job_scheduler] Job 87: complete_job: Updating cache
[2017-06-15 12:23:52,692: INFO/job_scheduler] Job 87: locked_item 140623538072144 <class 'chroma_core.models.filesystem.ManagedFilesystem'> False True
[2017-06-15 12:23:52,708: INFO/job_scheduler] Job 87 completing (errored=False, cancelled=False)
[2017-06-15 12:23:52,730: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 0 tasked)
[2017-06-15 12:23:52,771: INFO/job_scheduler] Job 88: RunJobThread.run
[2017-06-15 12:23:52,772: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com unconfigure_target_ha {'uuid': u'2967121584722494284', 'ha_label': u'testfs-MDT0000_69ad68', 'primary': False}
[2017-06-15 12:23:52,820: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d31dfa50-b5c9-4bf9-8070-a41e53b4768d
[2017-06-15 12:23:52,820: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d31dfa50-b5c9-4bf9-8070-a41e53b4768d
[2017-06-15 12:24:11,013: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:24:11,013: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:24:11,013: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:24:11,013: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:24:11,014: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d31dfa50-b5c9-4bf9-8070-a41e53b4768d
[2017-06-15 12:24:11,014: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d31dfa50-b5c9-4bf9-8070-a41e53b4768d
[2017-06-15 12:24:11,040: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d31dfa50-b5c9-4bf9-8070-a41e53b4768d
[2017-06-15 12:24:11,040: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d31dfa50-b5c9-4bf9-8070-a41e53b4768d
[2017-06-15 12:24:11,045: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com unconfigure_target_ha {'uuid': u'2967121584722494284', 'ha_label': u'testfs-MDT0000_69ad68', 'primary': True}
[2017-06-15 12:24:11,059: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2929efa5-ec6c-45d7-b5be-c0ee587aa376
[2017-06-15 12:24:11,059: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2929efa5-ec6c-45d7-b5be-c0ee587aa376
[2017-06-15 12:24:30,236: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:24:30,236: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:24:30,236: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:24:30,236: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:24:30,236: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2929efa5-ec6c-45d7-b5be-c0ee587aa376
[2017-06-15 12:24:30,236: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2929efa5-ec6c-45d7-b5be-c0ee587aa376
[2017-06-15 12:24:30,278: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2929efa5-ec6c-45d7-b5be-c0ee587aa376
[2017-06-15 12:24:30,278: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2929efa5-ec6c-45d7-b5be-c0ee587aa376
[2017-06-15 12:24:30,279: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com unconfigure_target_store {'uuid': u'2967121584722494284'}
[2017-06-15 12:24:30,292: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 535364f3-c434-40d4-807b-8d35c5b11af2
[2017-06-15 12:24:30,292: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 535364f3-c434-40d4-807b-8d35c5b11af2
[2017-06-15 12:24:30,369: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:24:30,369: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:24:30,370: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:24:30,370: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:24:30,370: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 535364f3-c434-40d4-807b-8d35c5b11af2
[2017-06-15 12:24:30,370: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 535364f3-c434-40d4-807b-8d35c5b11af2
[2017-06-15 12:24:30,411: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 535364f3-c434-40d4-807b-8d35c5b11af2
[2017-06-15 12:24:30,411: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 535364f3-c434-40d4-807b-8d35c5b11af2
[2017-06-15 12:24:30,412: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com unconfigure_target_store {'uuid': u'2967121584722494284'}
[2017-06-15 12:24:30,424: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6d9c8374-2385-44f6-a0fa-767ca4cc87f9
[2017-06-15 12:24:30,424: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6d9c8374-2385-44f6-a0fa-767ca4cc87f9
[2017-06-15 12:24:30,491: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:24:30,491: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:24:30,491: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:24:30,491: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:24:30,491: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6d9c8374-2385-44f6-a0fa-767ca4cc87f9
[2017-06-15 12:24:30,491: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6d9c8374-2385-44f6-a0fa-767ca4cc87f9
[2017-06-15 12:24:30,540: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6d9c8374-2385-44f6-a0fa-767ca4cc87f9
[2017-06-15 12:24:30,540: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6d9c8374-2385-44f6-a0fa-767ca4cc87f9
[2017-06-15 12:24:30,541: INFO/job_scheduler] Job 88 finished 4 steps successfully
[2017-06-15 12:24:30,560: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.target.ManagedTarget'> 2
[2017-06-15 12:24:30,560: INFO/job] StatefulObject.set_state testfs-MDT0000 unmounted->removed (intentional=True) 140623336525008
[2017-06-15 12:24:30,599: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm8.lotus.hpdd.lab.intel.com at severity 20
[2017-06-15 12:24:30,608: INFO/job] Job 88: StateChangeJob complete, setting state removed on testfs-MDT0000
[2017-06-15 12:24:30,609: INFO/job] Job 88: not_deleted=None
[2017-06-15 12:24:30,609: INFO/job_scheduler] Job 88: complete_job: Updating cache
[2017-06-15 12:24:30,609: INFO/job_scheduler] Job 88: locked_item 140623336525008 <class 'chroma_core.models.target.ManagedTarget'> False None
[2017-06-15 12:24:30,609: INFO/job_scheduler] Job 88 completing (errored=False, cancelled=False)
[2017-06-15 12:24:30,646: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-15 12:24:30,715: INFO/job_scheduler] Job 89: RunJobThread.run
[2017-06-15 12:24:30,716: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com purge_configuration {'mgs_device_path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk1/MGS', 'mgs_device_type': 'zfs', 'filesystem_name': u'testfs'}
[2017-06-15 12:24:30,728: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e68e4546-9597-4e29-b419-fd2d3578a467
[2017-06-15 12:24:30,728: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e68e4546-9597-4e29-b419-fd2d3578a467
[2017-06-15 12:24:31,817: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:24:31,817: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:24:31,818: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:24:31,818: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:24:31,818: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e68e4546-9597-4e29-b419-fd2d3578a467
[2017-06-15 12:24:31,818: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e68e4546-9597-4e29-b419-fd2d3578a467
[2017-06-15 12:24:31,845: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e68e4546-9597-4e29-b419-fd2d3578a467
[2017-06-15 12:24:31,845: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e68e4546-9597-4e29-b419-fd2d3578a467
[2017-06-15 12:24:31,845: INFO/job_scheduler] Job 89 finished 1 steps successfully
[2017-06-15 12:24:31,895: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.filesystem.ManagedFilesystem'> 1
[2017-06-15 12:24:31,896: INFO/job] StatefulObject.set_state testfs stopped->removed (intentional=True) 140623538072144
[2017-06-15 12:24:31,898: INFO/job] Job 89: StateChangeJob complete, setting state removed on testfs
[2017-06-15 12:24:31,898: INFO/job] Job 89: not_deleted=None
[2017-06-15 12:24:31,898: INFO/job_scheduler] Job 89: complete_job: Updating cache
[2017-06-15 12:24:31,898: INFO/job_scheduler] Job 89: locked_item 140623538072144 <class 'chroma_core.models.filesystem.ManagedFilesystem'> False None
[2017-06-15 12:24:31,898: INFO/job_scheduler] Job 89: locked_item 140624142268816 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-15 12:24:31,901: INFO/job_scheduler] Job 89 completing (errored=False, cancelled=False)
[2017-06-15 12:24:31,978: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:24:32,869: INFO/job_scheduler] notify: Dropping update of 1 (MGS) because it has been updated since
[2017-06-15 12:24:32,869: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:24:35,358: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:24:44,401: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:24:57,411: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:24:57,425: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:25:12,645: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 12:25:12,648: INFO/command_plan] set_state: <class 'chroma_core.models.target.ManagedTarget'>-1 transitioning from mounted to removed
[2017-06-15 12:25:12,683: INFO/command_plan] Created command 32 (Remove target MGS from the filesystem
*** WARNING *** this is destructive to your filesystem and irreversible) with 2 jobs
[2017-06-15 12:25:12,712: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 0 tasked)
[2017-06-15 12:25:12,781: INFO/job_scheduler] Job 90: RunJobThread.run
[2017-06-15 12:25:12,782: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com stop_target {'ha_label': u'MGS_9e39ca'}
[2017-06-15 12:25:12,828: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d602e9ee-6bb5-44d5-8fc4-503515006048
[2017-06-15 12:25:12,828: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d602e9ee-6bb5-44d5-8fc4-503515006048
[2017-06-15 12:25:29,000: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:25:29,000: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:25:29,001: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:25:29,001: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:25:29,001: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d602e9ee-6bb5-44d5-8fc4-503515006048
[2017-06-15 12:25:29,001: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d602e9ee-6bb5-44d5-8fc4-503515006048
[2017-06-15 12:25:29,001: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d602e9ee-6bb5-44d5-8fc4-503515006048
[2017-06-15 12:25:29,001: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d602e9ee-6bb5-44d5-8fc4-503515006048
[2017-06-15 12:25:29,003: INFO/job_scheduler] Job 90 finished 1 steps successfully
[2017-06-15 12:25:29,135: INFO/job] StatefulObject.set_state MGS mounted->unmounted (intentional=True) 140623739372624
[2017-06-15 12:25:29,143: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on MGS at severity 30
[2017-06-15 12:25:29,147: INFO/job] Job 90: StateChangeJob complete, setting state unmounted on MGS
[2017-06-15 12:25:29,147: INFO/job] Job 90: not_deleted=True
[2017-06-15 12:25:29,147: INFO/job_scheduler] Job 90: complete_job: Updating cache
[2017-06-15 12:25:29,147: INFO/job_scheduler] Job 90: locked_item 140623739372624 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-15 12:25:29,152: INFO/job_scheduler] Job 90 completing (errored=False, cancelled=False)
[2017-06-15 12:25:29,202: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-15 12:25:29,261: INFO/job_scheduler] Job 91: RunJobThread.run
[2017-06-15 12:25:29,261: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com unconfigure_target_ha {'uuid': u'18405115617293209603', 'ha_label': u'MGS_9e39ca', 'primary': False}
[2017-06-15 12:25:29,290: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 53f79405-e81c-4595-90a6-68e407ea36f3
[2017-06-15 12:25:29,290: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 53f79405-e81c-4595-90a6-68e407ea36f3
[2017-06-15 12:25:33,009: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 12:25:35,466: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 12:25:38,060: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 12:25:38,070: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 12:25:47,441: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:25:47,441: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:25:47,441: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:25:47,441: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:25:47,441: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 53f79405-e81c-4595-90a6-68e407ea36f3
[2017-06-15 12:25:47,441: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 53f79405-e81c-4595-90a6-68e407ea36f3
[2017-06-15 12:25:47,460: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 53f79405-e81c-4595-90a6-68e407ea36f3
[2017-06-15 12:25:47,460: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 53f79405-e81c-4595-90a6-68e407ea36f3
[2017-06-15 12:25:47,462: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com unconfigure_target_ha {'uuid': u'18405115617293209603', 'ha_label': u'MGS_9e39ca', 'primary': True}
[2017-06-15 12:25:47,475: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b5053276-1cae-412d-a7a4-fe15a5fa9fca
[2017-06-15 12:25:47,475: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b5053276-1cae-412d-a7a4-fe15a5fa9fca
[2017-06-15 12:26:06,640: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:26:06,640: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:26:06,640: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:26:06,640: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:26:06,641: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b5053276-1cae-412d-a7a4-fe15a5fa9fca
[2017-06-15 12:26:06,641: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b5053276-1cae-412d-a7a4-fe15a5fa9fca
[2017-06-15 12:26:06,644: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b5053276-1cae-412d-a7a4-fe15a5fa9fca
[2017-06-15 12:26:06,644: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b5053276-1cae-412d-a7a4-fe15a5fa9fca
[2017-06-15 12:26:06,645: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com unconfigure_target_store {'uuid': u'18405115617293209603'}
[2017-06-15 12:26:06,662: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4221e815-9b33-49b5-a7ad-eb115175c4f6
[2017-06-15 12:26:06,662: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4221e815-9b33-49b5-a7ad-eb115175c4f6
[2017-06-15 12:26:06,717: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:26:06,717: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:26:06,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:26:06,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:26:06,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4221e815-9b33-49b5-a7ad-eb115175c4f6
[2017-06-15 12:26:06,719: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4221e815-9b33-49b5-a7ad-eb115175c4f6
[2017-06-15 12:26:06,731: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4221e815-9b33-49b5-a7ad-eb115175c4f6
[2017-06-15 12:26:06,731: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4221e815-9b33-49b5-a7ad-eb115175c4f6
[2017-06-15 12:26:06,734: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com unconfigure_target_store {'uuid': u'18405115617293209603'}
[2017-06-15 12:26:06,748: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e2dfe513-e05c-4720-ae1e-44f9ad041f5e
[2017-06-15 12:26:06,748: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e2dfe513-e05c-4720-ae1e-44f9ad041f5e
[2017-06-15 12:26:06,864: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:26:06,864: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:26:06,864: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:26:06,864: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:26:06,865: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e2dfe513-e05c-4720-ae1e-44f9ad041f5e
[2017-06-15 12:26:06,865: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e2dfe513-e05c-4720-ae1e-44f9ad041f5e
[2017-06-15 12:26:06,872: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e2dfe513-e05c-4720-ae1e-44f9ad041f5e
[2017-06-15 12:26:06,872: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e2dfe513-e05c-4720-ae1e-44f9ad041f5e
[2017-06-15 12:26:06,872: INFO/job_scheduler] Job 91 finished 4 steps successfully
[2017-06-15 12:26:06,911: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.target.ManagedTarget'> 1
[2017-06-15 12:26:06,911: INFO/job] StatefulObject.set_state MGS unmounted->removed (intentional=True) 140623739372624
[2017-06-15 12:26:06,948: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 20
[2017-06-15 12:26:06,961: INFO/job] Job 91: StateChangeJob complete, setting state removed on MGS
[2017-06-15 12:26:06,961: INFO/job] Job 91: not_deleted=None
[2017-06-15 12:26:06,962: INFO/job_scheduler] Job 91: complete_job: Updating cache
[2017-06-15 12:26:06,962: INFO/job_scheduler] Job 91: locked_item 140623739372624 <class 'chroma_core.models.target.ManagedTarget'> False None
[2017-06-15 12:26:06,962: INFO/job_scheduler] Job 91 completing (errored=False, cancelled=False)
[2017-06-15 12:26:07,005: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 12:28:06,496: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 12:28:06,503: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-1 transitioning from managed to removed
[2017-06-15 12:28:06,623: INFO/command_plan] Created command 33 (Remove host lotus-32vm5.lotus.hpdd.lab.intel.com from configuration) with 9 jobs
[2017-06-15 12:28:06,655: INFO/job_scheduler] run_next: 4 runnable jobs of (9 pending, 0 tasked)
[2017-06-15 12:28:06,702: INFO/job_scheduler] Job 92: RunJobThread.run
[2017-06-15 12:28:06,703: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-06-15 12:28:06,718: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 43a96c86-8e37-4497-a1bc-4bb6ef3de5ff
[2017-06-15 12:28:06,718: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 43a96c86-8e37-4497-a1bc-4bb6ef3de5ff
[2017-06-15 12:28:06,760: INFO/job_scheduler] Job 93: RunJobThread.run
[2017-06-15 12:28:06,760: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unconfigure_rsyslog {}
[2017-06-15 12:28:06,796: INFO/job_scheduler] Job 94: RunJobThread.run
[2017-06-15 12:28:06,796: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_lnet {}
[2017-06-15 12:28:06,820: INFO/job_scheduler] Job 95: RunJobThread.run
[2017-06-15 12:28:06,820: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unconfigure_ntp {}
[2017-06-15 12:28:06,838: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 50efa372-84ee-4e04-a121-ef08b89dee5a
[2017-06-15 12:28:06,838: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 50efa372-84ee-4e04-a121-ef08b89dee5a
[2017-06-15 12:28:06,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ba66a4cf-423b-421b-803e-f06f1371a142
[2017-06-15 12:28:06,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ba66a4cf-423b-421b-803e-f06f1371a142
[2017-06-15 12:28:06,946: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 24300eab-f7be-4fee-82d6-533c8b351cdd
[2017-06-15 12:28:06,946: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 24300eab-f7be-4fee-82d6-533c8b351cdd
[2017-06-15 12:28:08,021: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-06-15 12:28:08,038: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-06-15 12:28:13,041: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:13,041: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:13,041: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:13,041: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:13,041: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 24300eab-f7be-4fee-82d6-533c8b351cdd
[2017-06-15 12:28:13,041: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 24300eab-f7be-4fee-82d6-533c8b351cdd
[2017-06-15 12:28:13,063: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 24300eab-f7be-4fee-82d6-533c8b351cdd
[2017-06-15 12:28:13,063: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 24300eab-f7be-4fee-82d6-533c8b351cdd
[2017-06-15 12:28:13,064: INFO/job_scheduler] Job 95 finished 1 steps successfully
[2017-06-15 12:28:13,247: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration configured->unconfigured (intentional=True) 140623537936976
[2017-06-15 12:28:13,250: INFO/job] Job 95: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration
[2017-06-15 12:28:13,251: INFO/job] Job 95: not_deleted=True
[2017-06-15 12:28:13,251: INFO/job_scheduler] Job 95: complete_job: Updating cache
[2017-06-15 12:28:13,251: INFO/job_scheduler] Job 95: locked_item 140623537936976 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-06-15 12:28:13,253: INFO/job_scheduler] Job 95 completing (errored=False, cancelled=False)
[2017-06-15 12:28:13,264: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 3 tasked)
[2017-06-15 12:28:13,957: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:13,957: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:13,957: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:13,957: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:13,958: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 50efa372-84ee-4e04-a121-ef08b89dee5a
[2017-06-15 12:28:13,958: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 50efa372-84ee-4e04-a121-ef08b89dee5a
[2017-06-15 12:28:14,006: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 50efa372-84ee-4e04-a121-ef08b89dee5a
[2017-06-15 12:28:14,006: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 50efa372-84ee-4e04-a121-ef08b89dee5a
[2017-06-15 12:28:14,007: INFO/job_scheduler] Job 93 finished 1 steps successfully
[2017-06-15 12:28:14,122: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration configured->unconfigured (intentional=True) 140623538077200
[2017-06-15 12:28:14,126: INFO/job] Job 93: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-06-15 12:28:14,126: INFO/job] Job 93: not_deleted=True
[2017-06-15 12:28:14,126: INFO/job_scheduler] Job 93: complete_job: Updating cache
[2017-06-15 12:28:14,126: INFO/job_scheduler] Job 93: locked_item 140623538077200 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-06-15 12:28:14,129: INFO/job_scheduler] Job 93 completing (errored=False, cancelled=False)
[2017-06-15 12:28:14,139: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 2 tasked)
[2017-06-15 12:28:14,802: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:14,802: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:14,802: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:14,802: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:14,802: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 43a96c86-8e37-4497-a1bc-4bb6ef3de5ff
[2017-06-15 12:28:14,802: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 43a96c86-8e37-4497-a1bc-4bb6ef3de5ff
[2017-06-15 12:28:14,840: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 43a96c86-8e37-4497-a1bc-4bb6ef3de5ff
[2017-06-15 12:28:14,840: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 43a96c86-8e37-4497-a1bc-4bb6ef3de5ff
[2017-06-15 12:28:14,841: INFO/job_scheduler] Job 92 finished 1 steps successfully
[2017-06-15 12:28:14,888: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration started->stopped (intentional=True) 140623739732368
[2017-06-15 12:28:14,895: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-06-15 12:28:14,904: INFO/job] Job 92: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-15 12:28:14,904: INFO/job] Job 92: not_deleted=True
[2017-06-15 12:28:14,904: INFO/job_scheduler] Job 92: complete_job: Updating cache
[2017-06-15 12:28:14,905: INFO/job_scheduler] Job 92: locked_item 140623739732368 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-15 12:28:14,912: INFO/job_scheduler] Job 92 completing (errored=False, cancelled=False)
[2017-06-15 12:28:14,930: INFO/job_scheduler] run_next: 1 runnable jobs of (5 pending, 1 tasked)
[2017-06-15 12:28:14,936: INFO/job_scheduler] Job 97: RunJobThread.run
[2017-06-15 12:28:14,937: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_corosync2 {}
[2017-06-15 12:28:14,959: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 27f53072-f67b-4186-a6b9-29aa277f7982
[2017-06-15 12:28:14,959: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 27f53072-f67b-4186-a6b9-29aa277f7982
[2017-06-15 12:28:16,636: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 12:28:16,645: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-2 transitioning from managed to removed
[2017-06-15 12:28:16,781: INFO/command_plan] Created command 34 (Remove host lotus-32vm6.lotus.hpdd.lab.intel.com from configuration) with 9 jobs
[2017-06-15 12:28:16,808: INFO/job_scheduler] run_next: 4 runnable jobs of (13 pending, 2 tasked)
[2017-06-15 12:28:16,827: INFO/job_scheduler] Job 101: RunJobThread.run
[2017-06-15 12:28:16,828: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com unconfigure_rsyslog {}
[2017-06-15 12:28:16,840: INFO/job_scheduler] Job 102: RunJobThread.run
[2017-06-15 12:28:16,840: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com stop_lnet {}
[2017-06-15 12:28:16,842: INFO/job_scheduler] Job 103: RunJobThread.run
[2017-06-15 12:28:16,842: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-06-15 12:28:16,844: INFO/job_scheduler] Job 104: RunJobThread.run
[2017-06-15 12:28:16,845: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com unconfigure_ntp {}
[2017-06-15 12:28:16,855: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 14105b84-4b20-472e-a1af-4ace201bde58
[2017-06-15 12:28:16,855: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 14105b84-4b20-472e-a1af-4ace201bde58
[2017-06-15 12:28:16,870: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c22ee3c0-dd07-4c8a-8e50-29df9f4e7d55
[2017-06-15 12:28:16,870: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c22ee3c0-dd07-4c8a-8e50-29df9f4e7d55
[2017-06-15 12:28:16,887: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f756d30d-5820-4eec-9416-15a36644e702
[2017-06-15 12:28:16,887: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f756d30d-5820-4eec-9416-15a36644e702
[2017-06-15 12:28:16,905: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 69c531f2-4d4f-40e4-89b8-f6228e124907
[2017-06-15 12:28:16,905: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 69c531f2-4d4f-40e4-89b8-f6228e124907
[2017-06-15 12:28:21,968: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:21,968: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:21,968: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:21,968: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:21,968: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f756d30d-5820-4eec-9416-15a36644e702
[2017-06-15 12:28:21,968: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f756d30d-5820-4eec-9416-15a36644e702
[2017-06-15 12:28:22,005: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f756d30d-5820-4eec-9416-15a36644e702
[2017-06-15 12:28:22,005: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f756d30d-5820-4eec-9416-15a36644e702
[2017-06-15 12:28:22,007: INFO/job_scheduler] Job 104 finished 1 steps successfully
[2017-06-15 12:28:22,096: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration configured->unconfigured (intentional=True) 140623538038736
[2017-06-15 12:28:22,100: INFO/job] Job 104: StateChangeJob complete, setting state unconfigured on lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration
[2017-06-15 12:28:22,100: INFO/job] Job 104: not_deleted=True
[2017-06-15 12:28:22,100: INFO/job_scheduler] Job 104: complete_job: Updating cache
[2017-06-15 12:28:22,100: INFO/job_scheduler] Job 104: locked_item 140623538038736 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-06-15 12:28:22,103: INFO/job_scheduler] Job 104 completing (errored=False, cancelled=False)
[2017-06-15 12:28:22,112: INFO/job_scheduler] run_next: 0 runnable jobs of (9 pending, 5 tasked)
[2017-06-15 12:28:23,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:23,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:23,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:23,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:23,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 27f53072-f67b-4186-a6b9-29aa277f7982
[2017-06-15 12:28:23,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 27f53072-f67b-4186-a6b9-29aa277f7982
[2017-06-15 12:28:23,126: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 27f53072-f67b-4186-a6b9-29aa277f7982
[2017-06-15 12:28:23,126: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 27f53072-f67b-4186-a6b9-29aa277f7982
[2017-06-15 12:28:23,126: INFO/job_scheduler] Job 97 finished 1 steps successfully
[2017-06-15 12:28:23,154: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration started->stopped (intentional=True) 140623537934800
[2017-06-15 12:28:23,162: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-06-15 12:28:23,175: INFO/job] Job 97: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 12:28:23,176: INFO/job] Job 97: not_deleted=True
[2017-06-15 12:28:23,177: INFO/job_scheduler] Job 97: complete_job: Updating cache
[2017-06-15 12:28:23,177: INFO/job_scheduler] Job 97: locked_item 140623537934800 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-15 12:28:23,192: INFO/job_scheduler] Job 97 completing (errored=False, cancelled=False)
[2017-06-15 12:28:23,279: INFO/job_scheduler] run_next: 1 runnable jobs of (9 pending, 4 tasked)
[2017-06-15 12:28:23,283: INFO/job_scheduler] Job 99: RunJobThread.run
[2017-06-15 12:28:23,284: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unconfigure_corosync2 {'mcast_port': 65087, 'host_fqdn': u'lotus-32vm5.lotus.hpdd.lab.intel.com'}
[2017-06-15 12:28:23,303: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 709aa845-ab33-48f5-a21a-340f31e59e13
[2017-06-15 12:28:23,303: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 709aa845-ab33-48f5-a21a-340f31e59e13
[2017-06-15 12:28:24,008: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:24,008: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:24,009: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:24,009: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:24,009: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 14105b84-4b20-472e-a1af-4ace201bde58
[2017-06-15 12:28:24,009: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 14105b84-4b20-472e-a1af-4ace201bde58
[2017-06-15 12:28:24,026: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 14105b84-4b20-472e-a1af-4ace201bde58
[2017-06-15 12:28:24,026: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 14105b84-4b20-472e-a1af-4ace201bde58
[2017-06-15 12:28:24,027: INFO/job_scheduler] Job 101 finished 1 steps successfully
[2017-06-15 12:28:24,129: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration configured->unconfigured (intentional=True) 140623538039120
[2017-06-15 12:28:24,132: INFO/job] Job 101: StateChangeJob complete, setting state unconfigured on lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-06-15 12:28:24,132: INFO/job] Job 101: not_deleted=True
[2017-06-15 12:28:24,132: INFO/job_scheduler] Job 101: complete_job: Updating cache
[2017-06-15 12:28:24,132: INFO/job_scheduler] Job 101: locked_item 140623538039120 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-06-15 12:28:24,135: INFO/job_scheduler] Job 101 completing (errored=False, cancelled=False)
[2017-06-15 12:28:24,145: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-06-15 12:28:24,231: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:24,231: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:24,231: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:24,231: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:24,232: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 69c531f2-4d4f-40e4-89b8-f6228e124907
[2017-06-15 12:28:24,232: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 69c531f2-4d4f-40e4-89b8-f6228e124907
[2017-06-15 12:28:24,252: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-06-15 12:28:24,266: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 4 tasked)
[2017-06-15 12:28:24,276: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 69c531f2-4d4f-40e4-89b8-f6228e124907
[2017-06-15 12:28:24,276: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 69c531f2-4d4f-40e4-89b8-f6228e124907
[2017-06-15 12:28:24,276: INFO/job_scheduler] Job 103 finished 1 steps successfully
[2017-06-15 12:28:24,337: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration started->stopped (intentional=True) 140623739734032
[2017-06-15 12:28:24,344: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-06-15 12:28:24,347: INFO/job] Job 103: StateChangeJob complete, setting state stopped on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-15 12:28:24,347: INFO/job] Job 103: not_deleted=True
[2017-06-15 12:28:24,347: INFO/job_scheduler] Job 103: complete_job: Updating cache
[2017-06-15 12:28:24,347: INFO/job_scheduler] Job 103: locked_item 140623739734032 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-15 12:28:24,356: INFO/job_scheduler] Job 103 completing (errored=False, cancelled=False)
[2017-06-15 12:28:24,379: INFO/job_scheduler] run_next: 1 runnable jobs of (8 pending, 3 tasked)
[2017-06-15 12:28:24,385: INFO/job_scheduler] Job 106: RunJobThread.run
[2017-06-15 12:28:24,387: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com stop_corosync2 {}
[2017-06-15 12:28:24,409: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 979c362a-1a5f-4afe-b6a1-6a465ac2d0fa
[2017-06-15 12:28:24,409: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 979c362a-1a5f-4afe-b6a1-6a465ac2d0fa
[2017-06-15 12:28:25,173: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-06-15 12:28:25,196: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 4 tasked)
[2017-06-15 12:28:26,100: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:26,100: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:26,101: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:26,101: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:26,101: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ba66a4cf-423b-421b-803e-f06f1371a142
[2017-06-15 12:28:26,101: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ba66a4cf-423b-421b-803e-f06f1371a142
[2017-06-15 12:28:26,120: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ba66a4cf-423b-421b-803e-f06f1371a142
[2017-06-15 12:28:26,120: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ba66a4cf-423b-421b-803e-f06f1371a142
[2017-06-15 12:28:26,121: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-15 12:28:26,136: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 247a0cdf-d899-4be8-abf4-e8bda3fef485
[2017-06-15 12:28:26,136: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 247a0cdf-d899-4be8-abf4-e8bda3fef485
[2017-06-15 12:28:26,235: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:26,235: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:26,235: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:26,235: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:26,235: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 247a0cdf-d899-4be8-abf4-e8bda3fef485
[2017-06-15 12:28:26,235: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 247a0cdf-d899-4be8-abf4-e8bda3fef485
[2017-06-15 12:28:26,251: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 247a0cdf-d899-4be8-abf4-e8bda3fef485
[2017-06-15 12:28:26,251: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 247a0cdf-d899-4be8-abf4-e8bda3fef485
[2017-06-15 12:28:26,359: INFO/job_scheduler] Job 94 finished 2 steps successfully
[2017-06-15 12:28:26,405: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_up->lnet_down (intentional=True) 140623537940688
[2017-06-15 12:28:26,413: INFO/job] Job 94: StateChangeJob complete, setting state lnet_down on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-15 12:28:26,413: INFO/job] Job 94: not_deleted=True
[2017-06-15 12:28:26,413: INFO/job_scheduler] Job 94: complete_job: Updating cache
[2017-06-15 12:28:26,413: INFO/job_scheduler] Job 94: locked_item 140623537940688 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-15 12:28:26,418: INFO/job_scheduler] Job 94 completing (errored=False, cancelled=False)
[2017-06-15 12:28:26,450: INFO/job_scheduler] run_next: 1 runnable jobs of (7 pending, 3 tasked)
[2017-06-15 12:28:26,489: INFO/job_scheduler] Job 96: RunJobThread.run
[2017-06-15 12:28:26,490: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unload_lnet {}
[2017-06-15 12:28:26,534: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 066b61eb-e55e-4974-b558-37a83af820a1
[2017-06-15 12:28:26,534: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 066b61eb-e55e-4974-b558-37a83af820a1
[2017-06-15 12:28:27,659: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 12:28:27,680: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-3 transitioning from managed to removed
[2017-06-15 12:28:27,816: INFO/command_plan] Created command 35 (Remove host lotus-32vm7.lotus.hpdd.lab.intel.com from configuration) with 9 jobs
[2017-06-15 12:28:27,998: INFO/job_scheduler] run_next: 4 runnable jobs of (15 pending, 4 tasked)
[2017-06-15 12:28:28,066: INFO/job_scheduler] Job 110: RunJobThread.run
[2017-06-15 12:28:28,067: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com unconfigure_ntp {}
[2017-06-15 12:28:28,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3261a83b-b22f-43d3-82ee-c6247075aec5
[2017-06-15 12:28:28,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3261a83b-b22f-43d3-82ee-c6247075aec5
[2017-06-15 12:28:28,088: INFO/job_scheduler] Job 111: RunJobThread.run
[2017-06-15 12:28:28,089: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com stop_lnet {}
[2017-06-15 12:28:28,114: INFO/job_scheduler] Job 112: RunJobThread.run
[2017-06-15 12:28:28,117: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3c101a65-2dcd-499b-a58f-013aafeb57a2
[2017-06-15 12:28:28,117: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3c101a65-2dcd-499b-a58f-013aafeb57a2
[2017-06-15 12:28:28,124: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-06-15 12:28:28,118: INFO/job_scheduler] Job 113: RunJobThread.run
[2017-06-15 12:28:28,135: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com unconfigure_rsyslog {}
[2017-06-15 12:28:28,183: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 31f02c0f-ea1d-4eec-bc34-10d7f1a66ca6
[2017-06-15 12:28:28,183: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 31f02c0f-ea1d-4eec-bc34-10d7f1a66ca6
[2017-06-15 12:28:28,203: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 00c50aa5-42cc-4602-a19e-0894bdc04766
[2017-06-15 12:28:28,203: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 00c50aa5-42cc-4602-a19e-0894bdc04766
[2017-06-15 12:28:28,637: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:28,637: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:28,638: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:28,638: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:28,638: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 066b61eb-e55e-4974-b558-37a83af820a1
[2017-06-15 12:28:28,638: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 066b61eb-e55e-4974-b558-37a83af820a1
[2017-06-15 12:28:28,664: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 066b61eb-e55e-4974-b558-37a83af820a1
[2017-06-15 12:28:28,664: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 066b61eb-e55e-4974-b558-37a83af820a1
[2017-06-15 12:28:28,665: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-15 12:28:28,686: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 85840122-0f0b-4f92-899e-41bf1fdb8d9a
[2017-06-15 12:28:28,686: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 85840122-0f0b-4f92-899e-41bf1fdb8d9a
[2017-06-15 12:28:28,763: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:28,763: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:28,763: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:28,763: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:28,764: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 85840122-0f0b-4f92-899e-41bf1fdb8d9a
[2017-06-15 12:28:28,764: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 85840122-0f0b-4f92-899e-41bf1fdb8d9a
[2017-06-15 12:28:28,804: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 85840122-0f0b-4f92-899e-41bf1fdb8d9a
[2017-06-15 12:28:28,804: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 85840122-0f0b-4f92-899e-41bf1fdb8d9a
[2017-06-15 12:28:28,897: INFO/job_scheduler] Job 96 finished 2 steps successfully
[2017-06-15 12:28:28,944: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_unloaded (intentional=True) 140623537940688
[2017-06-15 12:28:28,949: INFO/job] Job 96: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-15 12:28:28,950: INFO/job] Job 96: not_deleted=True
[2017-06-15 12:28:28,950: INFO/job_scheduler] Job 96: complete_job: Updating cache
[2017-06-15 12:28:28,950: INFO/job_scheduler] Job 96: locked_item 140623537940688 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-15 12:28:28,952: INFO/job_scheduler] Job 96 completing (errored=False, cancelled=False)
[2017-06-15 12:28:28,970: INFO/job_scheduler] run_next: 1 runnable jobs of (11 pending, 7 tasked)
[2017-06-15 12:28:28,975: INFO/job_scheduler] Job 98: RunJobThread.run
[2017-06-15 12:28:28,975: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unconfigure_lnet {}
[2017-06-15 12:28:28,990: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0906c7ef-d39a-4b90-a3d6-85f98c3086f0
[2017-06-15 12:28:28,990: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0906c7ef-d39a-4b90-a3d6-85f98c3086f0
[2017-06-15 12:28:29,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:29,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:29,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:29,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:29,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0906c7ef-d39a-4b90-a3d6-85f98c3086f0
[2017-06-15 12:28:29,088: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0906c7ef-d39a-4b90-a3d6-85f98c3086f0
[2017-06-15 12:28:29,110: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0906c7ef-d39a-4b90-a3d6-85f98c3086f0
[2017-06-15 12:28:29,110: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0906c7ef-d39a-4b90-a3d6-85f98c3086f0
[2017-06-15 12:28:29,110: INFO/job] Invalid result None fixed up on called to unconfigure_lnet with args {}
[2017-06-15 12:28:29,110: INFO/job_scheduler] Job 98 finished 1 steps successfully
[2017-06-15 12:28:29,135: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->unconfigured (intentional=True) 140623537940688
[2017-06-15 12:28:29,144: INFO/job] Job 98: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-15 12:28:29,144: INFO/job] Job 98: not_deleted=True
[2017-06-15 12:28:29,144: INFO/job_scheduler] Job 98: complete_job: Updating cache
[2017-06-15 12:28:29,144: INFO/job_scheduler] Job 98: locked_item 140623537940688 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-15 12:28:29,149: INFO/job_scheduler] Job 98 completing (errored=False, cancelled=False)
[2017-06-15 12:28:29,171: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 7 tasked)
[2017-06-15 12:28:30,413: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:30,413: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:30,413: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:30,413: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:30,414: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 709aa845-ab33-48f5-a21a-340f31e59e13
[2017-06-15 12:28:30,414: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 709aa845-ab33-48f5-a21a-340f31e59e13
[2017-06-15 12:28:30,423: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 709aa845-ab33-48f5-a21a-340f31e59e13
[2017-06-15 12:28:30,423: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 709aa845-ab33-48f5-a21a-340f31e59e13
[2017-06-15 12:28:30,424: INFO/job_scheduler] Job 99 finished 1 steps successfully
[2017-06-15 12:28:30,635: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->unconfigured (intentional=True) 140623537934800
[2017-06-15 12:28:30,640: INFO/job] Job 99: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 12:28:30,640: INFO/job] Job 99: not_deleted=True
[2017-06-15 12:28:30,640: INFO/job_scheduler] Job 99: complete_job: Updating cache
[2017-06-15 12:28:30,641: INFO/job_scheduler] Job 99: locked_item 140623537934800 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-15 12:28:30,645: INFO/job_scheduler] Job 99 completing (errored=False, cancelled=False)
[2017-06-15 12:28:30,660: INFO/job_scheduler] run_next: 1 runnable jobs of (10 pending, 6 tasked)
[2017-06-15 12:28:30,683: INFO/job_scheduler] Job 100: RunJobThread.run
[2017-06-15 12:28:30,685: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com deregister_server {}
[2017-06-15 12:28:30,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 27965c44-7b0c-4402-9bd5-b4e2f87d2c6d
[2017-06-15 12:28:30,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 27965c44-7b0c-4402-9bd5-b4e2f87d2c6d
[2017-06-15 12:28:30,788: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:30,788: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:30,788: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:30,788: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 12:28:30,789: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 27965c44-7b0c-4402-9bd5-b4e2f87d2c6d
[2017-06-15 12:28:30,789: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 27965c44-7b0c-4402-9bd5-b4e2f87d2c6d
[2017-06-15 12:28:30,816: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 27965c44-7b0c-4402-9bd5-b4e2f87d2c6d
[2017-06-15 12:28:30,816: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 27965c44-7b0c-4402-9bd5-b4e2f87d2c6d
[2017-06-15 12:28:30,836: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 12:28:30,836: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration unconfigured->removed (intentional=False) 140623739748176
[2017-06-15 12:28:30,844: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.ntp.NTPConfiguration'> 1
[2017-06-15 12:28:30,861: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 12:28:30,861: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->removed (intentional=False) 140623739794576
[2017-06-15 12:28:30,871: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.corosync2.Corosync2Configuration'> 1
[2017-06-15 12:28:30,886: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 12:28:30,886: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->removed (intentional=False) 140623739745296
[2017-06-15 12:28:30,899: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> 1
[2017-06-15 12:28:30,920: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 12:28:30,920: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration unconfigured->removed (intentional=False) 140623739747856
[2017-06-15 12:28:30,928: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> 1
[2017-06-15 12:28:30,943: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 12:28:30,944: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->removed (intentional=False) 140623739746960
[2017-06-15 12:28:30,953: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> 1
[2017-06-15 12:28:31,074: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:31,074: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:31,074: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:31,074: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:31,074: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c22ee3c0-dd07-4c8a-8e50-29df9f4e7d55
[2017-06-15 12:28:31,074: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c22ee3c0-dd07-4c8a-8e50-29df9f4e7d55
[2017-06-15 12:28:31,104: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c22ee3c0-dd07-4c8a-8e50-29df9f4e7d55
[2017-06-15 12:28:31,104: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c22ee3c0-dd07-4c8a-8e50-29df9f4e7d55
[2017-06-15 12:28:31,106: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-15 12:28:31,146: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1266b675-8a7a-41f5-9301-6f96442826ab
[2017-06-15 12:28:31,146: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1266b675-8a7a-41f5-9301-6f96442826ab
[2017-06-15 12:28:31,855: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->removed (intentional=False) 140623537934800
[2017-06-15 12:28:31,877: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-06-15 12:28:31,885: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.corosync2.Corosync2Configuration'> 1
[2017-06-15 12:28:31,919: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration unconfigured->removed (intentional=False) 140623537936976
[2017-06-15 12:28:31,926: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.ntp.NTPConfiguration'> 1
[2017-06-15 12:28:31,946: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->removed (intentional=False) 140623538077200
[2017-06-15 12:28:31,950: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> 1
[2017-06-15 12:28:31,982: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.host.ManagedHost'> 1
[2017-06-15 12:28:31,986: INFO/job_scheduler] Job 100 finished 2 steps successfully
[2017-06-15 12:28:32,036: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com managed->removed (intentional=True) 140624142212624
[2017-06-15 12:28:32,040: INFO/job] Job 100: StateChangeJob complete, setting state removed on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 12:28:32,040: INFO/job] Job 100: not_deleted=None
[2017-06-15 12:28:32,040: INFO/job_scheduler] Job 100: complete_job: Updating cache
[2017-06-15 12:28:32,041: INFO/job_scheduler] Job 100: locked_item 140624142212624 <class 'chroma_core.models.host.ManagedHost'> True None
[2017-06-15 12:28:32,041: INFO/job_scheduler] Job 100 completing (errored=False, cancelled=False)
[2017-06-15 12:28:32,117: INFO/job_scheduler] run_next: 0 runnable jobs of (9 pending, 6 tasked)
[2017-06-15 12:28:32,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:32,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:32,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:32,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:32,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1266b675-8a7a-41f5-9301-6f96442826ab
[2017-06-15 12:28:32,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1266b675-8a7a-41f5-9301-6f96442826ab
[2017-06-15 12:28:32,264: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1266b675-8a7a-41f5-9301-6f96442826ab
[2017-06-15 12:28:32,264: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1266b675-8a7a-41f5-9301-6f96442826ab
[2017-06-15 12:28:32,490: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:32,490: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:32,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:32,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:32,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 979c362a-1a5f-4afe-b6a1-6a465ac2d0fa
[2017-06-15 12:28:32,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 979c362a-1a5f-4afe-b6a1-6a465ac2d0fa
[2017-06-15 12:28:32,530: INFO/job_scheduler] Job 102 finished 2 steps successfully
[2017-06-15 12:28:32,542: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 979c362a-1a5f-4afe-b6a1-6a465ac2d0fa
[2017-06-15 12:28:32,542: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 979c362a-1a5f-4afe-b6a1-6a465ac2d0fa
[2017-06-15 12:28:32,543: INFO/job_scheduler] Job 106 finished 1 steps successfully
[2017-06-15 12:28:32,553: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration lnet_up->lnet_down (intentional=True) 140623538037264
[2017-06-15 12:28:32,558: INFO/job] Job 102: StateChangeJob complete, setting state lnet_down on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-15 12:28:32,558: INFO/job] Job 102: not_deleted=True
[2017-06-15 12:28:32,559: INFO/job_scheduler] Job 102: complete_job: Updating cache
[2017-06-15 12:28:32,559: INFO/job_scheduler] Job 102: locked_item 140623538037264 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-15 12:28:32,562: INFO/job_scheduler] Job 102 completing (errored=False, cancelled=False)
[2017-06-15 12:28:32,578: INFO/job_scheduler] run_next: 1 runnable jobs of (9 pending, 5 tasked)
[2017-06-15 12:28:32,593: INFO/job_scheduler] Job 105: RunJobThread.run
[2017-06-15 12:28:32,594: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com unload_lnet {}
[2017-06-15 12:28:32,626: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc eacc3c51-2da2-4160-9578-dd94766b3224
[2017-06-15 12:28:32,626: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc eacc3c51-2da2-4160-9578-dd94766b3224
[2017-06-15 12:28:32,653: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration started->stopped (intentional=True) 140623538040528
[2017-06-15 12:28:32,675: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-06-15 12:28:32,694: INFO/job] Job 106: StateChangeJob complete, setting state stopped on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 12:28:32,694: INFO/job] Job 106: not_deleted=True
[2017-06-15 12:28:32,695: INFO/job_scheduler] Job 106: complete_job: Updating cache
[2017-06-15 12:28:32,695: INFO/job_scheduler] Job 106: locked_item 140623538040528 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-15 12:28:32,706: INFO/job_scheduler] Job 106 completing (errored=False, cancelled=False)
[2017-06-15 12:28:32,727: INFO/job_scheduler] run_next: 1 runnable jobs of (8 pending, 5 tasked)
[2017-06-15 12:28:32,741: INFO/job_scheduler] Job 107: RunJobThread.run
[2017-06-15 12:28:32,742: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com unconfigure_corosync2 {'mcast_port': 65087, 'host_fqdn': u'lotus-32vm6.lotus.hpdd.lab.intel.com'}
[2017-06-15 12:28:32,755: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f3203ee0-29bc-44df-8e4b-705f35c1119d
[2017-06-15 12:28:32,755: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f3203ee0-29bc-44df-8e4b-705f35c1119d
[2017-06-15 12:28:33,447: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 6 tasked)
[2017-06-15 12:28:33,467: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 6 tasked)
[2017-06-15 12:28:34,220: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:34,220: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:34,221: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:34,221: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:34,221: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3261a83b-b22f-43d3-82ee-c6247075aec5
[2017-06-15 12:28:34,221: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3261a83b-b22f-43d3-82ee-c6247075aec5
[2017-06-15 12:28:34,257: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3261a83b-b22f-43d3-82ee-c6247075aec5
[2017-06-15 12:28:34,257: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3261a83b-b22f-43d3-82ee-c6247075aec5
[2017-06-15 12:28:34,258: INFO/job_scheduler] Job 110 finished 1 steps successfully
[2017-06-15 12:28:34,308: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 6 tasked)
[2017-06-15 12:28:34,335: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 6 tasked)
[2017-06-15 12:28:34,360: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com NTP configuration configured->unconfigured (intentional=True) 140623739748880
[2017-06-15 12:28:34,363: INFO/job] Job 110: StateChangeJob complete, setting state unconfigured on lotus-32vm7.lotus.hpdd.lab.intel.com NTP configuration
[2017-06-15 12:28:34,364: INFO/job] Job 110: not_deleted=True
[2017-06-15 12:28:34,364: INFO/job_scheduler] Job 110: complete_job: Updating cache
[2017-06-15 12:28:34,364: INFO/job_scheduler] Job 110: locked_item 140623739748880 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-06-15 12:28:34,366: INFO/job_scheduler] Job 110 completing (errored=False, cancelled=False)
[2017-06-15 12:28:34,377: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 5 tasked)
[2017-06-15 12:28:34,722: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:34,722: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:34,722: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:34,722: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:34,722: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc eacc3c51-2da2-4160-9578-dd94766b3224
[2017-06-15 12:28:34,722: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc eacc3c51-2da2-4160-9578-dd94766b3224
[2017-06-15 12:28:34,740: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc eacc3c51-2da2-4160-9578-dd94766b3224
[2017-06-15 12:28:34,740: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc eacc3c51-2da2-4160-9578-dd94766b3224
[2017-06-15 12:28:34,741: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-15 12:28:34,757: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 06ea12a7-15c5-4c4d-876c-986c9a642159
[2017-06-15 12:28:34,757: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 06ea12a7-15c5-4c4d-876c-986c9a642159
[2017-06-15 12:28:34,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:34,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:34,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:34,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:34,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 06ea12a7-15c5-4c4d-876c-986c9a642159
[2017-06-15 12:28:34,841: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 06ea12a7-15c5-4c4d-876c-986c9a642159
[2017-06-15 12:28:34,871: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 06ea12a7-15c5-4c4d-876c-986c9a642159
[2017-06-15 12:28:34,871: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 06ea12a7-15c5-4c4d-876c-986c9a642159
[2017-06-15 12:28:34,972: INFO/job_scheduler] Job 105 finished 2 steps successfully
[2017-06-15 12:28:34,985: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_unloaded (intentional=True) 140623538037264
[2017-06-15 12:28:34,990: INFO/job] Job 105: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-15 12:28:34,990: INFO/job] Job 105: not_deleted=True
[2017-06-15 12:28:34,990: INFO/job_scheduler] Job 105: complete_job: Updating cache
[2017-06-15 12:28:34,991: INFO/job_scheduler] Job 105: locked_item 140623538037264 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-15 12:28:34,993: INFO/job_scheduler] Job 105 completing (errored=False, cancelled=False)
[2017-06-15 12:28:35,011: INFO/job_scheduler] run_next: 1 runnable jobs of (7 pending, 4 tasked)
[2017-06-15 12:28:35,031: INFO/job_scheduler] Job 108: RunJobThread.run
[2017-06-15 12:28:35,031: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com unconfigure_lnet {}
[2017-06-15 12:28:35,048: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 116520c6-acf0-4ab4-bcad-4826c3253ef2
[2017-06-15 12:28:35,048: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 116520c6-acf0-4ab4-bcad-4826c3253ef2
[2017-06-15 12:28:35,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:35,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:35,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:35,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:35,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 116520c6-acf0-4ab4-bcad-4826c3253ef2
[2017-06-15 12:28:35,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 116520c6-acf0-4ab4-bcad-4826c3253ef2
[2017-06-15 12:28:35,166: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 116520c6-acf0-4ab4-bcad-4826c3253ef2
[2017-06-15 12:28:35,166: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 116520c6-acf0-4ab4-bcad-4826c3253ef2
[2017-06-15 12:28:35,166: INFO/job] Invalid result None fixed up on called to unconfigure_lnet with args {}
[2017-06-15 12:28:35,166: INFO/job_scheduler] Job 108 finished 1 steps successfully
[2017-06-15 12:28:35,195: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->unconfigured (intentional=True) 140623538037264
[2017-06-15 12:28:35,206: INFO/job] Job 108: StateChangeJob complete, setting state unconfigured on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-15 12:28:35,206: INFO/job] Job 108: not_deleted=True
[2017-06-15 12:28:35,206: INFO/job_scheduler] Job 108: complete_job: Updating cache
[2017-06-15 12:28:35,206: INFO/job_scheduler] Job 108: locked_item 140623538037264 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-15 12:28:35,215: INFO/job_scheduler] Job 108 completing (errored=False, cancelled=False)
[2017-06-15 12:28:35,236: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-06-15 12:28:35,361: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:35,361: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:35,362: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:35,362: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:35,362: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 00c50aa5-42cc-4602-a19e-0894bdc04766
[2017-06-15 12:28:35,362: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 00c50aa5-42cc-4602-a19e-0894bdc04766
[2017-06-15 12:28:35,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 00c50aa5-42cc-4602-a19e-0894bdc04766
[2017-06-15 12:28:35,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 00c50aa5-42cc-4602-a19e-0894bdc04766
[2017-06-15 12:28:35,382: INFO/job_scheduler] Job 113 finished 1 steps successfully
[2017-06-15 12:28:35,494: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com RSyslog configuration configured->unconfigured (intentional=True) 140623739751952
[2017-06-15 12:28:35,497: INFO/job] Job 113: StateChangeJob complete, setting state unconfigured on lotus-32vm7.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-06-15 12:28:35,498: INFO/job] Job 113: not_deleted=True
[2017-06-15 12:28:35,498: INFO/job_scheduler] Job 113: complete_job: Updating cache
[2017-06-15 12:28:35,498: INFO/job_scheduler] Job 113: locked_item 140623739751952 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-06-15 12:28:35,505: INFO/job_scheduler] Job 113 completing (errored=False, cancelled=False)
[2017-06-15 12:28:35,527: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 3 tasked)
[2017-06-15 12:28:36,259: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:36,259: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:36,259: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:36,259: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:36,259: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 31f02c0f-ea1d-4eec-bc34-10d7f1a66ca6
[2017-06-15 12:28:36,259: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 31f02c0f-ea1d-4eec-bc34-10d7f1a66ca6
[2017-06-15 12:28:36,308: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 31f02c0f-ea1d-4eec-bc34-10d7f1a66ca6
[2017-06-15 12:28:36,308: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 31f02c0f-ea1d-4eec-bc34-10d7f1a66ca6
[2017-06-15 12:28:36,309: INFO/job_scheduler] Job 112 finished 1 steps successfully
[2017-06-15 12:28:36,377: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration started->stopped (intentional=True) 140623538052368
[2017-06-15 12:28:36,385: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-06-15 12:28:36,388: INFO/job] Job 112: StateChangeJob complete, setting state stopped on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-15 12:28:36,389: INFO/job] Job 112: not_deleted=True
[2017-06-15 12:28:36,389: INFO/job_scheduler] Job 112: complete_job: Updating cache
[2017-06-15 12:28:36,391: INFO/job_scheduler] Job 112: locked_item 140623538052368 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-15 12:28:36,398: INFO/job_scheduler] Job 112 completing (errored=False, cancelled=False)
[2017-06-15 12:28:36,427: INFO/job_scheduler] run_next: 1 runnable jobs of (6 pending, 2 tasked)
[2017-06-15 12:28:36,437: INFO/job_scheduler] Job 114: RunJobThread.run
[2017-06-15 12:28:36,438: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com stop_corosync2 {}
[2017-06-15 12:28:36,462: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 208243db-b1e9-4793-a48f-e4fbb416cfa8
[2017-06-15 12:28:36,462: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 208243db-b1e9-4793-a48f-e4fbb416cfa8
[2017-06-15 12:28:38,041: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 12:28:38,055: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-4 transitioning from managed to removed
[2017-06-15 12:28:38,139: INFO/command_plan] Created command 36 (Remove host lotus-32vm8.lotus.hpdd.lab.intel.com from configuration) with 9 jobs
[2017-06-15 12:28:38,192: INFO/job_scheduler] run_next: 4 runnable jobs of (14 pending, 3 tasked)
[2017-06-15 12:28:38,248: INFO/job_scheduler] Job 119: RunJobThread.run
[2017-06-15 12:28:38,249: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-06-15 12:28:38,264: INFO/job_scheduler] Job 120: RunJobThread.run
[2017-06-15 12:28:38,265: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com stop_lnet {}
[2017-06-15 12:28:38,267: INFO/job_scheduler] Job 121: RunJobThread.run
[2017-06-15 12:28:38,268: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com unconfigure_ntp {}
[2017-06-15 12:28:38,272: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 77e50615-c9fe-4b25-aa85-ed429b34cc1f
[2017-06-15 12:28:38,272: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 77e50615-c9fe-4b25-aa85-ed429b34cc1f
[2017-06-15 12:28:38,290: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f016c1ac-72bd-43ea-80bf-8bea1fc3699c
[2017-06-15 12:28:38,290: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f016c1ac-72bd-43ea-80bf-8bea1fc3699c
[2017-06-15 12:28:38,299: INFO/job_scheduler] Job 122: RunJobThread.run
[2017-06-15 12:28:38,300: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com unconfigure_rsyslog {}
[2017-06-15 12:28:38,306: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 063e4880-54fd-4977-8316-86c9be830a4b
[2017-06-15 12:28:38,306: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 063e4880-54fd-4977-8316-86c9be830a4b
[2017-06-15 12:28:38,321: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ca5db6b7-bb5c-4af8-96dd-3036756365ae
[2017-06-15 12:28:38,321: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ca5db6b7-bb5c-4af8-96dd-3036756365ae
[2017-06-15 12:28:38,911: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:38,911: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:38,911: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:38,911: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:38,911: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f3203ee0-29bc-44df-8e4b-705f35c1119d
[2017-06-15 12:28:38,911: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f3203ee0-29bc-44df-8e4b-705f35c1119d
[2017-06-15 12:28:38,935: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f3203ee0-29bc-44df-8e4b-705f35c1119d
[2017-06-15 12:28:38,935: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f3203ee0-29bc-44df-8e4b-705f35c1119d
[2017-06-15 12:28:38,935: INFO/job_scheduler] Job 107 finished 1 steps successfully
[2017-06-15 12:28:39,016: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->unconfigured (intentional=True) 140623538040528
[2017-06-15 12:28:39,022: INFO/job] Job 107: StateChangeJob complete, setting state unconfigured on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 12:28:39,022: INFO/job] Job 107: not_deleted=True
[2017-06-15 12:28:39,022: INFO/job_scheduler] Job 107: complete_job: Updating cache
[2017-06-15 12:28:39,023: INFO/job_scheduler] Job 107: locked_item 140623538040528 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-15 12:28:39,026: INFO/job_scheduler] Job 107 completing (errored=False, cancelled=False)
[2017-06-15 12:28:39,141: INFO/job_scheduler] run_next: 1 runnable jobs of (10 pending, 6 tasked)
[2017-06-15 12:28:39,163: INFO/job_scheduler] Job 109: RunJobThread.run
[2017-06-15 12:28:39,165: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com deregister_server {}
[2017-06-15 12:28:39,179: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7088ec85-4685-46e1-8ebe-9261e46da203
[2017-06-15 12:28:39,179: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7088ec85-4685-46e1-8ebe-9261e46da203
[2017-06-15 12:28:39,233: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:39,233: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:39,234: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:39,234: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 12:28:39,234: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7088ec85-4685-46e1-8ebe-9261e46da203
[2017-06-15 12:28:39,234: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7088ec85-4685-46e1-8ebe-9261e46da203
[2017-06-15 12:28:39,246: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7088ec85-4685-46e1-8ebe-9261e46da203
[2017-06-15 12:28:39,246: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7088ec85-4685-46e1-8ebe-9261e46da203
[2017-06-15 12:28:39,263: INFO/job] Deleting dependent lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration for host lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 12:28:39,263: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration unconfigured->removed (intentional=False) 140623739746384
[2017-06-15 12:28:39,310: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> 2
[2017-06-15 12:28:39,326: INFO/job] Deleting dependent lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration for host lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 12:28:39,326: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->removed (intentional=False) 140623537911568
[2017-06-15 12:28:39,337: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.corosync2.Corosync2Configuration'> 2
[2017-06-15 12:28:39,351: INFO/job] Deleting dependent lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration for host lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 12:28:39,351: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration unconfigured->removed (intentional=False) 140623538053584
[2017-06-15 12:28:39,359: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.ntp.NTPConfiguration'> 2
[2017-06-15 12:28:39,377: INFO/job] Deleting dependent lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration for host lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 12:28:39,377: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->removed (intentional=False) 140623739744656
[2017-06-15 12:28:39,382: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> 2
[2017-06-15 12:28:39,400: INFO/job] Deleting dependent lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration for host lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 12:28:39,400: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->removed (intentional=False) 140623538054416
[2017-06-15 12:28:39,418: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> 2
[2017-06-15 12:28:40,074: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->removed (intentional=False) 140623538040528
[2017-06-15 12:28:40,176: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-06-15 12:28:40,193: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.corosync2.Corosync2Configuration'> 2
[2017-06-15 12:28:40,232: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration unconfigured->removed (intentional=False) 140623538038736
[2017-06-15 12:28:40,244: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.ntp.NTPConfiguration'> 2
[2017-06-15 12:28:40,258: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->removed (intentional=False) 140623538039120
[2017-06-15 12:28:40,266: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> 2
[2017-06-15 12:28:40,313: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.host.ManagedHost'> 2
[2017-06-15 12:28:40,326: INFO/job_scheduler] Job 109 finished 2 steps successfully
[2017-06-15 12:28:40,374: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com managed->removed (intentional=True) 140623538123280
[2017-06-15 12:28:40,379: INFO/job] Job 109: StateChangeJob complete, setting state removed on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 12:28:40,379: INFO/job] Job 109: not_deleted=None
[2017-06-15 12:28:40,379: INFO/job_scheduler] Job 109: complete_job: Updating cache
[2017-06-15 12:28:40,379: INFO/job_scheduler] Job 109: locked_item 140623538123280 <class 'chroma_core.models.host.ManagedHost'> True None
[2017-06-15 12:28:40,379: INFO/job_scheduler] Job 109 completing (errored=False, cancelled=False)
[2017-06-15 12:28:40,458: INFO/job_scheduler] run_next: 0 runnable jobs of (9 pending, 6 tasked)
[2017-06-15 12:28:42,477: INFO/job_scheduler] run_next: 0 runnable jobs of (9 pending, 6 tasked)
[2017-06-15 12:28:42,494: INFO/job_scheduler] run_next: 0 runnable jobs of (9 pending, 6 tasked)
[2017-06-15 12:28:44,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:44,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:44,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:44,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:44,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3c101a65-2dcd-499b-a58f-013aafeb57a2
[2017-06-15 12:28:44,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3c101a65-2dcd-499b-a58f-013aafeb57a2
[2017-06-15 12:28:44,313: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3c101a65-2dcd-499b-a58f-013aafeb57a2
[2017-06-15 12:28:44,313: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3c101a65-2dcd-499b-a58f-013aafeb57a2
[2017-06-15 12:28:44,314: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-15 12:28:44,328: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 46af7f25-0772-49bf-adf3-f81863f85653
[2017-06-15 12:28:44,328: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 46af7f25-0772-49bf-adf3-f81863f85653
[2017-06-15 12:28:44,390: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:44,390: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:44,390: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:44,390: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:44,390: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 46af7f25-0772-49bf-adf3-f81863f85653
[2017-06-15 12:28:44,390: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 46af7f25-0772-49bf-adf3-f81863f85653
[2017-06-15 12:28:44,395: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 46af7f25-0772-49bf-adf3-f81863f85653
[2017-06-15 12:28:44,395: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 46af7f25-0772-49bf-adf3-f81863f85653
[2017-06-15 12:28:44,475: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:28:44,475: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:28:44,475: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:28:44,475: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:28:44,476: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 063e4880-54fd-4977-8316-86c9be830a4b
[2017-06-15 12:28:44,476: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 063e4880-54fd-4977-8316-86c9be830a4b
[2017-06-15 12:28:44,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 063e4880-54fd-4977-8316-86c9be830a4b
[2017-06-15 12:28:44,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 063e4880-54fd-4977-8316-86c9be830a4b
[2017-06-15 12:28:44,479: INFO/job_scheduler] Job 121 finished 1 steps successfully
[2017-06-15 12:28:44,546: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:44,546: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:44,546: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:44,546: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:44,546: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 208243db-b1e9-4793-a48f-e4fbb416cfa8
[2017-06-15 12:28:44,546: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 208243db-b1e9-4793-a48f-e4fbb416cfa8
[2017-06-15 12:28:44,583: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 208243db-b1e9-4793-a48f-e4fbb416cfa8
[2017-06-15 12:28:44,583: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 208243db-b1e9-4793-a48f-e4fbb416cfa8
[2017-06-15 12:28:44,584: INFO/job_scheduler] Job 114 finished 1 steps successfully
[2017-06-15 12:28:44,610: INFO/job_scheduler] Job 111 finished 2 steps successfully
[2017-06-15 12:28:44,791: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com NTP configuration configured->unconfigured (intentional=True) 140623538038480
[2017-06-15 12:28:44,795: INFO/job] Job 121: StateChangeJob complete, setting state unconfigured on lotus-32vm8.lotus.hpdd.lab.intel.com NTP configuration
[2017-06-15 12:28:44,795: INFO/job] Job 121: not_deleted=True
[2017-06-15 12:28:44,795: INFO/job_scheduler] Job 121: complete_job: Updating cache
[2017-06-15 12:28:44,795: INFO/job_scheduler] Job 121: locked_item 140623538038480 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-06-15 12:28:44,798: INFO/job_scheduler] Job 121 completing (errored=False, cancelled=False)
[2017-06-15 12:28:44,817: INFO/job_scheduler] run_next: 0 runnable jobs of (9 pending, 5 tasked)
[2017-06-15 12:28:44,856: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration started->stopped (intentional=True) 140623739749392
[2017-06-15 12:28:44,863: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-06-15 12:28:44,878: INFO/job] Job 114: StateChangeJob complete, setting state stopped on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 12:28:44,878: INFO/job] Job 114: not_deleted=True
[2017-06-15 12:28:44,878: INFO/job_scheduler] Job 114: complete_job: Updating cache
[2017-06-15 12:28:44,878: INFO/job_scheduler] Job 114: locked_item 140623739749392 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-15 12:28:44,882: INFO/job_scheduler] Job 114 completing (errored=False, cancelled=False)
[2017-06-15 12:28:44,899: INFO/job_scheduler] run_next: 1 runnable jobs of (9 pending, 4 tasked)
[2017-06-15 12:28:44,903: INFO/job_scheduler] Job 117: RunJobThread.run
[2017-06-15 12:28:44,904: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com unconfigure_corosync2 {'mcast_port': 49429, 'host_fqdn': u'lotus-32vm7.lotus.hpdd.lab.intel.com'}
[2017-06-15 12:28:44,929: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration lnet_up->lnet_down (intentional=True) 140623537940880
[2017-06-15 12:28:44,938: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d5e3699b-a1d1-4f24-96d1-b438f0155f2d
[2017-06-15 12:28:44,938: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d5e3699b-a1d1-4f24-96d1-b438f0155f2d
[2017-06-15 12:28:44,949: INFO/job] Job 111: StateChangeJob complete, setting state lnet_down on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-15 12:28:44,949: INFO/job] Job 111: not_deleted=True
[2017-06-15 12:28:44,949: INFO/job_scheduler] Job 111: complete_job: Updating cache
[2017-06-15 12:28:44,949: INFO/job_scheduler] Job 111: locked_item 140623537940880 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-15 12:28:44,964: INFO/job_scheduler] Job 111 completing (errored=False, cancelled=False)
[2017-06-15 12:28:45,009: INFO/job_scheduler] run_next: 1 runnable jobs of (8 pending, 4 tasked)
[2017-06-15 12:28:45,026: INFO/job_scheduler] Job 115: RunJobThread.run
[2017-06-15 12:28:45,027: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com unload_lnet {}
[2017-06-15 12:28:45,063: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 098c7147-ae5f-4d9e-a0dc-766eacccb1b3
[2017-06-15 12:28:45,063: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 098c7147-ae5f-4d9e-a0dc-766eacccb1b3
[2017-06-15 12:28:45,382: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:28:45,382: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:28:45,382: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:28:45,382: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:28:45,383: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 77e50615-c9fe-4b25-aa85-ed429b34cc1f
[2017-06-15 12:28:45,383: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 77e50615-c9fe-4b25-aa85-ed429b34cc1f
[2017-06-15 12:28:45,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 77e50615-c9fe-4b25-aa85-ed429b34cc1f
[2017-06-15 12:28:45,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 77e50615-c9fe-4b25-aa85-ed429b34cc1f
[2017-06-15 12:28:45,400: INFO/job_scheduler] Job 119 finished 1 steps successfully
[2017-06-15 12:28:45,467: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration started->stopped (intentional=True) 140623538034000
[2017-06-15 12:28:45,475: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-06-15 12:28:45,481: INFO/job] Job 119: StateChangeJob complete, setting state stopped on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-15 12:28:45,482: INFO/job] Job 119: not_deleted=True
[2017-06-15 12:28:45,482: INFO/job_scheduler] Job 119: complete_job: Updating cache
[2017-06-15 12:28:45,483: INFO/job_scheduler] Job 119: locked_item 140623538034000 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-15 12:28:45,492: INFO/job_scheduler] Job 119 completing (errored=False, cancelled=False)
[2017-06-15 12:28:45,502: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:28:45,502: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:28:45,509: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:28:45,508: INFO/job_scheduler] run_next: 1 runnable jobs of (7 pending, 4 tasked)
[2017-06-15 12:28:45,509: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:28:45,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ca5db6b7-bb5c-4af8-96dd-3036756365ae
[2017-06-15 12:28:45,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ca5db6b7-bb5c-4af8-96dd-3036756365ae
[2017-06-15 12:28:45,528: INFO/job_scheduler] Job 123: RunJobThread.run
[2017-06-15 12:28:45,529: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com stop_corosync2 {}
[2017-06-15 12:28:45,543: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ca5db6b7-bb5c-4af8-96dd-3036756365ae
[2017-06-15 12:28:45,543: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ca5db6b7-bb5c-4af8-96dd-3036756365ae
[2017-06-15 12:28:45,543: INFO/job_scheduler] Job 122 finished 1 steps successfully
[2017-06-15 12:28:45,548: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 56d24fc5-b379-45a1-b4fd-658ae4e030de
[2017-06-15 12:28:45,548: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 56d24fc5-b379-45a1-b4fd-658ae4e030de
[2017-06-15 12:28:45,758: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com RSyslog configuration configured->unconfigured (intentional=True) 140623538037008
[2017-06-15 12:28:45,761: INFO/job] Job 122: StateChangeJob complete, setting state unconfigured on lotus-32vm8.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-06-15 12:28:45,761: INFO/job] Job 122: not_deleted=True
[2017-06-15 12:28:45,761: INFO/job_scheduler] Job 122: complete_job: Updating cache
[2017-06-15 12:28:45,761: INFO/job_scheduler] Job 122: locked_item 140623538037008 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-06-15 12:28:45,764: INFO/job_scheduler] Job 122 completing (errored=False, cancelled=False)
[2017-06-15 12:28:45,774: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 4 tasked)
[2017-06-15 12:28:47,150: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:47,150: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:47,150: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:47,150: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:47,151: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 098c7147-ae5f-4d9e-a0dc-766eacccb1b3
[2017-06-15 12:28:47,151: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 098c7147-ae5f-4d9e-a0dc-766eacccb1b3
[2017-06-15 12:28:47,178: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 098c7147-ae5f-4d9e-a0dc-766eacccb1b3
[2017-06-15 12:28:47,178: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 098c7147-ae5f-4d9e-a0dc-766eacccb1b3
[2017-06-15 12:28:47,179: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-15 12:28:47,191: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d96e0244-d386-438e-930d-84ffb144c906
[2017-06-15 12:28:47,191: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d96e0244-d386-438e-930d-84ffb144c906
[2017-06-15 12:28:47,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:47,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:47,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:47,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:47,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d96e0244-d386-438e-930d-84ffb144c906
[2017-06-15 12:28:47,271: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d96e0244-d386-438e-930d-84ffb144c906
[2017-06-15 12:28:47,310: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d96e0244-d386-438e-930d-84ffb144c906
[2017-06-15 12:28:47,310: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d96e0244-d386-438e-930d-84ffb144c906
[2017-06-15 12:28:47,443: INFO/job_scheduler] Job 115 finished 2 steps successfully
[2017-06-15 12:28:47,510: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_unloaded (intentional=True) 140623537940880
[2017-06-15 12:28:47,516: INFO/job] Job 115: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-15 12:28:47,516: INFO/job] Job 115: not_deleted=True
[2017-06-15 12:28:47,516: INFO/job_scheduler] Job 115: complete_job: Updating cache
[2017-06-15 12:28:47,517: INFO/job_scheduler] Job 115: locked_item 140623537940880 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-15 12:28:47,523: INFO/job_scheduler] Job 115 completing (errored=False, cancelled=False)
[2017-06-15 12:28:47,550: INFO/job_scheduler] run_next: 1 runnable jobs of (6 pending, 3 tasked)
[2017-06-15 12:28:47,565: INFO/job_scheduler] Job 116: RunJobThread.run
[2017-06-15 12:28:47,569: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com unconfigure_lnet {}
[2017-06-15 12:28:47,592: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d7805c04-1934-49bf-949b-6de3c0b50f13
[2017-06-15 12:28:47,592: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d7805c04-1934-49bf-949b-6de3c0b50f13
[2017-06-15 12:28:47,671: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:47,671: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:47,671: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:47,671: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:47,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d7805c04-1934-49bf-949b-6de3c0b50f13
[2017-06-15 12:28:47,672: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d7805c04-1934-49bf-949b-6de3c0b50f13
[2017-06-15 12:28:47,681: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-06-15 12:28:47,713: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 4 tasked)
[2017-06-15 12:28:47,720: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d7805c04-1934-49bf-949b-6de3c0b50f13
[2017-06-15 12:28:47,720: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d7805c04-1934-49bf-949b-6de3c0b50f13
[2017-06-15 12:28:47,721: INFO/job] Invalid result None fixed up on called to unconfigure_lnet with args {}
[2017-06-15 12:28:47,721: INFO/job_scheduler] Job 116 finished 1 steps successfully
[2017-06-15 12:28:47,770: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->unconfigured (intentional=True) 140623537940880
[2017-06-15 12:28:47,776: INFO/job] Job 116: StateChangeJob complete, setting state unconfigured on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-15 12:28:47,776: INFO/job] Job 116: not_deleted=True
[2017-06-15 12:28:47,776: INFO/job_scheduler] Job 116: complete_job: Updating cache
[2017-06-15 12:28:47,776: INFO/job_scheduler] Job 116: locked_item 140623537940880 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-15 12:28:47,780: INFO/job_scheduler] Job 116 completing (errored=False, cancelled=False)
[2017-06-15 12:28:47,799: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 3 tasked)
[2017-06-15 12:28:52,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:52,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:52,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:52,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:52,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d5e3699b-a1d1-4f24-96d1-b438f0155f2d
[2017-06-15 12:28:52,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d5e3699b-a1d1-4f24-96d1-b438f0155f2d
[2017-06-15 12:28:52,054: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d5e3699b-a1d1-4f24-96d1-b438f0155f2d
[2017-06-15 12:28:52,054: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d5e3699b-a1d1-4f24-96d1-b438f0155f2d
[2017-06-15 12:28:52,055: INFO/job_scheduler] Job 117 finished 1 steps successfully
[2017-06-15 12:28:52,131: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->unconfigured (intentional=True) 140623739749392
[2017-06-15 12:28:52,138: INFO/job] Job 117: StateChangeJob complete, setting state unconfigured on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 12:28:52,138: INFO/job] Job 117: not_deleted=True
[2017-06-15 12:28:52,138: INFO/job_scheduler] Job 117: complete_job: Updating cache
[2017-06-15 12:28:52,138: INFO/job_scheduler] Job 117: locked_item 140623739749392 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-15 12:28:52,143: INFO/job_scheduler] Job 117 completing (errored=False, cancelled=False)
[2017-06-15 12:28:52,157: INFO/job_scheduler] run_next: 1 runnable jobs of (5 pending, 2 tasked)
[2017-06-15 12:28:52,178: INFO/job_scheduler] Job 118: RunJobThread.run
[2017-06-15 12:28:52,180: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com deregister_server {}
[2017-06-15 12:28:52,195: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cc2ab84e-4064-4ac0-ba91-3fae60000dea
[2017-06-15 12:28:52,195: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cc2ab84e-4064-4ac0-ba91-3fae60000dea
[2017-06-15 12:28:52,314: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:52,314: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:52,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:52,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 12:28:52,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cc2ab84e-4064-4ac0-ba91-3fae60000dea
[2017-06-15 12:28:52,315: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cc2ab84e-4064-4ac0-ba91-3fae60000dea
[2017-06-15 12:28:52,359: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cc2ab84e-4064-4ac0-ba91-3fae60000dea
[2017-06-15 12:28:52,359: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cc2ab84e-4064-4ac0-ba91-3fae60000dea
[2017-06-15 12:28:52,393: INFO/job] Deleting dependent lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration for host lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 12:28:52,393: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->removed (intentional=False) 140623739840912
[2017-06-15 12:28:52,415: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> 3
[2017-06-15 12:28:52,437: INFO/job] Deleting dependent lotus-32vm7.lotus.hpdd.lab.intel.com RSyslog configuration for host lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 12:28:52,437: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->removed (intentional=False) 140623739842448
[2017-06-15 12:28:52,443: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> 3
[2017-06-15 12:28:52,457: INFO/job] Deleting dependent lotus-32vm7.lotus.hpdd.lab.intel.com NTP configuration for host lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 12:28:52,458: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com NTP configuration unconfigured->removed (intentional=False) 140623739841936
[2017-06-15 12:28:52,463: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.ntp.NTPConfiguration'> 3
[2017-06-15 12:28:52,482: INFO/job] Deleting dependent lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration for host lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 12:28:52,483: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration unconfigured->removed (intentional=False) 140623739838864
[2017-06-15 12:28:52,491: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> 3
[2017-06-15 12:28:52,509: INFO/job] Deleting dependent lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration for host lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 12:28:52,509: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->removed (intentional=False) 140623739782288
[2017-06-15 12:28:52,530: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.corosync2.Corosync2Configuration'> 3
[2017-06-15 12:28:53,058: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->removed (intentional=False) 140623739749392
[2017-06-15 12:28:53,073: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-06-15 12:28:53,097: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.corosync2.Corosync2Configuration'> 3
[2017-06-15 12:28:53,123: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com NTP configuration unconfigured->removed (intentional=False) 140623739748880
[2017-06-15 12:28:53,128: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.ntp.NTPConfiguration'> 3
[2017-06-15 12:28:53,256: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->removed (intentional=False) 140623739751952
[2017-06-15 12:28:53,262: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> 3
[2017-06-15 12:28:53,285: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.host.ManagedHost'> 3
[2017-06-15 12:28:53,290: INFO/job_scheduler] Job 118 finished 2 steps successfully
[2017-06-15 12:28:53,348: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com managed->removed (intentional=True) 140623739811216
[2017-06-15 12:28:53,352: INFO/job] Job 118: StateChangeJob complete, setting state removed on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 12:28:53,353: INFO/job] Job 118: not_deleted=None
[2017-06-15 12:28:53,353: INFO/job_scheduler] Job 118: complete_job: Updating cache
[2017-06-15 12:28:53,353: INFO/job_scheduler] Job 118: locked_item 140623739811216 <class 'chroma_core.models.host.ManagedHost'> True None
[2017-06-15 12:28:53,353: INFO/job_scheduler] Job 118 completing (errored=False, cancelled=False)
[2017-06-15 12:28:53,409: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-15 12:28:53,641: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:28:53,641: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:28:53,641: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:28:53,641: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:28:53,641: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 56d24fc5-b379-45a1-b4fd-658ae4e030de
[2017-06-15 12:28:53,641: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 56d24fc5-b379-45a1-b4fd-658ae4e030de
[2017-06-15 12:28:53,669: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 56d24fc5-b379-45a1-b4fd-658ae4e030de
[2017-06-15 12:28:53,669: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 56d24fc5-b379-45a1-b4fd-658ae4e030de
[2017-06-15 12:28:53,670: INFO/job_scheduler] Job 123 finished 1 steps successfully
[2017-06-15 12:28:53,723: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration started->stopped (intentional=True) 140623537940496
[2017-06-15 12:28:53,730: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-06-15 12:28:53,742: INFO/job] Job 123: StateChangeJob complete, setting state stopped on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 12:28:53,742: INFO/job] Job 123: not_deleted=True
[2017-06-15 12:28:53,742: INFO/job_scheduler] Job 123: complete_job: Updating cache
[2017-06-15 12:28:53,742: INFO/job_scheduler] Job 123: locked_item 140623537940496 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-15 12:28:53,746: INFO/job_scheduler] Job 123 completing (errored=False, cancelled=False)
[2017-06-15 12:28:53,757: INFO/job_scheduler] run_next: 1 runnable jobs of (4 pending, 1 tasked)
[2017-06-15 12:28:53,760: INFO/job_scheduler] Job 125: RunJobThread.run
[2017-06-15 12:28:53,761: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com unconfigure_corosync2 {'mcast_port': 49429, 'host_fqdn': u'lotus-32vm8.lotus.hpdd.lab.intel.com'}
[2017-06-15 12:28:53,774: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a0ece145-bba7-4332-bcd0-b41c51a6783a
[2017-06-15 12:28:53,774: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a0ece145-bba7-4332-bcd0-b41c51a6783a
[2017-06-15 12:28:58,536: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:28:58,536: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:28:58,536: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:28:58,536: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:28:58,536: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f016c1ac-72bd-43ea-80bf-8bea1fc3699c
[2017-06-15 12:28:58,536: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f016c1ac-72bd-43ea-80bf-8bea1fc3699c
[2017-06-15 12:28:58,568: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f016c1ac-72bd-43ea-80bf-8bea1fc3699c
[2017-06-15 12:28:58,568: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f016c1ac-72bd-43ea-80bf-8bea1fc3699c
[2017-06-15 12:28:58,569: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-15 12:28:58,583: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 26c4dd05-9125-4b5d-be2e-6792af5fda51
[2017-06-15 12:28:58,583: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 26c4dd05-9125-4b5d-be2e-6792af5fda51
[2017-06-15 12:28:58,654: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:28:58,654: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:28:58,654: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:28:58,654: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:28:58,654: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 26c4dd05-9125-4b5d-be2e-6792af5fda51
[2017-06-15 12:28:58,654: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 26c4dd05-9125-4b5d-be2e-6792af5fda51
[2017-06-15 12:28:58,697: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 26c4dd05-9125-4b5d-be2e-6792af5fda51
[2017-06-15 12:28:58,697: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 26c4dd05-9125-4b5d-be2e-6792af5fda51
[2017-06-15 12:28:58,827: INFO/job_scheduler] Job 120 finished 2 steps successfully
[2017-06-15 12:28:58,839: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration lnet_up->lnet_down (intentional=True) 140623538039248
[2017-06-15 12:28:58,845: INFO/job] Job 120: StateChangeJob complete, setting state lnet_down on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-15 12:28:58,845: INFO/job] Job 120: not_deleted=True
[2017-06-15 12:28:58,845: INFO/job_scheduler] Job 120: complete_job: Updating cache
[2017-06-15 12:28:58,845: INFO/job_scheduler] Job 120: locked_item 140623538039248 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-15 12:28:58,847: INFO/job_scheduler] Job 120 completing (errored=False, cancelled=False)
[2017-06-15 12:28:58,864: INFO/job_scheduler] run_next: 1 runnable jobs of (3 pending, 1 tasked)
[2017-06-15 12:28:58,875: INFO/job_scheduler] Job 124: RunJobThread.run
[2017-06-15 12:28:58,875: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com unload_lnet {}
[2017-06-15 12:28:58,913: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 11d385ce-1a70-46b9-a4b7-7707b2715c6a
[2017-06-15 12:28:58,913: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 11d385ce-1a70-46b9-a4b7-7707b2715c6a
[2017-06-15 12:28:59,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:28:59,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:28:59,977: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:28:59,977: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:28:59,977: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a0ece145-bba7-4332-bcd0-b41c51a6783a
[2017-06-15 12:28:59,977: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a0ece145-bba7-4332-bcd0-b41c51a6783a
[2017-06-15 12:28:59,989: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a0ece145-bba7-4332-bcd0-b41c51a6783a
[2017-06-15 12:28:59,989: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a0ece145-bba7-4332-bcd0-b41c51a6783a
[2017-06-15 12:28:59,990: INFO/job_scheduler] Job 125 finished 1 steps successfully
[2017-06-15 12:29:00,089: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->unconfigured (intentional=True) 140623537940496
[2017-06-15 12:29:00,095: INFO/job] Job 125: StateChangeJob complete, setting state unconfigured on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 12:29:00,095: INFO/job] Job 125: not_deleted=True
[2017-06-15 12:29:00,095: INFO/job_scheduler] Job 125: complete_job: Updating cache
[2017-06-15 12:29:00,095: INFO/job_scheduler] Job 125: locked_item 140623537940496 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-15 12:29:00,099: INFO/job_scheduler] Job 125 completing (errored=False, cancelled=False)
[2017-06-15 12:29:00,114: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-15 12:29:01,046: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:29:01,046: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:29:01,046: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:29:01,046: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:29:01,046: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 11d385ce-1a70-46b9-a4b7-7707b2715c6a
[2017-06-15 12:29:01,046: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 11d385ce-1a70-46b9-a4b7-7707b2715c6a
[2017-06-15 12:29:01,077: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 11d385ce-1a70-46b9-a4b7-7707b2715c6a
[2017-06-15 12:29:01,077: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 11d385ce-1a70-46b9-a4b7-7707b2715c6a
[2017-06-15 12:29:01,078: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-15 12:29:01,096: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0deb82e6-f70b-48aa-aa1b-a1443cc5d133
[2017-06-15 12:29:01,096: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0deb82e6-f70b-48aa-aa1b-a1443cc5d133
[2017-06-15 12:29:01,216: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:29:01,216: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:29:01,216: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:29:01,216: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:29:01,216: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0deb82e6-f70b-48aa-aa1b-a1443cc5d133
[2017-06-15 12:29:01,216: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0deb82e6-f70b-48aa-aa1b-a1443cc5d133
[2017-06-15 12:29:01,261: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0deb82e6-f70b-48aa-aa1b-a1443cc5d133
[2017-06-15 12:29:01,261: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0deb82e6-f70b-48aa-aa1b-a1443cc5d133
[2017-06-15 12:29:01,362: INFO/job_scheduler] Job 124 finished 2 steps successfully
[2017-06-15 12:29:01,380: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_unloaded (intentional=True) 140623538039248
[2017-06-15 12:29:01,385: INFO/job] Job 124: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-15 12:29:01,385: INFO/job] Job 124: not_deleted=True
[2017-06-15 12:29:01,385: INFO/job_scheduler] Job 124: complete_job: Updating cache
[2017-06-15 12:29:01,385: INFO/job_scheduler] Job 124: locked_item 140623538039248 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-15 12:29:01,387: INFO/job_scheduler] Job 124 completing (errored=False, cancelled=False)
[2017-06-15 12:29:01,406: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 0 tasked)
[2017-06-15 12:29:01,423: INFO/job_scheduler] Job 126: RunJobThread.run
[2017-06-15 12:29:01,424: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com unconfigure_lnet {}
[2017-06-15 12:29:01,443: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2082fdf5-197b-46d4-ac2a-91abc0c8126f
[2017-06-15 12:29:01,443: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2082fdf5-197b-46d4-ac2a-91abc0c8126f
[2017-06-15 12:29:01,491: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:29:01,491: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:29:01,491: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:29:01,491: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:29:01,492: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2082fdf5-197b-46d4-ac2a-91abc0c8126f
[2017-06-15 12:29:01,492: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2082fdf5-197b-46d4-ac2a-91abc0c8126f
[2017-06-15 12:29:01,524: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2082fdf5-197b-46d4-ac2a-91abc0c8126f
[2017-06-15 12:29:01,524: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2082fdf5-197b-46d4-ac2a-91abc0c8126f
[2017-06-15 12:29:01,524: INFO/job] Invalid result None fixed up on called to unconfigure_lnet with args {}
[2017-06-15 12:29:01,524: INFO/job_scheduler] Job 126 finished 1 steps successfully
[2017-06-15 12:29:01,605: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->unconfigured (intentional=True) 140623538039248
[2017-06-15 12:29:01,611: INFO/job] Job 126: StateChangeJob complete, setting state unconfigured on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-15 12:29:01,611: INFO/job] Job 126: not_deleted=True
[2017-06-15 12:29:01,611: INFO/job_scheduler] Job 126: complete_job: Updating cache
[2017-06-15 12:29:01,612: INFO/job_scheduler] Job 126: locked_item 140623538039248 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-15 12:29:01,616: INFO/job_scheduler] Job 126 completing (errored=False, cancelled=False)
[2017-06-15 12:29:01,630: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-15 12:29:01,670: INFO/job_scheduler] Job 127: RunJobThread.run
[2017-06-15 12:29:01,672: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com deregister_server {}
[2017-06-15 12:29:01,693: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5e95926d-9217-4e1d-a426-c3a1b4ed5018
[2017-06-15 12:29:01,693: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5e95926d-9217-4e1d-a426-c3a1b4ed5018
[2017-06-15 12:29:01,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:29:01,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:29:01,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:29:01,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 12:29:01,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5e95926d-9217-4e1d-a426-c3a1b4ed5018
[2017-06-15 12:29:01,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5e95926d-9217-4e1d-a426-c3a1b4ed5018
[2017-06-15 12:29:01,812: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5e95926d-9217-4e1d-a426-c3a1b4ed5018
[2017-06-15 12:29:01,812: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5e95926d-9217-4e1d-a426-c3a1b4ed5018
[2017-06-15 12:29:01,845: INFO/job] Deleting dependent lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration for host lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 12:29:01,845: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->removed (intentional=False) 140623739829200
[2017-06-15 12:29:01,866: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> 4
[2017-06-15 12:29:01,882: INFO/job] Deleting dependent lotus-32vm8.lotus.hpdd.lab.intel.com NTP configuration for host lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 12:29:01,882: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com NTP configuration unconfigured->removed (intentional=False) 140623739752144
[2017-06-15 12:29:01,888: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.ntp.NTPConfiguration'> 4
[2017-06-15 12:29:01,906: INFO/job] Deleting dependent lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration for host lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 12:29:01,907: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->removed (intentional=False) 140623739473744
[2017-06-15 12:29:01,916: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.corosync2.Corosync2Configuration'> 4
[2017-06-15 12:29:01,931: INFO/job] Deleting dependent lotus-32vm8.lotus.hpdd.lab.intel.com RSyslog configuration for host lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 12:29:01,931: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->removed (intentional=False) 140623739471952
[2017-06-15 12:29:01,937: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> 4
[2017-06-15 12:29:01,955: INFO/job] Deleting dependent lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration for host lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 12:29:01,955: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration unconfigured->removed (intentional=False) 140624142288016
[2017-06-15 12:29:01,963: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> 4
[2017-06-15 12:29:02,506: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->removed (intentional=False) 140623537940496
[2017-06-15 12:29:02,522: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-06-15 12:29:02,545: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.corosync2.Corosync2Configuration'> 4
[2017-06-15 12:29:02,572: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com NTP configuration unconfigured->removed (intentional=False) 140623538038480
[2017-06-15 12:29:02,577: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.ntp.NTPConfiguration'> 4
[2017-06-15 12:29:02,588: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->removed (intentional=False) 140623538037008
[2017-06-15 12:29:02,593: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> 4
[2017-06-15 12:29:02,617: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.host.ManagedHost'> 4
[2017-06-15 12:29:02,622: INFO/job_scheduler] Job 127 finished 2 steps successfully
[2017-06-15 12:29:02,648: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com managed->removed (intentional=True) 140623538005968
[2017-06-15 12:29:02,654: INFO/job] Job 127: StateChangeJob complete, setting state removed on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 12:29:02,654: INFO/job] Job 127: not_deleted=None
[2017-06-15 12:29:02,654: INFO/job_scheduler] Job 127: complete_job: Updating cache
[2017-06-15 12:29:02,654: INFO/job_scheduler] Job 127: locked_item 140623538005968 <class 'chroma_core.models.host.ManagedHost'> True None
[2017-06-15 12:29:02,655: INFO/job_scheduler] Job 127 completing (errored=False, cancelled=False)
[2017-06-15 12:29:02,702: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:13:13,743: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:13:13,744: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:13:13,761: INFO/command_plan] add_jobs: created Job 128 (Test for host connectivity)
[2017-06-15 16:13:13,807: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:13:13,808: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:13:13,904: INFO/command_plan] add_jobs: created Job 129 (Test for host connectivity)
[2017-06-15 16:13:13,913: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-15 16:13:13,923: INFO/job_scheduler] Job 128: RunJobThread.run
[2017-06-15 16:13:13,925: INFO/job_scheduler] Job 129: RunJobThread.run
[2017-06-15 16:13:13,937: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:13:14,046: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:13:14,047: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:13:14,082: INFO/command_plan] add_jobs: created Job 130 (Test for host connectivity)
[2017-06-15 16:13:14,158: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-15 16:13:14,170: INFO/job_scheduler] Job 130: RunJobThread.run
[2017-06-15 16:13:14,204: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:13:14,205: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:13:14,220: INFO/command_plan] add_jobs: created Job 131 (Test for host connectivity)
[2017-06-15 16:13:14,238: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-15 16:13:14,248: INFO/job_scheduler] Job 131: RunJobThread.run
[2017-06-15 16:13:22,230: INFO/job_scheduler] Job 128 finished 1 steps successfully
[2017-06-15 16:13:22,269: INFO/job_scheduler] Job 128: complete_job: Updating cache
[2017-06-15 16:13:22,270: INFO/job_scheduler] Job 128 completing (errored=False, cancelled=False)
[2017-06-15 16:13:22,310: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:13:22,313: INFO/job_scheduler] Job 129 finished 1 steps successfully
[2017-06-15 16:13:22,327: INFO/job_scheduler] Job 129: complete_job: Updating cache
[2017-06-15 16:13:22,327: INFO/job_scheduler] Job 129 completing (errored=False, cancelled=False)
[2017-06-15 16:13:22,378: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:13:34,372: INFO/job_scheduler] Job 131 finished 1 steps successfully
[2017-06-15 16:13:34,391: INFO/job_scheduler] Job 131: complete_job: Updating cache
[2017-06-15 16:13:34,391: INFO/job_scheduler] Job 131 completing (errored=False, cancelled=False)
[2017-06-15 16:13:34,433: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 16:13:34,478: INFO/job_scheduler] Job 130 finished 1 steps successfully
[2017-06-15 16:13:34,515: INFO/job_scheduler] Job 130: complete_job: Updating cache
[2017-06-15 16:13:34,516: INFO/job_scheduler] Job 130 completing (errored=False, cancelled=False)
[2017-06-15 16:13:34,550: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:13:34,719: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:13:34,719: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:13:34,732: INFO/command_plan] add_jobs: created Job 132 (Test for host connectivity)
[2017-06-15 16:13:34,785: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:13:34,785: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:13:34,807: INFO/command_plan] add_jobs: created Job 133 (Test for host connectivity)
[2017-06-15 16:13:34,817: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-15 16:13:34,835: INFO/job_scheduler] Job 132: RunJobThread.run
[2017-06-15 16:13:34,836: INFO/job_scheduler] Job 133: RunJobThread.run
[2017-06-15 16:13:34,852: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:13:34,917: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:13:34,917: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:13:34,941: INFO/command_plan] add_jobs: created Job 134 (Test for host connectivity)
[2017-06-15 16:13:34,963: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-15 16:13:34,970: INFO/job_scheduler] Job 134: RunJobThread.run
[2017-06-15 16:13:35,109: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:13:35,110: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:13:35,124: INFO/command_plan] add_jobs: created Job 135 (Test for host connectivity)
[2017-06-15 16:13:35,156: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-15 16:13:35,162: INFO/job_scheduler] Job 135: RunJobThread.run
[2017-06-15 16:13:41,114: INFO/job_scheduler] Job 133 finished 1 steps successfully
[2017-06-15 16:13:41,131: INFO/job_scheduler] Job 133: complete_job: Updating cache
[2017-06-15 16:13:41,131: INFO/job_scheduler] Job 133 completing (errored=False, cancelled=False)
[2017-06-15 16:13:41,173: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:13:41,185: INFO/job_scheduler] Job 134 finished 1 steps successfully
[2017-06-15 16:13:41,187: INFO/job_scheduler] Job 132 finished 1 steps successfully
[2017-06-15 16:13:41,206: INFO/job_scheduler] Job 134: complete_job: Updating cache
[2017-06-15 16:13:41,206: INFO/job_scheduler] Job 134 completing (errored=False, cancelled=False)
[2017-06-15 16:13:41,248: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:13:41,264: INFO/job_scheduler] Job 132: complete_job: Updating cache
[2017-06-15 16:13:41,264: INFO/job_scheduler] Job 132 completing (errored=False, cancelled=False)
[2017-06-15 16:13:41,274: INFO/job_scheduler] Job 135 finished 1 steps successfully
[2017-06-15 16:13:41,312: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 16:13:41,337: INFO/job_scheduler] Job 135: complete_job: Updating cache
[2017-06-15 16:13:41,338: INFO/job_scheduler] Job 135 completing (errored=False, cancelled=False)
[2017-06-15 16:13:41,390: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:13:42,559: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:13:42,559: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:13:42,614: INFO/command_plan] add_jobs: created Job 136 (Test for host connectivity)
[2017-06-15 16:13:42,651: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:13:42,652: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:13:42,666: INFO/command_plan] add_jobs: created Job 137 (Test for host connectivity)
[2017-06-15 16:13:42,822: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-15 16:13:42,824: INFO/job_scheduler] Job 136: RunJobThread.run
[2017-06-15 16:13:42,825: INFO/job_scheduler] Job 137: RunJobThread.run
[2017-06-15 16:13:42,835: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:13:42,887: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:13:42,887: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:13:42,905: INFO/command_plan] add_jobs: created Job 138 (Test for host connectivity)
[2017-06-15 16:13:42,936: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-15 16:13:42,953: INFO/job_scheduler] Job 138: RunJobThread.run
[2017-06-15 16:13:43,025: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:13:43,025: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:13:43,047: INFO/command_plan] add_jobs: created Job 139 (Test for host connectivity)
[2017-06-15 16:13:43,199: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-15 16:13:43,201: INFO/job_scheduler] Job 139: RunJobThread.run
[2017-06-15 16:13:48,925: INFO/job_scheduler] Job 137 finished 1 steps successfully
[2017-06-15 16:13:48,988: INFO/job_scheduler] Job 137: complete_job: Updating cache
[2017-06-15 16:13:48,988: INFO/job_scheduler] Job 137 completing (errored=False, cancelled=False)
[2017-06-15 16:13:49,071: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:13:49,247: INFO/job_scheduler] Job 136 finished 1 steps successfully
[2017-06-15 16:13:49,266: INFO/job_scheduler] Job 138 finished 1 steps successfully
[2017-06-15 16:13:49,338: INFO/job_scheduler] Job 139 finished 1 steps successfully
[2017-06-15 16:13:49,354: INFO/job_scheduler] Job 136: complete_job: Updating cache
[2017-06-15 16:13:49,355: INFO/job_scheduler] Job 136 completing (errored=False, cancelled=False)
[2017-06-15 16:13:49,413: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:13:49,421: INFO/job_scheduler] Job 138: complete_job: Updating cache
[2017-06-15 16:13:49,421: INFO/job_scheduler] Job 138 completing (errored=False, cancelled=False)
[2017-06-15 16:13:49,463: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 16:13:49,471: INFO/job_scheduler] Job 139: complete_job: Updating cache
[2017-06-15 16:13:49,471: INFO/job_scheduler] Job 139 completing (errored=False, cancelled=False)
[2017-06-15 16:13:49,543: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:13:50,713: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:13:50,714: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:13:50,729: INFO/command_plan] add_jobs: created Job 140 (Test for host connectivity)
[2017-06-15 16:13:50,760: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-15 16:13:50,766: INFO/job_scheduler] Job 140: RunJobThread.run
[2017-06-15 16:13:50,792: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:13:50,793: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:13:50,823: INFO/command_plan] add_jobs: created Job 141 (Test for host connectivity)
[2017-06-15 16:13:50,848: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-15 16:13:50,858: INFO/job_scheduler] Job 141: RunJobThread.run
[2017-06-15 16:13:50,943: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:13:50,945: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:13:50,987: INFO/command_plan] add_jobs: created Job 142 (Test for host connectivity)
[2017-06-15 16:13:51,016: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-15 16:13:51,055: INFO/job_scheduler] Job 142: RunJobThread.run
[2017-06-15 16:13:51,103: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:13:51,104: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:13:51,133: INFO/command_plan] add_jobs: created Job 143 (Test for host connectivity)
[2017-06-15 16:13:51,157: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-15 16:13:51,168: INFO/job_scheduler] Job 143: RunJobThread.run
[2017-06-15 16:13:56,959: INFO/job_scheduler] Job 140 finished 1 steps successfully
[2017-06-15 16:13:56,969: INFO/job_scheduler] Job 141 finished 1 steps successfully
[2017-06-15 16:13:57,029: INFO/job_scheduler] Job 140: complete_job: Updating cache
[2017-06-15 16:13:57,030: INFO/job_scheduler] Job 140 completing (errored=False, cancelled=False)
[2017-06-15 16:13:57,096: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:13:57,137: INFO/job_scheduler] Job 141: complete_job: Updating cache
[2017-06-15 16:13:57,137: INFO/job_scheduler] Job 141 completing (errored=False, cancelled=False)
[2017-06-15 16:13:57,231: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:13:57,390: INFO/job_scheduler] Job 143 finished 1 steps successfully
[2017-06-15 16:13:57,421: INFO/job_scheduler] Job 143: complete_job: Updating cache
[2017-06-15 16:13:57,422: INFO/job_scheduler] Job 143 completing (errored=False, cancelled=False)
[2017-06-15 16:13:57,464: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 16:13:57,516: INFO/job_scheduler] Job 142 finished 1 steps successfully
[2017-06-15 16:13:57,547: INFO/job_scheduler] Job 142: complete_job: Updating cache
[2017-06-15 16:13:57,547: INFO/job_scheduler] Job 142 completing (errored=False, cancelled=False)
[2017-06-15 16:13:57,604: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:13:58,782: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:13:58,784: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:13:58,804: INFO/command_plan] add_jobs: created Job 144 (Test for host connectivity)
[2017-06-15 16:13:58,820: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-15 16:13:58,823: INFO/job_scheduler] Job 144: RunJobThread.run
[2017-06-15 16:13:58,892: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:13:58,892: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:13:58,912: INFO/command_plan] add_jobs: created Job 145 (Test for host connectivity)
[2017-06-15 16:13:59,002: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:13:59,003: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:13:59,029: INFO/command_plan] add_jobs: created Job 146 (Test for host connectivity)
[2017-06-15 16:13:59,037: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-15 16:13:59,044: INFO/job_scheduler] Job 145: RunJobThread.run
[2017-06-15 16:13:59,052: INFO/job_scheduler] Job 146: RunJobThread.run
[2017-06-15 16:13:59,074: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:13:59,150: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:13:59,152: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:13:59,172: INFO/command_plan] add_jobs: created Job 147 (Test for host connectivity)
[2017-06-15 16:13:59,246: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-15 16:13:59,273: INFO/job_scheduler] Job 147: RunJobThread.run
[2017-06-15 16:14:05,100: INFO/job_scheduler] Job 144 finished 1 steps successfully
[2017-06-15 16:14:05,128: INFO/job_scheduler] Job 144: complete_job: Updating cache
[2017-06-15 16:14:05,128: INFO/job_scheduler] Job 144 completing (errored=False, cancelled=False)
[2017-06-15 16:14:05,187: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:14:05,219: INFO/job_scheduler] Job 145 finished 1 steps successfully
[2017-06-15 16:14:05,269: INFO/job_scheduler] Job 145: complete_job: Updating cache
[2017-06-15 16:14:05,269: INFO/job_scheduler] Job 145 completing (errored=False, cancelled=False)
[2017-06-15 16:14:05,329: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:14:05,398: INFO/job_scheduler] Job 146 finished 1 steps successfully
[2017-06-15 16:14:05,412: INFO/job_scheduler] Job 146: complete_job: Updating cache
[2017-06-15 16:14:05,412: INFO/job_scheduler] Job 146 completing (errored=False, cancelled=False)
[2017-06-15 16:14:05,450: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 16:14:05,573: INFO/job_scheduler] Job 147 finished 1 steps successfully
[2017-06-15 16:14:05,594: INFO/job_scheduler] Job 147: complete_job: Updating cache
[2017-06-15 16:14:05,595: INFO/job_scheduler] Job 147 completing (errored=False, cancelled=False)
[2017-06-15 16:14:05,636: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:14:06,818: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:14:06,820: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:14:06,836: INFO/command_plan] add_jobs: created Job 148 (Test for host connectivity)
[2017-06-15 16:14:06,852: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-15 16:14:06,872: INFO/job_scheduler] Job 148: RunJobThread.run
[2017-06-15 16:14:06,901: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:14:06,901: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:14:06,920: INFO/command_plan] add_jobs: created Job 149 (Test for host connectivity)
[2017-06-15 16:14:06,946: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-15 16:14:06,966: INFO/job_scheduler] Job 149: RunJobThread.run
[2017-06-15 16:14:07,043: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:14:07,043: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:14:07,080: INFO/command_plan] add_jobs: created Job 150 (Test for host connectivity)
[2017-06-15 16:14:07,135: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-15 16:14:07,140: INFO/job_scheduler] Job 150: RunJobThread.run
[2017-06-15 16:14:07,193: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:14:07,193: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:14:07,213: INFO/command_plan] add_jobs: created Job 151 (Test for host connectivity)
[2017-06-15 16:14:07,221: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-15 16:14:07,241: INFO/job_scheduler] Job 151: RunJobThread.run
[2017-06-15 16:14:13,097: INFO/job_scheduler] Job 149 finished 1 steps successfully
[2017-06-15 16:14:13,144: INFO/job_scheduler] Job 149: complete_job: Updating cache
[2017-06-15 16:14:13,144: INFO/job_scheduler] Job 149 completing (errored=False, cancelled=False)
[2017-06-15 16:14:13,204: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:14:13,208: INFO/job_scheduler] Job 148 finished 1 steps successfully
[2017-06-15 16:14:13,223: INFO/job_scheduler] Job 148: complete_job: Updating cache
[2017-06-15 16:14:13,223: INFO/job_scheduler] Job 148 completing (errored=False, cancelled=False)
[2017-06-15 16:14:13,296: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:14:13,411: INFO/job_scheduler] Job 150 finished 1 steps successfully
[2017-06-15 16:14:13,428: INFO/job_scheduler] Job 150: complete_job: Updating cache
[2017-06-15 16:14:13,428: INFO/job_scheduler] Job 150 completing (errored=False, cancelled=False)
[2017-06-15 16:14:13,480: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 16:14:13,558: INFO/job_scheduler] Job 151 finished 1 steps successfully
[2017-06-15 16:14:13,610: INFO/job_scheduler] Job 151: complete_job: Updating cache
[2017-06-15 16:14:13,611: INFO/job_scheduler] Job 151 completing (errored=False, cancelled=False)
[2017-06-15 16:14:13,652: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:14:14,825: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:14:14,826: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:14:14,844: INFO/command_plan] add_jobs: created Job 152 (Test for host connectivity)
[2017-06-15 16:14:14,881: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:14:14,881: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:14:14,902: INFO/command_plan] add_jobs: created Job 153 (Test for host connectivity)
[2017-06-15 16:14:14,912: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-15 16:14:14,920: INFO/job_scheduler] Job 152: RunJobThread.run
[2017-06-15 16:14:14,952: INFO/job_scheduler] Job 153: RunJobThread.run
[2017-06-15 16:14:14,983: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:14:14,984: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:14:14,996: INFO/command_plan] add_jobs: created Job 154 (Test for host connectivity)
[2017-06-15 16:14:15,008: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-15 16:14:15,013: INFO/job_scheduler] Job 154: RunJobThread.run
[2017-06-15 16:14:15,044: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:14:15,179: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:14:15,179: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:14:15,210: INFO/command_plan] add_jobs: created Job 155 (Test for host connectivity)
[2017-06-15 16:14:15,482: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-15 16:14:15,487: INFO/job_scheduler] Job 155: RunJobThread.run
[2017-06-15 16:14:21,135: INFO/job_scheduler] Job 153 finished 1 steps successfully
[2017-06-15 16:14:21,176: INFO/job_scheduler] Job 153: complete_job: Updating cache
[2017-06-15 16:14:21,176: INFO/job_scheduler] Job 153 completing (errored=False, cancelled=False)
[2017-06-15 16:14:21,228: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:14:21,377: INFO/job_scheduler] Job 154 finished 1 steps successfully
[2017-06-15 16:14:21,412: INFO/job_scheduler] Job 152 finished 1 steps successfully
[2017-06-15 16:14:21,418: INFO/job_scheduler] Job 154: complete_job: Updating cache
[2017-06-15 16:14:21,418: INFO/job_scheduler] Job 154 completing (errored=False, cancelled=False)
[2017-06-15 16:14:21,460: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:14:21,477: INFO/job_scheduler] Job 152: complete_job: Updating cache
[2017-06-15 16:14:21,477: INFO/job_scheduler] Job 152 completing (errored=False, cancelled=False)
[2017-06-15 16:14:21,511: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 16:14:21,952: INFO/job_scheduler] Job 155 finished 1 steps successfully
[2017-06-15 16:14:22,010: INFO/job_scheduler] Job 155: complete_job: Updating cache
[2017-06-15 16:14:22,010: INFO/job_scheduler] Job 155 completing (errored=False, cancelled=False)
[2017-06-15 16:14:22,056: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:14:23,230: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:14:23,230: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:14:23,251: INFO/command_plan] add_jobs: created Job 156 (Test for host connectivity)
[2017-06-15 16:14:23,271: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-15 16:14:23,289: INFO/job_scheduler] Job 156: RunJobThread.run
[2017-06-15 16:14:23,325: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:14:23,325: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:14:23,337: INFO/command_plan] add_jobs: created Job 157 (Test for host connectivity)
[2017-06-15 16:14:23,380: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-15 16:14:23,388: INFO/job_scheduler] Job 157: RunJobThread.run
[2017-06-15 16:14:23,447: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:14:23,448: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:14:23,480: INFO/command_plan] add_jobs: created Job 158 (Test for host connectivity)
[2017-06-15 16:14:23,545: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-15 16:14:23,565: INFO/job_scheduler] Job 158: RunJobThread.run
[2017-06-15 16:14:23,613: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:14:23,615: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:14:23,634: INFO/command_plan] add_jobs: created Job 159 (Test for host connectivity)
[2017-06-15 16:14:23,685: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-15 16:14:23,716: INFO/job_scheduler] Job 159: RunJobThread.run
[2017-06-15 16:14:29,550: INFO/job_scheduler] Job 156 finished 1 steps successfully
[2017-06-15 16:14:29,577: INFO/job_scheduler] Job 157 finished 1 steps successfully
[2017-06-15 16:14:29,605: INFO/job_scheduler] Job 156: complete_job: Updating cache
[2017-06-15 16:14:29,606: INFO/job_scheduler] Job 156 completing (errored=False, cancelled=False)
[2017-06-15 16:14:29,645: INFO/job_scheduler] Job 158 finished 1 steps successfully
[2017-06-15 16:14:29,653: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:14:29,676: INFO/job_scheduler] Job 157: complete_job: Updating cache
[2017-06-15 16:14:29,677: INFO/job_scheduler] Job 157 completing (errored=False, cancelled=False)
[2017-06-15 16:14:29,753: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:14:29,768: INFO/job_scheduler] Job 158: complete_job: Updating cache
[2017-06-15 16:14:29,768: INFO/job_scheduler] Job 158 completing (errored=False, cancelled=False)
[2017-06-15 16:14:29,825: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 16:14:29,891: INFO/job_scheduler] Job 159 finished 1 steps successfully
[2017-06-15 16:14:29,925: INFO/job_scheduler] Job 159: complete_job: Updating cache
[2017-06-15 16:14:29,926: INFO/job_scheduler] Job 159 completing (errored=False, cancelled=False)
[2017-06-15 16:14:29,959: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:14:31,128: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:14:31,133: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:14:31,150: INFO/command_plan] add_jobs: created Job 160 (Test for host connectivity)
[2017-06-15 16:14:31,182: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-15 16:14:31,192: INFO/job_scheduler] Job 160: RunJobThread.run
[2017-06-15 16:14:31,314: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:14:31,315: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:14:31,350: INFO/command_plan] add_jobs: created Job 161 (Test for host connectivity)
[2017-06-15 16:14:31,368: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-15 16:14:31,372: INFO/job_scheduler] Job 161: RunJobThread.run
[2017-06-15 16:14:31,410: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:14:31,411: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:14:31,421: INFO/command_plan] add_jobs: created Job 162 (Test for host connectivity)
[2017-06-15 16:14:31,476: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-15 16:14:31,492: INFO/job_scheduler] Job 162: RunJobThread.run
[2017-06-15 16:14:31,561: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:14:31,561: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:14:31,584: INFO/command_plan] add_jobs: created Job 163 (Test for host connectivity)
[2017-06-15 16:14:31,615: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-15 16:14:31,650: INFO/job_scheduler] Job 163: RunJobThread.run
[2017-06-15 16:14:37,484: INFO/job_scheduler] Job 161 finished 1 steps successfully
[2017-06-15 16:14:37,540: INFO/job_scheduler] Job 160 finished 1 steps successfully
[2017-06-15 16:14:37,542: INFO/job_scheduler] Job 161: complete_job: Updating cache
[2017-06-15 16:14:37,543: INFO/job_scheduler] Job 161 completing (errored=False, cancelled=False)
[2017-06-15 16:14:37,624: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:14:37,658: INFO/job_scheduler] Job 160: complete_job: Updating cache
[2017-06-15 16:14:37,658: INFO/job_scheduler] Job 160 completing (errored=False, cancelled=False)
[2017-06-15 16:14:37,740: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:14:37,810: INFO/job_scheduler] Job 162 finished 1 steps successfully
[2017-06-15 16:14:37,816: INFO/job_scheduler] Job 162: complete_job: Updating cache
[2017-06-15 16:14:37,816: INFO/job_scheduler] Job 162 completing (errored=False, cancelled=False)
[2017-06-15 16:14:37,883: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 16:14:37,921: INFO/job_scheduler] Job 163 finished 1 steps successfully
[2017-06-15 16:14:37,966: INFO/job_scheduler] Job 163: complete_job: Updating cache
[2017-06-15 16:14:37,966: INFO/job_scheduler] Job 163 completing (errored=False, cancelled=False)
[2017-06-15 16:14:38,027: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:14:39,220: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:14:39,222: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:14:39,232: INFO/command_plan] add_jobs: created Job 164 (Test for host connectivity)
[2017-06-15 16:14:39,247: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-15 16:14:39,257: INFO/job_scheduler] Job 164: RunJobThread.run
[2017-06-15 16:14:39,300: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:14:39,301: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:14:39,324: INFO/command_plan] add_jobs: created Job 165 (Test for host connectivity)
[2017-06-15 16:14:39,359: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-15 16:14:39,372: INFO/job_scheduler] Job 165: RunJobThread.run
[2017-06-15 16:14:39,397: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:14:39,399: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:14:39,429: INFO/command_plan] add_jobs: created Job 166 (Test for host connectivity)
[2017-06-15 16:14:39,453: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-15 16:14:39,474: INFO/job_scheduler] Job 166: RunJobThread.run
[2017-06-15 16:14:39,562: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:14:39,562: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:14:39,576: INFO/command_plan] add_jobs: created Job 167 (Test for host connectivity)
[2017-06-15 16:14:39,616: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-15 16:14:39,621: INFO/job_scheduler] Job 167: RunJobThread.run
[2017-06-15 16:14:45,567: INFO/job_scheduler] Job 165 finished 1 steps successfully
[2017-06-15 16:14:45,572: INFO/job_scheduler] Job 164 finished 1 steps successfully
[2017-06-15 16:14:45,606: INFO/job_scheduler] Job 165: complete_job: Updating cache
[2017-06-15 16:14:45,606: INFO/job_scheduler] Job 165 completing (errored=False, cancelled=False)
[2017-06-15 16:14:45,664: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:14:45,680: INFO/job_scheduler] Job 164: complete_job: Updating cache
[2017-06-15 16:14:45,681: INFO/job_scheduler] Job 164 completing (errored=False, cancelled=False)
[2017-06-15 16:14:45,733: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:14:45,765: INFO/job_scheduler] Job 167 finished 1 steps successfully
[2017-06-15 16:14:45,773: INFO/job_scheduler] Job 166 finished 1 steps successfully
[2017-06-15 16:14:45,797: INFO/job_scheduler] Job 167: complete_job: Updating cache
[2017-06-15 16:14:45,797: INFO/job_scheduler] Job 167 completing (errored=False, cancelled=False)
[2017-06-15 16:14:45,922: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 16:14:45,940: INFO/job_scheduler] Job 166: complete_job: Updating cache
[2017-06-15 16:14:45,940: INFO/job_scheduler] Job 166 completing (errored=False, cancelled=False)
[2017-06-15 16:14:45,991: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:14:47,193: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:14:47,194: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:14:47,213: INFO/command_plan] add_jobs: created Job 168 (Test for host connectivity)
[2017-06-15 16:14:47,247: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:14:47,247: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:14:47,257: INFO/command_plan] add_jobs: created Job 169 (Test for host connectivity)
[2017-06-15 16:14:47,278: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-15 16:14:47,298: INFO/job_scheduler] Job 168: RunJobThread.run
[2017-06-15 16:14:47,306: INFO/job_scheduler] Job 169: RunJobThread.run
[2017-06-15 16:14:47,338: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:14:47,339: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:14:47,366: INFO/command_plan] add_jobs: created Job 170 (Test for host connectivity)
[2017-06-15 16:14:47,390: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-15 16:14:47,432: INFO/job_scheduler] Job 170: RunJobThread.run
[2017-06-15 16:14:47,539: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:14:47,542: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:14:47,571: INFO/command_plan] add_jobs: created Job 171 (Test for host connectivity)
[2017-06-15 16:14:47,601: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-15 16:14:47,647: INFO/job_scheduler] Job 171: RunJobThread.run
[2017-06-15 16:14:47,685: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-15 16:14:53,456: INFO/job_scheduler] Job 169 finished 1 steps successfully
[2017-06-15 16:14:53,520: INFO/job_scheduler] Job 169: complete_job: Updating cache
[2017-06-15 16:14:53,521: INFO/job_scheduler] Job 169 completing (errored=False, cancelled=False)
[2017-06-15 16:14:53,546: INFO/job_scheduler] Job 168 finished 1 steps successfully
[2017-06-15 16:14:53,571: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:14:53,579: INFO/job_scheduler] Job 168: complete_job: Updating cache
[2017-06-15 16:14:53,579: INFO/job_scheduler] Job 168 completing (errored=False, cancelled=False)
[2017-06-15 16:14:53,630: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:14:53,661: INFO/job_scheduler] Job 170 finished 1 steps successfully
[2017-06-15 16:14:53,681: INFO/job_scheduler] Job 170: complete_job: Updating cache
[2017-06-15 16:14:53,681: INFO/job_scheduler] Job 170 completing (errored=False, cancelled=False)
[2017-06-15 16:14:53,737: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 16:14:53,738: INFO/job_scheduler] Job 171 finished 1 steps successfully
[2017-06-15 16:14:53,772: INFO/job_scheduler] Job 171: complete_job: Updating cache
[2017-06-15 16:14:53,772: INFO/job_scheduler] Job 171 completing (errored=False, cancelled=False)
[2017-06-15 16:14:53,820: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:14:55,033: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:14:55,033: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:14:55,053: INFO/command_plan] add_jobs: created Job 172 (Test for host connectivity)
[2017-06-15 16:14:55,086: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-15 16:14:55,088: INFO/job_scheduler] Job 172: RunJobThread.run
[2017-06-15 16:14:55,124: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:14:55,125: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:14:55,164: INFO/command_plan] add_jobs: created Job 173 (Test for host connectivity)
[2017-06-15 16:14:55,247: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:14:55,248: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:14:55,262: INFO/command_plan] add_jobs: created Job 174 (Test for host connectivity)
[2017-06-15 16:14:55,289: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-15 16:14:55,296: INFO/job_scheduler] Job 173: RunJobThread.run
[2017-06-15 16:14:55,296: INFO/job_scheduler] Job 174: RunJobThread.run
[2017-06-15 16:14:55,312: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:14:55,437: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:14:55,438: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:14:55,475: INFO/command_plan] add_jobs: created Job 175 (Test for host connectivity)
[2017-06-15 16:14:55,545: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-15 16:14:55,583: INFO/job_scheduler] Job 175: RunJobThread.run
[2017-06-15 16:15:01,446: INFO/job_scheduler] Job 172 finished 1 steps successfully
[2017-06-15 16:15:01,502: INFO/job_scheduler] Job 172: complete_job: Updating cache
[2017-06-15 16:15:01,502: INFO/job_scheduler] Job 172 completing (errored=False, cancelled=False)
[2017-06-15 16:15:01,535: INFO/job_scheduler] Job 174 finished 1 steps successfully
[2017-06-15 16:15:01,546: INFO/job_scheduler] Job 173 finished 1 steps successfully
[2017-06-15 16:15:01,602: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:15:01,653: INFO/job_scheduler] Job 174: complete_job: Updating cache
[2017-06-15 16:15:01,653: INFO/job_scheduler] Job 174 completing (errored=False, cancelled=False)
[2017-06-15 16:15:01,711: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:15:01,719: INFO/job_scheduler] Job 173: complete_job: Updating cache
[2017-06-15 16:15:01,719: INFO/job_scheduler] Job 173 completing (errored=False, cancelled=False)
[2017-06-15 16:15:01,781: INFO/job_scheduler] Job 175 finished 1 steps successfully
[2017-06-15 16:15:01,805: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 16:15:01,820: INFO/job_scheduler] Job 175: complete_job: Updating cache
[2017-06-15 16:15:01,820: INFO/job_scheduler] Job 175 completing (errored=False, cancelled=False)
[2017-06-15 16:15:01,877: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:15:03,036: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:15:03,037: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:15:03,060: INFO/command_plan] add_jobs: created Job 176 (Test for host connectivity)
[2017-06-15 16:15:03,095: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-15 16:15:03,110: INFO/job_scheduler] Job 176: RunJobThread.run
[2017-06-15 16:15:03,252: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:15:03,253: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:15:03,302: INFO/command_plan] add_jobs: created Job 177 (Test for host connectivity)
[2017-06-15 16:15:03,364: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-15 16:15:03,367: INFO/job_scheduler] Job 177: RunJobThread.run
[2017-06-15 16:15:03,407: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:15:03,407: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:15:03,427: INFO/command_plan] add_jobs: created Job 178 (Test for host connectivity)
[2017-06-15 16:15:03,472: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-15 16:15:03,487: INFO/job_scheduler] Job 178: RunJobThread.run
[2017-06-15 16:15:03,553: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:15:03,555: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:15:03,578: INFO/command_plan] add_jobs: created Job 179 (Test for host connectivity)
[2017-06-15 16:15:03,602: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-15 16:15:03,618: INFO/job_scheduler] Job 179: RunJobThread.run
[2017-06-15 16:15:09,362: INFO/job_scheduler] Job 176 finished 1 steps successfully
[2017-06-15 16:15:09,376: INFO/job_scheduler] Job 176: complete_job: Updating cache
[2017-06-15 16:15:09,376: INFO/job_scheduler] Job 176 completing (errored=False, cancelled=False)
[2017-06-15 16:15:09,419: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:15:09,542: INFO/job_scheduler] Job 177 finished 1 steps successfully
[2017-06-15 16:15:09,567: INFO/job_scheduler] Job 177: complete_job: Updating cache
[2017-06-15 16:15:09,567: INFO/job_scheduler] Job 177 completing (errored=False, cancelled=False)
[2017-06-15 16:15:09,728: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:15:09,853: INFO/job_scheduler] Job 179 finished 1 steps successfully
[2017-06-15 16:15:09,909: INFO/job_scheduler] Job 179: complete_job: Updating cache
[2017-06-15 16:15:09,909: INFO/job_scheduler] Job 179 completing (errored=False, cancelled=False)
[2017-06-15 16:15:09,950: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 16:15:10,038: INFO/job_scheduler] Job 178 finished 1 steps successfully
[2017-06-15 16:15:10,084: INFO/job_scheduler] Job 178: complete_job: Updating cache
[2017-06-15 16:15:10,084: INFO/job_scheduler] Job 178 completing (errored=False, cancelled=False)
[2017-06-15 16:15:10,128: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:15:11,310: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:15:11,311: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:15:11,325: INFO/command_plan] add_jobs: created Job 180 (Test for host connectivity)
[2017-06-15 16:15:11,353: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-15 16:15:11,360: INFO/job_scheduler] Job 180: RunJobThread.run
[2017-06-15 16:15:11,396: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:15:11,397: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:15:11,410: INFO/command_plan] add_jobs: created Job 181 (Test for host connectivity)
[2017-06-15 16:15:11,428: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-15 16:15:11,432: INFO/job_scheduler] Job 181: RunJobThread.run
[2017-06-15 16:15:11,498: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:15:11,498: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:15:11,558: INFO/command_plan] add_jobs: created Job 182 (Test for host connectivity)
[2017-06-15 16:15:11,627: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-15 16:15:11,631: INFO/job_scheduler] Job 182: RunJobThread.run
[2017-06-15 16:15:11,664: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:15:11,666: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:15:11,684: INFO/command_plan] add_jobs: created Job 183 (Test for host connectivity)
[2017-06-15 16:15:11,702: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-15 16:15:11,711: INFO/job_scheduler] Job 183: RunJobThread.run
[2017-06-15 16:15:17,633: INFO/job_scheduler] Job 180 finished 1 steps successfully
[2017-06-15 16:15:17,690: INFO/job_scheduler] Job 180: complete_job: Updating cache
[2017-06-15 16:15:17,690: INFO/job_scheduler] Job 180 completing (errored=False, cancelled=False)
[2017-06-15 16:15:17,741: INFO/job_scheduler] Job 181 finished 1 steps successfully
[2017-06-15 16:15:17,748: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:15:17,757: INFO/job_scheduler] Job 181: complete_job: Updating cache
[2017-06-15 16:15:17,757: INFO/job_scheduler] Job 181 completing (errored=False, cancelled=False)
[2017-06-15 16:15:17,800: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:15:17,967: INFO/job_scheduler] Job 182 finished 1 steps successfully
[2017-06-15 16:15:17,990: INFO/job_scheduler] Job 182: complete_job: Updating cache
[2017-06-15 16:15:17,990: INFO/job_scheduler] Job 182 completing (errored=False, cancelled=False)
[2017-06-15 16:15:18,044: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 16:15:18,271: INFO/job_scheduler] Job 183 finished 1 steps successfully
[2017-06-15 16:15:18,282: INFO/job_scheduler] Job 183: complete_job: Updating cache
[2017-06-15 16:15:18,282: INFO/job_scheduler] Job 183 completing (errored=False, cancelled=False)
[2017-06-15 16:15:18,327: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:15:19,542: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:15:19,542: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:15:19,565: INFO/command_plan] add_jobs: created Job 184 (Test for host connectivity)
[2017-06-15 16:15:19,602: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-15 16:15:19,629: INFO/job_scheduler] Job 184: RunJobThread.run
[2017-06-15 16:15:19,663: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:15:19,663: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:15:19,681: INFO/command_plan] add_jobs: created Job 185 (Test for host connectivity)
[2017-06-15 16:15:19,728: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:15:19,728: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:15:19,767: INFO/command_plan] add_jobs: created Job 186 (Test for host connectivity)
[2017-06-15 16:15:19,785: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-15 16:15:19,793: INFO/job_scheduler] Job 185: RunJobThread.run
[2017-06-15 16:15:19,794: INFO/job_scheduler] Job 186: RunJobThread.run
[2017-06-15 16:15:19,894: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:15:19,894: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:15:19,915: INFO/command_plan] add_jobs: created Job 187 (Test for host connectivity)
[2017-06-15 16:15:19,925: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-15 16:15:19,932: INFO/job_scheduler] Job 187: RunJobThread.run
[2017-06-15 16:15:20,018: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-15 16:15:25,920: INFO/job_scheduler] Job 184 finished 1 steps successfully
[2017-06-15 16:15:25,955: INFO/job_scheduler] Job 184: complete_job: Updating cache
[2017-06-15 16:15:25,955: INFO/job_scheduler] Job 184 completing (errored=False, cancelled=False)
[2017-06-15 16:15:25,998: INFO/job_scheduler] Job 185 finished 1 steps successfully
[2017-06-15 16:15:26,014: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:15:26,040: INFO/job_scheduler] Job 185: complete_job: Updating cache
[2017-06-15 16:15:26,040: INFO/job_scheduler] Job 185 completing (errored=False, cancelled=False)
[2017-06-15 16:15:26,090: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:15:26,107: INFO/job_scheduler] Job 186 finished 1 steps successfully
[2017-06-15 16:15:26,131: INFO/job_scheduler] Job 186: complete_job: Updating cache
[2017-06-15 16:15:26,131: INFO/job_scheduler] Job 186 completing (errored=False, cancelled=False)
[2017-06-15 16:15:26,189: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 16:15:26,241: INFO/job_scheduler] Job 187 finished 1 steps successfully
[2017-06-15 16:15:26,280: INFO/job_scheduler] Job 187: complete_job: Updating cache
[2017-06-15 16:15:26,281: INFO/job_scheduler] Job 187 completing (errored=False, cancelled=False)
[2017-06-15 16:15:26,330: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:15:27,515: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:15:27,516: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:15:27,530: INFO/command_plan] add_jobs: created Job 188 (Test for host connectivity)
[2017-06-15 16:15:27,547: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-15 16:15:27,552: INFO/job_scheduler] Job 188: RunJobThread.run
[2017-06-15 16:15:27,589: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:15:27,589: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:15:27,613: INFO/command_plan] add_jobs: created Job 189 (Test for host connectivity)
[2017-06-15 16:15:27,654: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-15 16:15:27,676: INFO/job_scheduler] Job 189: RunJobThread.run
[2017-06-15 16:15:27,741: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:15:27,742: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:15:27,772: INFO/command_plan] add_jobs: created Job 190 (Test for host connectivity)
[2017-06-15 16:15:27,797: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-15 16:15:27,839: INFO/job_scheduler] Job 190: RunJobThread.run
[2017-06-15 16:15:27,931: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:15:27,931: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:15:27,951: INFO/command_plan] add_jobs: created Job 191 (Test for host connectivity)
[2017-06-15 16:15:27,971: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-15 16:15:27,978: INFO/job_scheduler] Job 191: RunJobThread.run
[2017-06-15 16:15:33,876: INFO/job_scheduler] Job 189 finished 1 steps successfully
[2017-06-15 16:15:33,928: INFO/job_scheduler] Job 188 finished 1 steps successfully
[2017-06-15 16:15:33,929: INFO/job_scheduler] Job 189: complete_job: Updating cache
[2017-06-15 16:15:33,929: INFO/job_scheduler] Job 189 completing (errored=False, cancelled=False)
[2017-06-15 16:15:33,987: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:15:34,037: INFO/job_scheduler] Job 188: complete_job: Updating cache
[2017-06-15 16:15:34,040: INFO/job_scheduler] Job 188 completing (errored=False, cancelled=False)
[2017-06-15 16:15:34,098: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:15:34,173: INFO/job_scheduler] Job 191 finished 1 steps successfully
[2017-06-15 16:15:34,175: INFO/job_scheduler] Job 190 finished 1 steps successfully
[2017-06-15 16:15:34,228: INFO/job_scheduler] Job 191: complete_job: Updating cache
[2017-06-15 16:15:34,229: INFO/job_scheduler] Job 191 completing (errored=False, cancelled=False)
[2017-06-15 16:15:34,272: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 16:15:34,295: INFO/job_scheduler] Job 190: complete_job: Updating cache
[2017-06-15 16:15:34,295: INFO/job_scheduler] Job 190 completing (errored=False, cancelled=False)
[2017-06-15 16:15:34,345: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:15:35,499: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:15:35,499: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:15:35,512: INFO/command_plan] add_jobs: created Job 192 (Test for host connectivity)
[2017-06-15 16:15:35,545: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:15:35,545: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:15:35,571: INFO/command_plan] add_jobs: created Job 193 (Test for host connectivity)
[2017-06-15 16:15:35,589: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-15 16:15:35,617: INFO/job_scheduler] Job 192: RunJobThread.run
[2017-06-15 16:15:35,638: INFO/job_scheduler] Job 193: RunJobThread.run
[2017-06-15 16:15:35,646: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:15:35,759: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:15:35,760: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:15:35,786: INFO/command_plan] add_jobs: created Job 194 (Test for host connectivity)
[2017-06-15 16:15:35,854: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:15:35,854: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:15:35,878: INFO/command_plan] add_jobs: created Job 195 (Test for host connectivity)
[2017-06-15 16:15:35,901: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 2 tasked)
[2017-06-15 16:15:35,906: INFO/job_scheduler] Job 194: RunJobThread.run
[2017-06-15 16:15:35,907: INFO/job_scheduler] Job 195: RunJobThread.run
[2017-06-15 16:15:35,917: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-15 16:15:41,836: INFO/job_scheduler] Job 193 finished 1 steps successfully
[2017-06-15 16:15:41,877: INFO/job_scheduler] Job 193: complete_job: Updating cache
[2017-06-15 16:15:41,877: INFO/job_scheduler] Job 193 completing (errored=False, cancelled=False)
[2017-06-15 16:15:41,934: INFO/job_scheduler] Job 192 finished 1 steps successfully
[2017-06-15 16:15:42,052: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:15:42,095: INFO/job_scheduler] Job 192: complete_job: Updating cache
[2017-06-15 16:15:42,095: INFO/job_scheduler] Job 192 completing (errored=False, cancelled=False)
[2017-06-15 16:15:42,177: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:15:42,243: INFO/job_scheduler] Job 194 finished 1 steps successfully
[2017-06-15 16:15:42,255: INFO/job_scheduler] Job 194: complete_job: Updating cache
[2017-06-15 16:15:42,256: INFO/job_scheduler] Job 194 completing (errored=False, cancelled=False)
[2017-06-15 16:15:42,296: INFO/job_scheduler] Job 195 finished 1 steps successfully
[2017-06-15 16:15:42,316: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 16:15:42,330: INFO/job_scheduler] Job 195: complete_job: Updating cache
[2017-06-15 16:15:42,330: INFO/job_scheduler] Job 195 completing (errored=False, cancelled=False)
[2017-06-15 16:15:42,365: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:15:43,552: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:15:43,554: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:15:43,564: INFO/command_plan] add_jobs: created Job 196 (Test for host connectivity)
[2017-06-15 16:15:43,580: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-15 16:15:43,582: INFO/job_scheduler] Job 196: RunJobThread.run
[2017-06-15 16:15:43,664: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:15:43,665: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:15:43,679: INFO/command_plan] add_jobs: created Job 197 (Test for host connectivity)
[2017-06-15 16:15:43,735: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-15 16:15:43,750: INFO/job_scheduler] Job 197: RunJobThread.run
[2017-06-15 16:15:43,786: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:15:43,786: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:15:43,796: INFO/command_plan] add_jobs: created Job 198 (Test for host connectivity)
[2017-06-15 16:15:43,849: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-15 16:15:43,864: INFO/job_scheduler] Job 198: RunJobThread.run
[2017-06-15 16:15:43,923: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:15:43,923: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:15:43,946: INFO/command_plan] add_jobs: created Job 199 (Test for host connectivity)
[2017-06-15 16:15:43,976: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-15 16:15:43,981: INFO/job_scheduler] Job 199: RunJobThread.run
[2017-06-15 16:15:49,689: INFO/job_scheduler] Job 196 finished 1 steps successfully
[2017-06-15 16:15:49,723: INFO/job_scheduler] Job 196: complete_job: Updating cache
[2017-06-15 16:15:49,723: INFO/job_scheduler] Job 196 completing (errored=False, cancelled=False)
[2017-06-15 16:15:49,790: INFO/job_scheduler] Job 197 finished 1 steps successfully
[2017-06-15 16:15:49,806: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:15:49,867: INFO/job_scheduler] Job 197: complete_job: Updating cache
[2017-06-15 16:15:49,867: INFO/job_scheduler] Job 197 completing (errored=False, cancelled=False)
[2017-06-15 16:15:49,907: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:15:50,174: INFO/job_scheduler] Job 198 finished 1 steps successfully
[2017-06-15 16:15:50,174: INFO/job_scheduler] Job 199 finished 1 steps successfully
[2017-06-15 16:15:50,195: INFO/job_scheduler] Job 198: complete_job: Updating cache
[2017-06-15 16:15:50,195: INFO/job_scheduler] Job 198 completing (errored=False, cancelled=False)
[2017-06-15 16:15:50,237: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 16:15:50,253: INFO/job_scheduler] Job 199: complete_job: Updating cache
[2017-06-15 16:15:50,253: INFO/job_scheduler] Job 199 completing (errored=False, cancelled=False)
[2017-06-15 16:15:50,379: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:15:51,538: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:15:51,539: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:15:51,561: INFO/command_plan] add_jobs: created Job 200 (Test for host connectivity)
[2017-06-15 16:15:51,597: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:15:51,597: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:15:51,615: INFO/command_plan] add_jobs: created Job 201 (Test for host connectivity)
[2017-06-15 16:15:51,630: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-15 16:15:51,654: INFO/job_scheduler] Job 200: RunJobThread.run
[2017-06-15 16:15:51,656: INFO/job_scheduler] Job 201: RunJobThread.run
[2017-06-15 16:15:51,670: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:15:51,807: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:15:51,809: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:15:51,845: INFO/command_plan] add_jobs: created Job 202 (Test for host connectivity)
[2017-06-15 16:15:51,891: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-15 16:15:51,902: INFO/job_scheduler] Job 202: RunJobThread.run
[2017-06-15 16:15:51,931: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:15:51,932: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:15:52,037: INFO/command_plan] add_jobs: created Job 203 (Test for host connectivity)
[2017-06-15 16:15:52,081: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-15 16:15:52,101: INFO/job_scheduler] Job 203: RunJobThread.run
[2017-06-15 16:15:57,915: INFO/job_scheduler] Job 200 finished 1 steps successfully
[2017-06-15 16:15:57,952: INFO/job_scheduler] Job 200: complete_job: Updating cache
[2017-06-15 16:15:57,952: INFO/job_scheduler] Job 200 completing (errored=False, cancelled=False)
[2017-06-15 16:15:57,991: INFO/job_scheduler] Job 201 finished 1 steps successfully
[2017-06-15 16:15:58,002: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:15:58,035: INFO/job_scheduler] Job 201: complete_job: Updating cache
[2017-06-15 16:15:58,035: INFO/job_scheduler] Job 201 completing (errored=False, cancelled=False)
[2017-06-15 16:15:58,129: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:15:58,178: INFO/job_scheduler] Job 202 finished 1 steps successfully
[2017-06-15 16:15:58,218: INFO/job_scheduler] Job 202: complete_job: Updating cache
[2017-06-15 16:15:58,218: INFO/job_scheduler] Job 202 completing (errored=False, cancelled=False)
[2017-06-15 16:15:58,261: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 16:15:58,458: INFO/job_scheduler] Job 203 finished 1 steps successfully
[2017-06-15 16:15:58,493: INFO/job_scheduler] Job 203: complete_job: Updating cache
[2017-06-15 16:15:58,494: INFO/job_scheduler] Job 203 completing (errored=False, cancelled=False)
[2017-06-15 16:15:58,527: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:15:59,690: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:15:59,691: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:15:59,710: INFO/command_plan] add_jobs: created Job 204 (Test for host connectivity)
[2017-06-15 16:15:59,754: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:15:59,755: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:15:59,843: INFO/command_plan] add_jobs: created Job 205 (Test for host connectivity)
[2017-06-15 16:15:59,852: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-15 16:15:59,859: INFO/job_scheduler] Job 204: RunJobThread.run
[2017-06-15 16:15:59,864: INFO/job_scheduler] Job 205: RunJobThread.run
[2017-06-15 16:15:59,878: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:15:59,917: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:15:59,917: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:15:59,930: INFO/command_plan] add_jobs: created Job 206 (Test for host connectivity)
[2017-06-15 16:15:59,974: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-15 16:16:00,007: INFO/job_scheduler] Job 206: RunJobThread.run
[2017-06-15 16:16:00,083: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:16:00,084: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:16:00,122: INFO/command_plan] add_jobs: created Job 207 (Test for host connectivity)
[2017-06-15 16:16:00,155: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-15 16:16:00,163: INFO/job_scheduler] Job 207: RunJobThread.run
[2017-06-15 16:16:06,164: INFO/job_scheduler] Job 205 finished 1 steps successfully
[2017-06-15 16:16:06,225: INFO/job_scheduler] Job 205: complete_job: Updating cache
[2017-06-15 16:16:06,225: INFO/job_scheduler] Job 205 completing (errored=False, cancelled=False)
[2017-06-15 16:16:06,283: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:16:06,287: INFO/job_scheduler] Job 204 finished 1 steps successfully
[2017-06-15 16:16:06,325: INFO/job_scheduler] Job 204: complete_job: Updating cache
[2017-06-15 16:16:06,326: INFO/job_scheduler] Job 204 completing (errored=False, cancelled=False)
[2017-06-15 16:16:06,340: INFO/job_scheduler] Job 206 finished 1 steps successfully
[2017-06-15 16:16:06,397: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:16:06,409: INFO/job_scheduler] Job 206: complete_job: Updating cache
[2017-06-15 16:16:06,409: INFO/job_scheduler] Job 206 completing (errored=False, cancelled=False)
[2017-06-15 16:16:06,437: INFO/job_scheduler] Job 207 finished 1 steps successfully
[2017-06-15 16:16:06,454: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 16:16:06,469: INFO/job_scheduler] Job 207: complete_job: Updating cache
[2017-06-15 16:16:06,469: INFO/job_scheduler] Job 207 completing (errored=False, cancelled=False)
[2017-06-15 16:16:06,535: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:16:07,749: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:16:07,752: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:16:07,775: INFO/command_plan] add_jobs: created Job 208 (Test for host connectivity)
[2017-06-15 16:16:07,805: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-15 16:16:07,808: INFO/job_scheduler] Job 208: RunJobThread.run
[2017-06-15 16:16:07,872: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:16:07,873: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:16:07,887: INFO/command_plan] add_jobs: created Job 209 (Test for host connectivity)
[2017-06-15 16:16:07,959: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:16:07,959: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:16:07,992: INFO/command_plan] add_jobs: created Job 210 (Test for host connectivity)
[2017-06-15 16:16:08,009: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-15 16:16:08,042: INFO/job_scheduler] Job 209: RunJobThread.run
[2017-06-15 16:16:08,057: INFO/job_scheduler] Job 210: RunJobThread.run
[2017-06-15 16:16:08,094: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:16:08,095: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:16:08,110: INFO/command_plan] add_jobs: created Job 211 (Test for host connectivity)
[2017-06-15 16:16:08,117: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-15 16:16:08,141: INFO/job_scheduler] Job 211: RunJobThread.run
[2017-06-15 16:16:08,168: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-15 16:16:14,145: INFO/job_scheduler] Job 208 finished 1 steps successfully
[2017-06-15 16:16:14,182: INFO/job_scheduler] Job 208: complete_job: Updating cache
[2017-06-15 16:16:14,182: INFO/job_scheduler] Job 208 completing (errored=False, cancelled=False)
[2017-06-15 16:16:14,224: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:16:14,260: INFO/job_scheduler] Job 209 finished 1 steps successfully
[2017-06-15 16:16:14,298: INFO/job_scheduler] Job 209: complete_job: Updating cache
[2017-06-15 16:16:14,299: INFO/job_scheduler] Job 209 completing (errored=False, cancelled=False)
[2017-06-15 16:16:14,309: INFO/job_scheduler] Job 211 finished 1 steps successfully
[2017-06-15 16:16:14,352: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:16:14,365: INFO/job_scheduler] Job 211: complete_job: Updating cache
[2017-06-15 16:16:14,365: INFO/job_scheduler] Job 211 completing (errored=False, cancelled=False)
[2017-06-15 16:16:14,422: INFO/job_scheduler] Job 210 finished 1 steps successfully
[2017-06-15 16:16:14,434: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 16:16:14,450: INFO/job_scheduler] Job 210: complete_job: Updating cache
[2017-06-15 16:16:14,451: INFO/job_scheduler] Job 210 completing (errored=False, cancelled=False)
[2017-06-15 16:16:14,492: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:16:15,706: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:16:15,706: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:16:15,723: INFO/command_plan] add_jobs: created Job 212 (Test for host connectivity)
[2017-06-15 16:16:15,759: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-15 16:16:15,761: INFO/job_scheduler] Job 212: RunJobThread.run
[2017-06-15 16:16:15,823: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:16:15,824: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:16:15,839: INFO/command_plan] add_jobs: created Job 213 (Test for host connectivity)
[2017-06-15 16:16:15,918: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:16:15,919: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:16:15,956: INFO/command_plan] add_jobs: created Job 214 (Test for host connectivity)
[2017-06-15 16:16:15,965: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-15 16:16:15,969: INFO/job_scheduler] Job 213: RunJobThread.run
[2017-06-15 16:16:15,970: INFO/job_scheduler] Job 214: RunJobThread.run
[2017-06-15 16:16:15,998: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:16:16,082: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:16:16,083: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:16:16,119: INFO/command_plan] add_jobs: created Job 215 (Test for host connectivity)
[2017-06-15 16:16:16,165: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-15 16:16:16,187: INFO/job_scheduler] Job 215: RunJobThread.run
[2017-06-15 16:16:22,075: INFO/job_scheduler] Job 212 finished 1 steps successfully
[2017-06-15 16:16:22,130: INFO/job_scheduler] Job 212: complete_job: Updating cache
[2017-06-15 16:16:22,130: INFO/job_scheduler] Job 212 completing (errored=False, cancelled=False)
[2017-06-15 16:16:22,168: INFO/job_scheduler] Job 213 finished 1 steps successfully
[2017-06-15 16:16:22,188: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:16:22,223: INFO/job_scheduler] Job 213: complete_job: Updating cache
[2017-06-15 16:16:22,223: INFO/job_scheduler] Job 213 completing (errored=False, cancelled=False)
[2017-06-15 16:16:22,324: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:16:22,395: INFO/job_scheduler] Job 214 finished 1 steps successfully
[2017-06-15 16:16:22,414: INFO/job_scheduler] Job 214: complete_job: Updating cache
[2017-06-15 16:16:22,414: INFO/job_scheduler] Job 214 completing (errored=False, cancelled=False)
[2017-06-15 16:16:22,468: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 16:16:22,610: INFO/job_scheduler] Job 215 finished 1 steps successfully
[2017-06-15 16:16:22,655: INFO/job_scheduler] Job 215: complete_job: Updating cache
[2017-06-15 16:16:22,655: INFO/job_scheduler] Job 215 completing (errored=False, cancelled=False)
[2017-06-15 16:16:22,689: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:16:23,874: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:16:23,876: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:16:23,930: INFO/command_plan] add_jobs: created Job 216 (Test for host connectivity)
[2017-06-15 16:16:23,955: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-15 16:16:23,960: INFO/job_scheduler] Job 216: RunJobThread.run
[2017-06-15 16:16:24,016: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:16:24,016: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:16:24,030: INFO/command_plan] add_jobs: created Job 217 (Test for host connectivity)
[2017-06-15 16:16:24,096: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-15 16:16:24,103: INFO/job_scheduler] Job 217: RunJobThread.run
[2017-06-15 16:16:24,162: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:16:24,163: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:16:24,180: INFO/command_plan] add_jobs: created Job 218 (Test for host connectivity)
[2017-06-15 16:16:24,240: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-15 16:16:24,256: INFO/job_scheduler] Job 218: RunJobThread.run
[2017-06-15 16:16:24,314: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:16:24,316: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:16:24,338: INFO/command_plan] add_jobs: created Job 219 (Test for host connectivity)
[2017-06-15 16:16:24,372: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-15 16:16:24,380: INFO/job_scheduler] Job 219: RunJobThread.run
[2017-06-15 16:16:30,260: INFO/job_scheduler] Job 216 finished 1 steps successfully
[2017-06-15 16:16:30,295: INFO/job_scheduler] Job 216: complete_job: Updating cache
[2017-06-15 16:16:30,295: INFO/job_scheduler] Job 216 completing (errored=False, cancelled=False)
[2017-06-15 16:16:30,347: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:16:30,389: INFO/job_scheduler] Job 217 finished 1 steps successfully
[2017-06-15 16:16:30,471: INFO/job_scheduler] Job 217: complete_job: Updating cache
[2017-06-15 16:16:30,471: INFO/job_scheduler] Job 217 completing (errored=False, cancelled=False)
[2017-06-15 16:16:30,507: INFO/job_scheduler] Job 219 finished 1 steps successfully
[2017-06-15 16:16:30,537: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:16:30,549: INFO/job_scheduler] Job 219: complete_job: Updating cache
[2017-06-15 16:16:30,549: INFO/job_scheduler] Job 219 completing (errored=False, cancelled=False)
[2017-06-15 16:16:30,574: INFO/job_scheduler] Job 218 finished 1 steps successfully
[2017-06-15 16:16:30,629: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 16:16:30,637: INFO/job_scheduler] Job 218: complete_job: Updating cache
[2017-06-15 16:16:30,638: INFO/job_scheduler] Job 218 completing (errored=False, cancelled=False)
[2017-06-15 16:16:30,680: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:16:31,861: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:16:31,862: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:16:31,878: INFO/command_plan] add_jobs: created Job 220 (Test for host connectivity)
[2017-06-15 16:16:31,900: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-15 16:16:31,902: INFO/job_scheduler] Job 220: RunJobThread.run
[2017-06-15 16:16:31,941: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:16:31,942: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:16:31,970: INFO/command_plan] add_jobs: created Job 221 (Test for host connectivity)
[2017-06-15 16:16:32,025: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:16:32,061: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:16:32,079: INFO/command_plan] add_jobs: created Job 222 (Test for host connectivity)
[2017-06-15 16:16:32,095: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 1 tasked)
[2017-06-15 16:16:32,108: INFO/job_scheduler] Job 221: RunJobThread.run
[2017-06-15 16:16:32,108: INFO/job_scheduler] Job 222: RunJobThread.run
[2017-06-15 16:16:32,125: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:16:32,177: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:16:32,177: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:16:32,203: INFO/command_plan] add_jobs: created Job 223 (Test for host connectivity)
[2017-06-15 16:16:32,226: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-15 16:16:32,247: INFO/job_scheduler] Job 223: RunJobThread.run
[2017-06-15 16:16:38,126: INFO/job_scheduler] Job 220 finished 1 steps successfully
[2017-06-15 16:16:38,159: INFO/job_scheduler] Job 220: complete_job: Updating cache
[2017-06-15 16:16:38,159: INFO/job_scheduler] Job 220 completing (errored=False, cancelled=False)
[2017-06-15 16:16:38,203: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:16:38,319: INFO/job_scheduler] Job 222 finished 1 steps successfully
[2017-06-15 16:16:38,338: INFO/job_scheduler] Job 221 finished 1 steps successfully
[2017-06-15 16:16:38,359: INFO/job_scheduler] Job 222: complete_job: Updating cache
[2017-06-15 16:16:38,359: INFO/job_scheduler] Job 222 completing (errored=False, cancelled=False)
[2017-06-15 16:16:38,410: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:16:38,426: INFO/job_scheduler] Job 223 finished 1 steps successfully
[2017-06-15 16:16:38,430: INFO/job_scheduler] Job 221: complete_job: Updating cache
[2017-06-15 16:16:38,430: INFO/job_scheduler] Job 221 completing (errored=False, cancelled=False)
[2017-06-15 16:16:38,501: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 16:16:38,517: INFO/job_scheduler] Job 223: complete_job: Updating cache
[2017-06-15 16:16:38,517: INFO/job_scheduler] Job 223 completing (errored=False, cancelled=False)
[2017-06-15 16:16:38,568: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:16:39,748: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:16:39,749: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:16:39,767: INFO/command_plan] add_jobs: created Job 224 (Test for host connectivity)
[2017-06-15 16:16:39,785: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-15 16:16:39,786: INFO/job_scheduler] Job 224: RunJobThread.run
[2017-06-15 16:16:39,813: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:16:39,813: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:16:39,841: INFO/command_plan] add_jobs: created Job 225 (Test for host connectivity)
[2017-06-15 16:16:39,873: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-15 16:16:39,902: INFO/job_scheduler] Job 225: RunJobThread.run
[2017-06-15 16:16:39,972: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:16:39,974: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:16:40,018: INFO/command_plan] add_jobs: created Job 226 (Test for host connectivity)
[2017-06-15 16:16:40,067: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-15 16:16:40,069: INFO/job_scheduler] Job 226: RunJobThread.run
[2017-06-15 16:16:40,101: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:16:40,101: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:16:40,126: INFO/command_plan] add_jobs: created Job 227 (Test for host connectivity)
[2017-06-15 16:16:40,159: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-15 16:16:40,182: INFO/job_scheduler] Job 227: RunJobThread.run
[2017-06-15 16:16:46,030: INFO/job_scheduler] Job 225 finished 1 steps successfully
[2017-06-15 16:16:46,032: INFO/job_scheduler] Job 224 finished 1 steps successfully
[2017-06-15 16:16:46,091: INFO/job_scheduler] Job 225: complete_job: Updating cache
[2017-06-15 16:16:46,091: INFO/job_scheduler] Job 225 completing (errored=False, cancelled=False)
[2017-06-15 16:16:46,134: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:16:46,257: INFO/job_scheduler] Job 226 finished 1 steps successfully
[2017-06-15 16:16:46,305: INFO/job_scheduler] Job 227 finished 1 steps successfully
[2017-06-15 16:16:46,546: INFO/job_scheduler] Getting FQDN for 'lotus-32vm5.lotus.hpdd.lab.intel.com': lotus-32vm5
lotus-32vm5.lotus.hpdd.lab.intel.com

[2017-06-15 16:16:46,581: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:16:46,586: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-5 transitioning from undeployed to unconfigured
[2017-06-15 16:16:46,624: INFO/command_plan] Created command 137 (Deploying agent to host lotus-32vm5.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-06-15 16:16:46,633: INFO/job_scheduler] Job 224: complete_job: Updating cache
[2017-06-15 16:16:46,634: INFO/job_scheduler] Job 224 completing (errored=False, cancelled=False)
[2017-06-15 16:16:46,714: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-15 16:16:46,779: INFO/job_scheduler] Job 228: RunJobThread.run
[2017-06-15 16:16:46,817: INFO/job_scheduler] Job 226: complete_job: Updating cache
[2017-06-15 16:16:46,817: INFO/job_scheduler] Job 226 completing (errored=False, cancelled=False)
[2017-06-15 16:16:46,891: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:16:47,001: INFO/job_scheduler] Job 227: complete_job: Updating cache
[2017-06-15 16:16:47,001: INFO/job_scheduler] Job 227 completing (errored=False, cancelled=False)
[2017-06-15 16:16:47,084: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 16:16:47,085: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 16:16:47,610: INFO/job_scheduler] Getting FQDN for 'lotus-32vm6.lotus.hpdd.lab.intel.com': lotus-32vm6
lotus-32vm6.lotus.hpdd.lab.intel.com

[2017-06-15 16:16:47,640: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:16:47,643: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-6 transitioning from undeployed to unconfigured
[2017-06-15 16:16:47,677: INFO/command_plan] Created command 138 (Deploying agent to host lotus-32vm6.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-06-15 16:16:47,755: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-15 16:16:47,795: INFO/job_scheduler] Job 229: RunJobThread.run
[2017-06-15 16:16:47,815: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:16:48,306: INFO/job_scheduler] Getting FQDN for 'lotus-32vm7.lotus.hpdd.lab.intel.com': lotus-32vm7
lotus-32vm7.lotus.hpdd.lab.intel.com

[2017-06-15 16:16:48,348: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:16:48,354: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-7 transitioning from undeployed to unconfigured
[2017-06-15 16:16:48,400: INFO/command_plan] Created command 139 (Deploying agent to host lotus-32vm7.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-06-15 16:16:48,499: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-15 16:16:48,525: INFO/job_scheduler] Job 230: RunJobThread.run
[2017-06-15 16:16:48,543: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:16:49,075: INFO/job_scheduler] Getting FQDN for 'lotus-32vm8.lotus.hpdd.lab.intel.com': lotus-32vm8
lotus-32vm8.lotus.hpdd.lab.intel.com

[2017-06-15 16:16:49,106: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:16:49,112: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-8 transitioning from undeployed to unconfigured
[2017-06-15 16:16:49,139: INFO/command_plan] Created command 140 (Deploying agent to host lotus-32vm8.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-06-15 16:16:49,233: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-15 16:16:49,270: INFO/job_scheduler] Job 231: RunJobThread.run
[2017-06-15 16:16:49,287: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-15 16:16:49,926: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-15 16:16:56,864: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 16:16:56,865: INFO/job_scheduler]   Job 228 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-06-15 16:16:56,865: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-15 16:16:56,878: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 16:16:56,878: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/de200386-f10b-4b31-99a8-16621077f017
[2017-06-15 16:16:56,884: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/77c9887b-c4d2-4c59-a8c1-87709202cdf2
[2017-06-15 16:16:56,884: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/77c9887b-c4d2-4c59-a8c1-87709202cdf2
[2017-06-15 16:16:56,986: INFO/job_scheduler] Job 228 finished 1 steps successfully
[2017-06-15 16:16:57,030: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 140623739435472
[2017-06-15 16:16:57,036: INFO/job] Job 228: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 16:16:57,036: INFO/job] Job 228: not_deleted=True
[2017-06-15 16:16:57,036: INFO/job_scheduler] Job 228: complete_job: Updating cache
[2017-06-15 16:16:57,036: INFO/job_scheduler] Job 228: locked_item 140623739435472 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-15 16:16:57,039: INFO/job_scheduler] Job 228 completing (errored=False, cancelled=False)
[2017-06-15 16:16:57,106: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-5
[2017-06-15 16:16:57,107: INFO/job_scheduler] _notify: Updating .boot_time of item 5 (lotus-32vm5.lotus.hpdd.lab.intel.com) from None to 2017-06-14 09:00:19+00:00
[2017-06-15 16:16:57,111: INFO/job_scheduler] _notify: Set boot_time=2017-06-14 09:00:19+00:00 on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-5) and saved
[2017-06-15 16:16:57,121: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:16:57,373: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 16:16:57,373: INFO/job_scheduler]   Job 229 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-06-15 16:16:57,373: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-15 16:16:57,428: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 16:16:57,428: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/074d2a78-c78f-42b3-b75e-e25178c02bb5
[2017-06-15 16:16:57,430: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/0669153e-f215-42ea-ae03-8b02be875c8a
[2017-06-15 16:16:57,430: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/0669153e-f215-42ea-ae03-8b02be875c8a
[2017-06-15 16:16:57,443: INFO/job_scheduler] Job 229 finished 1 steps successfully
[2017-06-15 16:16:57,536: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 140623740091472
[2017-06-15 16:16:57,544: INFO/job] Job 229: StateChangeJob complete, setting state unconfigured on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 16:16:57,546: INFO/job] Job 229: not_deleted=True
[2017-06-15 16:16:57,546: INFO/job_scheduler] Job 229: complete_job: Updating cache
[2017-06-15 16:16:57,546: INFO/job_scheduler] Job 229: locked_item 140623740091472 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-15 16:16:57,548: INFO/job_scheduler] Job 229 completing (errored=False, cancelled=False)
[2017-06-15 16:16:57,608: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-6
[2017-06-15 16:16:57,608: INFO/job_scheduler] _notify: Updating .boot_time of item 6 (lotus-32vm6.lotus.hpdd.lab.intel.com) from None to 2017-04-25 07:09:19+00:00
[2017-06-15 16:16:57,613: INFO/job_scheduler] _notify: Set boot_time=2017-04-25 07:09:19+00:00 on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-6) and saved
[2017-06-15 16:16:57,620: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:16:58,225: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 16:16:58,226: INFO/job_scheduler]   Job 230 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-06-15 16:16:58,226: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-15 16:16:58,291: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 16:16:58,291: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/932fbbd0-2b28-49a0-b042-7cf0bec7f0cf
[2017-06-15 16:16:58,295: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/4054b950-dc98-4664-b3e6-499cf8c2223e
[2017-06-15 16:16:58,295: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/4054b950-dc98-4664-b3e6-499cf8c2223e
[2017-06-15 16:16:58,335: INFO/job_scheduler] Job 230 finished 1 steps successfully
[2017-06-15 16:16:58,450: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 140624142265296
[2017-06-15 16:16:58,464: INFO/job] Job 230: StateChangeJob complete, setting state unconfigured on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 16:16:58,464: INFO/job] Job 230: not_deleted=True
[2017-06-15 16:16:58,464: INFO/job_scheduler] Job 230: complete_job: Updating cache
[2017-06-15 16:16:58,464: INFO/job_scheduler] Job 230: locked_item 140624142265296 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-15 16:16:58,484: INFO/job_scheduler] Job 230 completing (errored=False, cancelled=False)
[2017-06-15 16:16:58,764: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-7
[2017-06-15 16:16:58,765: INFO/job_scheduler] _notify: Updating .boot_time of item 7 (lotus-32vm7.lotus.hpdd.lab.intel.com) from None to 2017-04-25 07:09:31+00:00
[2017-06-15 16:16:58,774: INFO/job_scheduler] _notify: Set boot_time=2017-04-25 07:09:31+00:00 on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-7) and saved
[2017-06-15 16:16:58,787: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 16:16:59,667: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 16:16:59,667: INFO/job_scheduler]   Job 231 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-06-15 16:16:59,667: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 16:16:59,679: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 16:16:59,679: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/201aff73-3c67-4458-af69-65f9885fa919
[2017-06-15 16:16:59,684: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1f9f06d9-de11-4011-9135-5f0a7709abb7
[2017-06-15 16:16:59,684: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1f9f06d9-de11-4011-9135-5f0a7709abb7
[2017-06-15 16:16:59,802: INFO/job_scheduler] Job 231 finished 1 steps successfully
[2017-06-15 16:16:59,905: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 140623739945552
[2017-06-15 16:16:59,911: INFO/job] Job 231: StateChangeJob complete, setting state unconfigured on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 16:16:59,911: INFO/job] Job 231: not_deleted=True
[2017-06-15 16:16:59,912: INFO/job_scheduler] Job 231: complete_job: Updating cache
[2017-06-15 16:16:59,912: INFO/job_scheduler] Job 231: locked_item 140623739945552 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-15 16:16:59,914: INFO/job_scheduler] Job 231 completing (errored=False, cancelled=False)
[2017-06-15 16:17:00,012: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-8
[2017-06-15 16:17:00,013: INFO/job_scheduler] _notify: Updating .boot_time of item 8 (lotus-32vm8.lotus.hpdd.lab.intel.com) from None to 2017-06-14 09:00:40+00:00
[2017-06-15 16:17:00,022: INFO/job_scheduler] _notify: Set boot_time=2017-06-14 09:00:40+00:00 on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-8) and saved
[2017-06-15 16:17:00,029: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:17:00,715: INFO/job_scheduler] _notify: Updating .properties of item 5 (lotus-32vm5.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": false, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-06-15 16:17:00,722: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": false, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-5) and saved
[2017-06-15 16:17:00,728: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:17:00,795: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:17:00,998: INFO/job_scheduler] _notify: Updating .properties of item 6 (lotus-32vm6.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": false, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-06-15 16:17:01,006: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": false, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-6) and saved
[2017-06-15 16:17:01,011: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:17:01,193: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:17:01,514: INFO/job_scheduler] _notify: Updating .properties of item 7 (lotus-32vm7.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": false, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-06-15 16:17:01,521: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": false, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-7) and saved
[2017-06-15 16:17:01,535: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:17:01,705: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:17:02,849: INFO/job_scheduler] _notify: Updating .properties of item 8 (lotus-32vm8.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": false, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-06-15 16:17:02,855: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": false, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-8) and saved
[2017-06-15 16:17:02,860: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:17:02,918: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 16:17:09,504: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:17:09,518: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-5 transitioning from unconfigured to managed
[2017-06-15 16:17:09,738: INFO/command_plan] Created command 141 (Setup managed host lotus-32vm5.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-06-15 16:17:09,907: INFO/job_scheduler] run_next: 1 runnable jobs of (11 pending, 0 tasked)
[2017-06-15 16:17:09,970: INFO/job_scheduler] Job 232: RunJobThread.run
[2017-06-15 16:17:09,971: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-06-15 16:17:10,005: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5413453b-4d7e-4726-bdb7-b8ccd531b7ab
[2017-06-15 16:17:10,005: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5413453b-4d7e-4726-bdb7-b8ccd531b7ab
[2017-06-15 16:17:10,079: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:17:10,096: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-6 transitioning from unconfigured to managed
[2017-06-15 16:17:10,298: INFO/command_plan] Created command 142 (Setup managed host lotus-32vm6.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-06-15 16:17:10,358: INFO/job_scheduler] run_next: 1 runnable jobs of (21 pending, 1 tasked)
[2017-06-15 16:17:10,391: INFO/job_scheduler] Job 243: RunJobThread.run
[2017-06-15 16:17:10,391: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-06-15 16:17:10,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 65bd9cbb-b0ee-4800-b92d-10f6c14953b3
[2017-06-15 16:17:10,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 65bd9cbb-b0ee-4800-b92d-10f6c14953b3
[2017-06-15 16:17:10,622: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:17:10,625: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-7 transitioning from unconfigured to managed
[2017-06-15 16:17:10,757: INFO/command_plan] Created command 143 (Setup managed host lotus-32vm7.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-06-15 16:17:10,841: INFO/job_scheduler] run_next: 1 runnable jobs of (31 pending, 2 tasked)
[2017-06-15 16:17:10,889: INFO/job_scheduler] Job 254: RunJobThread.run
[2017-06-15 16:17:10,890: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-06-15 16:17:10,931: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9b5c7b83-bfdd-43de-9e28-46c92d85f975
[2017-06-15 16:17:10,931: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9b5c7b83-bfdd-43de-9e28-46c92d85f975
[2017-06-15 16:17:11,141: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:17:11,143: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-8 transitioning from unconfigured to managed
[2017-06-15 16:17:11,286: INFO/command_plan] Created command 144 (Setup managed host lotus-32vm8.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-06-15 16:17:11,412: INFO/job_scheduler] run_next: 1 runnable jobs of (41 pending, 3 tasked)
[2017-06-15 16:17:11,458: INFO/job_scheduler] Job 265: RunJobThread.run
[2017-06-15 16:17:11,459: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-06-15 16:17:11,507: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6666ff4a-547e-44d5-ae37-5e5e74c80d6d
[2017-06-15 16:17:11,507: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6666ff4a-547e-44d5-ae37-5e5e74c80d6d
[2017-06-15 16:18:41,803: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1f9f06d9-de11-4011-9135-5f0a7709abb7
[2017-06-15 16:18:41,803: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1f9f06d9-de11-4011-9135-5f0a7709abb7
[2017-06-15 16:18:41,804: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1f9f06d9-de11-4011-9135-5f0a7709abb7
[2017-06-15 16:18:41,804: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1f9f06d9-de11-4011-9135-5f0a7709abb7
[2017-06-15 16:18:41,804: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6666ff4a-547e-44d5-ae37-5e5e74c80d6d
[2017-06-15 16:18:41,804: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6666ff4a-547e-44d5-ae37-5e5e74c80d6d
[2017-06-15 16:18:41,831: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6666ff4a-547e-44d5-ae37-5e5e74c80d6d
[2017-06-15 16:18:41,831: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6666ff4a-547e-44d5-ae37-5e5e74c80d6d
[2017-06-15 16:18:41,831: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-06-15 16:18:41.831486+00:00:
[2017-06-15 16:18:41,831: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-06-15 16:18:41.831486+00:00:
[2017-06-15 16:18:41,831: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-06-15 16:18:41,831: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-06-15 16:18:41,854: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 16:18:41,854: INFO/job_scheduler]   Job 265 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-15 16:18:41,854: INFO/job_scheduler]   Job 266 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 16:18:41,854: INFO/job_scheduler]   Job 267 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 16:18:41,855: INFO/job_scheduler]   Job 268 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 16:18:41,855: INFO/job_scheduler]   Job 269 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 16:18:41,855: INFO/job_scheduler]   Job 272 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 16:18:41,855: INFO/job_scheduler]   Job 275 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 16:18:41,855: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-15 16:18:41,857: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-06-15 16:18:41.857145+00:00:
[2017-06-15 16:18:41,857: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-06-15 16:18:41.857145+00:00:
[2017-06-15 16:18:41,857: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-06-15 16:18:41,857: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-06-15 16:18:41,869: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 16:18:41,869: INFO/job_scheduler]   Job 265 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-15 16:18:41,869: INFO/job_scheduler]   Job 266 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 16:18:41,869: INFO/job_scheduler]   Job 267 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 16:18:41,869: INFO/job_scheduler]   Job 268 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 16:18:41,869: INFO/job_scheduler]   Job 269 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 16:18:41,870: INFO/job_scheduler]   Job 272 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 16:18:41,870: INFO/job_scheduler]   Job 275 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 16:18:41,870: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-15 16:18:41,871: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com restart_agent {}
[2017-06-15 16:18:41,883: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 02da0ca3-886f-49ae-9314-bec2b13be0f6
[2017-06-15 16:18:41,883: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 02da0ca3-886f-49ae-9314-bec2b13be0f6
[2017-06-15 16:18:41,977: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1f9f06d9-de11-4011-9135-5f0a7709abb7
[2017-06-15 16:18:41,977: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1f9f06d9-de11-4011-9135-5f0a7709abb7
[2017-06-15 16:18:41,977: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1f9f06d9-de11-4011-9135-5f0a7709abb7
[2017-06-15 16:18:41,977: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1f9f06d9-de11-4011-9135-5f0a7709abb7
[2017-06-15 16:18:41,977: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 02da0ca3-886f-49ae-9314-bec2b13be0f6
[2017-06-15 16:18:41,977: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 02da0ca3-886f-49ae-9314-bec2b13be0f6
[2017-06-15 16:18:42,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 02da0ca3-886f-49ae-9314-bec2b13be0f6
[2017-06-15 16:18:42,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 02da0ca3-886f-49ae-9314-bec2b13be0f6
[2017-06-15 16:18:42,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm8.lotus.hpdd.lab.intel.com (old 1f9f06d9-de11-4011-9135-5f0a7709abb7)
[2017-06-15 16:18:42,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm8.lotus.hpdd.lab.intel.com (old 1f9f06d9-de11-4011-9135-5f0a7709abb7)
[2017-06-15 16:18:44,569: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/0669153e-f215-42ea-ae03-8b02be875c8a
[2017-06-15 16:18:44,569: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/0669153e-f215-42ea-ae03-8b02be875c8a
[2017-06-15 16:18:44,569: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/0669153e-f215-42ea-ae03-8b02be875c8a
[2017-06-15 16:18:44,569: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/0669153e-f215-42ea-ae03-8b02be875c8a
[2017-06-15 16:18:44,569: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 65bd9cbb-b0ee-4800-b92d-10f6c14953b3
[2017-06-15 16:18:44,569: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 65bd9cbb-b0ee-4800-b92d-10f6c14953b3
[2017-06-15 16:18:44,608: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 65bd9cbb-b0ee-4800-b92d-10f6c14953b3
[2017-06-15 16:18:44,608: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 65bd9cbb-b0ee-4800-b92d-10f6c14953b3
[2017-06-15 16:18:44,608: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-06-15 16:18:44.608594+00:00:
[2017-06-15 16:18:44,608: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-06-15 16:18:44.608594+00:00:
[2017-06-15 16:18:44,608: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-06-15 16:18:44,608: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-06-15 16:18:44,628: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 16:18:44,628: INFO/job_scheduler]   Job 243 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-15 16:18:44,628: INFO/job_scheduler]   Job 244 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 16:18:44,628: INFO/job_scheduler]   Job 245 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 16:18:44,628: INFO/job_scheduler]   Job 246 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 16:18:44,628: INFO/job_scheduler]   Job 247 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 16:18:44,629: INFO/job_scheduler]   Job 251 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 16:18:44,629: INFO/job_scheduler]   Job 253 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 16:18:44,629: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-15 16:18:44,631: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-06-15 16:18:44.631455+00:00:
[2017-06-15 16:18:44,631: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-06-15 16:18:44.631455+00:00:
[2017-06-15 16:18:44,631: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-06-15 16:18:44,631: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-06-15 16:18:44,650: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 16:18:44,651: INFO/job_scheduler]   Job 243 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-15 16:18:44,651: INFO/job_scheduler]   Job 244 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 16:18:44,651: INFO/job_scheduler]   Job 245 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 16:18:44,651: INFO/job_scheduler]   Job 246 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 16:18:44,651: INFO/job_scheduler]   Job 247 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 16:18:44,651: INFO/job_scheduler]   Job 251 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 16:18:44,651: INFO/job_scheduler]   Job 253 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 16:18:44,652: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-15 16:18:44,654: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com restart_agent {}
[2017-06-15 16:18:44,673: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 89d5ef58-454e-4bc4-997c-108cb605687b
[2017-06-15 16:18:44,673: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 89d5ef58-454e-4bc4-997c-108cb605687b
[2017-06-15 16:18:44,740: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/0669153e-f215-42ea-ae03-8b02be875c8a
[2017-06-15 16:18:44,740: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/0669153e-f215-42ea-ae03-8b02be875c8a
[2017-06-15 16:18:44,740: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/0669153e-f215-42ea-ae03-8b02be875c8a
[2017-06-15 16:18:44,740: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/0669153e-f215-42ea-ae03-8b02be875c8a
[2017-06-15 16:18:44,740: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 89d5ef58-454e-4bc4-997c-108cb605687b
[2017-06-15 16:18:44,740: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 89d5ef58-454e-4bc4-997c-108cb605687b
[2017-06-15 16:18:44,746: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 89d5ef58-454e-4bc4-997c-108cb605687b
[2017-06-15 16:18:44,746: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 89d5ef58-454e-4bc4-997c-108cb605687b
[2017-06-15 16:18:44,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm6.lotus.hpdd.lab.intel.com (old 0669153e-f215-42ea-ae03-8b02be875c8a)
[2017-06-15 16:18:44,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm6.lotus.hpdd.lab.intel.com (old 0669153e-f215-42ea-ae03-8b02be875c8a)
[2017-06-15 16:18:59,843: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1f9f06d9-de11-4011-9135-5f0a7709abb7
[2017-06-15 16:18:59,843: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1f9f06d9-de11-4011-9135-5f0a7709abb7
[2017-06-15 16:18:59,844: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 1f9f06d9-de11-4011-9135-5f0a7709abb7 because session terminated
[2017-06-15 16:18:59,844: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 1f9f06d9-de11-4011-9135-5f0a7709abb7 because session terminated
[2017-06-15 16:18:59,853: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:18:59,853: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:19:00,025: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm8.lotus.hpdd.lab.intel.com new 49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:19:00,025: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm8.lotus.hpdd.lab.intel.com new 49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:19:00,026: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-15 16:19:00,061: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1deb8f88-5b87-4ce8-b8d4-2778709f5964
[2017-06-15 16:19:00,061: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1deb8f88-5b87-4ce8-b8d4-2778709f5964
[2017-06-15 16:19:01,426: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/0669153e-f215-42ea-ae03-8b02be875c8a
[2017-06-15 16:19:01,426: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/0669153e-f215-42ea-ae03-8b02be875c8a
[2017-06-15 16:19:01,426: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 0669153e-f215-42ea-ae03-8b02be875c8a because session terminated
[2017-06-15 16:19:01,426: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 0669153e-f215-42ea-ae03-8b02be875c8a because session terminated
[2017-06-15 16:19:01,440: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 16:19:01,440: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 16:19:01,771: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm6.lotus.hpdd.lab.intel.com new d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 16:19:01,771: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm6.lotus.hpdd.lab.intel.com new d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 16:19:01,772: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-15 16:19:01,792: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 467f0e73-366b-429e-a4a6-5715f3e829d5
[2017-06-15 16:19:01,792: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 467f0e73-366b-429e-a4a6-5715f3e829d5
[2017-06-15 16:19:01,896: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 16:19:01,896: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 16:19:01,896: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 16:19:01,896: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 16:19:01,896: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 467f0e73-366b-429e-a4a6-5715f3e829d5
[2017-06-15 16:19:01,896: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 467f0e73-366b-429e-a4a6-5715f3e829d5
[2017-06-15 16:19:01,912: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 467f0e73-366b-429e-a4a6-5715f3e829d5
[2017-06-15 16:19:01,912: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 467f0e73-366b-429e-a4a6-5715f3e829d5
[2017-06-15 16:19:01,913: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-06-15 16:19:01,925: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 741dfe8f-5137-420a-b468-d15b74ca73a9
[2017-06-15 16:19:01,925: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 741dfe8f-5137-420a-b468-d15b74ca73a9
[2017-06-15 16:19:02,324: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:19:02,324: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:19:02,325: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:19:02,325: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:19:02,325: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1deb8f88-5b87-4ce8-b8d4-2778709f5964
[2017-06-15 16:19:02,325: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1deb8f88-5b87-4ce8-b8d4-2778709f5964
[2017-06-15 16:19:02,336: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1deb8f88-5b87-4ce8-b8d4-2778709f5964
[2017-06-15 16:19:02,336: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1deb8f88-5b87-4ce8-b8d4-2778709f5964
[2017-06-15 16:19:02,337: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-06-15 16:19:02,362: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 448c161c-1df4-4ea0-817e-fcba61242e98
[2017-06-15 16:19:02,362: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 448c161c-1df4-4ea0-817e-fcba61242e98
[2017-06-15 16:19:04,535: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 16:19:04,535: INFO/job_scheduler]   Job 265 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-15 16:19:04,535: INFO/job_scheduler]   Job 266 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:04,535: INFO/job_scheduler]   Job 267 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:04,535: INFO/job_scheduler]   Job 268 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:04,535: INFO/job_scheduler]   Job 269 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:04,535: INFO/job_scheduler]   Job 272 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:04,535: INFO/job_scheduler]   Job 275 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 16:19:04,536: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-15 16:19:06,127: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/4054b950-dc98-4664-b3e6-499cf8c2223e
[2017-06-15 16:19:06,127: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/4054b950-dc98-4664-b3e6-499cf8c2223e
[2017-06-15 16:19:06,127: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/4054b950-dc98-4664-b3e6-499cf8c2223e
[2017-06-15 16:19:06,127: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/4054b950-dc98-4664-b3e6-499cf8c2223e
[2017-06-15 16:19:06,127: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9b5c7b83-bfdd-43de-9e28-46c92d85f975
[2017-06-15 16:19:06,127: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9b5c7b83-bfdd-43de-9e28-46c92d85f975
[2017-06-15 16:19:06,146: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9b5c7b83-bfdd-43de-9e28-46c92d85f975
[2017-06-15 16:19:06,146: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9b5c7b83-bfdd-43de-9e28-46c92d85f975
[2017-06-15 16:19:06,146: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-06-15 16:19:06.146572+00:00:
[2017-06-15 16:19:06,146: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-06-15 16:19:06.146572+00:00:
[2017-06-15 16:19:06,146: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-06-15 16:19:06,146: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-06-15 16:19:06,160: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 16:19:06,160: INFO/job_scheduler]   Job 254 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-15 16:19:06,160: INFO/job_scheduler]   Job 255 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:06,160: INFO/job_scheduler]   Job 256 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:06,160: INFO/job_scheduler]   Job 257 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:06,160: INFO/job_scheduler]   Job 258 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:06,161: INFO/job_scheduler]   Job 262 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:06,161: INFO/job_scheduler]   Job 264 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 16:19:06,161: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-15 16:19:06,166: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-06-15 16:19:06.166277+00:00:
[2017-06-15 16:19:06,166: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-06-15 16:19:06.166277+00:00:
[2017-06-15 16:19:06,166: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-06-15 16:19:06,166: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-06-15 16:19:06,183: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 16:19:06,183: INFO/job_scheduler]   Job 254 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-15 16:19:06,183: INFO/job_scheduler]   Job 255 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:06,183: INFO/job_scheduler]   Job 256 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:06,183: INFO/job_scheduler]   Job 257 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:06,183: INFO/job_scheduler]   Job 258 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:06,184: INFO/job_scheduler]   Job 262 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:06,184: INFO/job_scheduler]   Job 264 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 16:19:06,184: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-15 16:19:06,194: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com restart_agent {}
[2017-06-15 16:19:06,208: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e1a69b0d-dba4-4fda-8884-a79c8557bf21
[2017-06-15 16:19:06,208: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e1a69b0d-dba4-4fda-8884-a79c8557bf21
[2017-06-15 16:19:06,208: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/77c9887b-c4d2-4c59-a8c1-87709202cdf2
[2017-06-15 16:19:06,208: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/77c9887b-c4d2-4c59-a8c1-87709202cdf2
[2017-06-15 16:19:06,209: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/77c9887b-c4d2-4c59-a8c1-87709202cdf2
[2017-06-15 16:19:06,209: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/77c9887b-c4d2-4c59-a8c1-87709202cdf2
[2017-06-15 16:19:06,209: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5413453b-4d7e-4726-bdb7-b8ccd531b7ab
[2017-06-15 16:19:06,209: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5413453b-4d7e-4726-bdb7-b8ccd531b7ab
[2017-06-15 16:19:06,256: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5413453b-4d7e-4726-bdb7-b8ccd531b7ab
[2017-06-15 16:19:06,256: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5413453b-4d7e-4726-bdb7-b8ccd531b7ab
[2017-06-15 16:19:06,256: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-06-15 16:19:06.256710+00:00:
[2017-06-15 16:19:06,256: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-06-15 16:19:06.256710+00:00:
[2017-06-15 16:19:06,256: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-06-15 16:19:06,256: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-06-15 16:19:06,276: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 16:19:06,276: INFO/job_scheduler]   Job 232 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-15 16:19:06,276: INFO/job_scheduler]   Job 233 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:06,276: INFO/job_scheduler]   Job 234 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:06,276: INFO/job_scheduler]   Job 235 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:06,276: INFO/job_scheduler]   Job 236 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:06,276: INFO/job_scheduler]   Job 240 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:06,276: INFO/job_scheduler]   Job 242 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 16:19:06,277: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-15 16:19:06,278: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-06-15 16:19:06.278913+00:00:
[2017-06-15 16:19:06,278: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-06-15 16:19:06.278913+00:00:
[2017-06-15 16:19:06,279: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-06-15 16:19:06,279: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-06-15 16:19:06,297: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 16:19:06,297: INFO/job_scheduler]   Job 232 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-15 16:19:06,297: INFO/job_scheduler]   Job 233 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:06,297: INFO/job_scheduler]   Job 234 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:06,297: INFO/job_scheduler]   Job 235 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:06,297: INFO/job_scheduler]   Job 236 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:06,297: INFO/job_scheduler]   Job 240 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:06,297: INFO/job_scheduler]   Job 242 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 16:19:06,298: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-15 16:19:06,299: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com restart_agent {}
[2017-06-15 16:19:06,316: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 96b5f81e-659d-444e-aef6-907c79ef2855
[2017-06-15 16:19:06,316: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 96b5f81e-659d-444e-aef6-907c79ef2855
[2017-06-15 16:19:06,324: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/4054b950-dc98-4664-b3e6-499cf8c2223e
[2017-06-15 16:19:06,324: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/4054b950-dc98-4664-b3e6-499cf8c2223e
[2017-06-15 16:19:06,324: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/4054b950-dc98-4664-b3e6-499cf8c2223e
[2017-06-15 16:19:06,324: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/4054b950-dc98-4664-b3e6-499cf8c2223e
[2017-06-15 16:19:06,324: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e1a69b0d-dba4-4fda-8884-a79c8557bf21
[2017-06-15 16:19:06,324: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e1a69b0d-dba4-4fda-8884-a79c8557bf21
[2017-06-15 16:19:06,325: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e1a69b0d-dba4-4fda-8884-a79c8557bf21
[2017-06-15 16:19:06,325: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e1a69b0d-dba4-4fda-8884-a79c8557bf21
[2017-06-15 16:19:06,325: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm7.lotus.hpdd.lab.intel.com (old 4054b950-dc98-4664-b3e6-499cf8c2223e)
[2017-06-15 16:19:06,325: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm7.lotus.hpdd.lab.intel.com (old 4054b950-dc98-4664-b3e6-499cf8c2223e)
[2017-06-15 16:19:06,401: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/77c9887b-c4d2-4c59-a8c1-87709202cdf2
[2017-06-15 16:19:06,401: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/77c9887b-c4d2-4c59-a8c1-87709202cdf2
[2017-06-15 16:19:06,401: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/77c9887b-c4d2-4c59-a8c1-87709202cdf2
[2017-06-15 16:19:06,401: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/77c9887b-c4d2-4c59-a8c1-87709202cdf2
[2017-06-15 16:19:06,401: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 96b5f81e-659d-444e-aef6-907c79ef2855
[2017-06-15 16:19:06,401: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 96b5f81e-659d-444e-aef6-907c79ef2855
[2017-06-15 16:19:06,438: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 96b5f81e-659d-444e-aef6-907c79ef2855
[2017-06-15 16:19:06,438: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 96b5f81e-659d-444e-aef6-907c79ef2855
[2017-06-15 16:19:06,438: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old 77c9887b-c4d2-4c59-a8c1-87709202cdf2)
[2017-06-15 16:19:06,438: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old 77c9887b-c4d2-4c59-a8c1-87709202cdf2)
[2017-06-15 16:19:06,684: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 16:19:06,685: INFO/job_scheduler]   Job 243 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-15 16:19:06,685: INFO/job_scheduler]   Job 244 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:06,685: INFO/job_scheduler]   Job 245 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:06,685: INFO/job_scheduler]   Job 246 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:06,685: INFO/job_scheduler]   Job 247 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:06,685: INFO/job_scheduler]   Job 251 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:06,685: INFO/job_scheduler]   Job 253 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 16:19:06,686: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-15 16:19:26,040: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/4054b950-dc98-4664-b3e6-499cf8c2223e
[2017-06-15 16:19:26,040: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/4054b950-dc98-4664-b3e6-499cf8c2223e
[2017-06-15 16:19:26,040: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 4054b950-dc98-4664-b3e6-499cf8c2223e because session terminated
[2017-06-15 16:19:26,040: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 4054b950-dc98-4664-b3e6-499cf8c2223e because session terminated
[2017-06-15 16:19:26,048: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 16:19:26,048: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 16:19:26,130: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/77c9887b-c4d2-4c59-a8c1-87709202cdf2
[2017-06-15 16:19:26,130: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/77c9887b-c4d2-4c59-a8c1-87709202cdf2
[2017-06-15 16:19:26,130: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 77c9887b-c4d2-4c59-a8c1-87709202cdf2 because session terminated
[2017-06-15 16:19:26,130: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 77c9887b-c4d2-4c59-a8c1-87709202cdf2 because session terminated
[2017-06-15 16:19:26,132: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 16:19:26,132: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 16:19:26,357: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm7.lotus.hpdd.lab.intel.com new f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 16:19:26,357: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm7.lotus.hpdd.lab.intel.com new f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 16:19:26,358: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-15 16:19:26,390: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0d46ccf9-1e65-4f96-8fe8-226b9f4c6b90
[2017-06-15 16:19:26,390: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0d46ccf9-1e65-4f96-8fe8-226b9f4c6b90
[2017-06-15 16:19:26,464: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 16:19:26,464: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 16:19:26,464: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-15 16:19:26,485: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a2d4fee6-8032-4f0d-af51-99439569b8e8
[2017-06-15 16:19:26,485: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a2d4fee6-8032-4f0d-af51-99439569b8e8
[2017-06-15 16:19:26,592: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 16:19:26,592: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 16:19:26,593: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 16:19:26,593: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 16:19:26,593: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a2d4fee6-8032-4f0d-af51-99439569b8e8
[2017-06-15 16:19:26,593: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a2d4fee6-8032-4f0d-af51-99439569b8e8
[2017-06-15 16:19:26,603: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a2d4fee6-8032-4f0d-af51-99439569b8e8
[2017-06-15 16:19:26,603: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a2d4fee6-8032-4f0d-af51-99439569b8e8
[2017-06-15 16:19:26,603: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-06-15 16:19:26,621: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ee7d71e4-eb0e-4c6b-b14e-0d79115fdab4
[2017-06-15 16:19:26,621: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ee7d71e4-eb0e-4c6b-b14e-0d79115fdab4
[2017-06-15 16:19:27,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 16:19:27,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 16:19:27,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 16:19:27,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 16:19:27,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0d46ccf9-1e65-4f96-8fe8-226b9f4c6b90
[2017-06-15 16:19:27,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0d46ccf9-1e65-4f96-8fe8-226b9f4c6b90
[2017-06-15 16:19:27,508: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0d46ccf9-1e65-4f96-8fe8-226b9f4c6b90
[2017-06-15 16:19:27,508: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0d46ccf9-1e65-4f96-8fe8-226b9f4c6b90
[2017-06-15 16:19:27,509: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-06-15 16:19:27,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2912cce4-638d-49a4-a828-3bba4bb0c7c6
[2017-06-15 16:19:27,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2912cce4-638d-49a4-a828-3bba4bb0c7c6
[2017-06-15 16:19:31,302: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 16:19:31,302: INFO/job_scheduler]   Job 254 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-15 16:19:31,302: INFO/job_scheduler]   Job 255 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:31,302: INFO/job_scheduler]   Job 256 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:31,302: INFO/job_scheduler]   Job 257 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:31,303: INFO/job_scheduler]   Job 258 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:31,303: INFO/job_scheduler]   Job 262 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:31,303: INFO/job_scheduler]   Job 264 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 16:19:31,303: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-15 16:19:31,436: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 16:19:31,436: INFO/job_scheduler]   Job 232 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-15 16:19:31,436: INFO/job_scheduler]   Job 233 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:31,436: INFO/job_scheduler]   Job 234 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:31,437: INFO/job_scheduler]   Job 235 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:31,437: INFO/job_scheduler]   Job 236 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:31,437: INFO/job_scheduler]   Job 240 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 16:19:31,437: INFO/job_scheduler]   Job 242 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 16:19:31,437: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-15 16:19:36,817: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:19:36,817: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:19:36,817: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:19:36,817: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:19:36,817: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 448c161c-1df4-4ea0-817e-fcba61242e98
[2017-06-15 16:19:36,817: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 448c161c-1df4-4ea0-817e-fcba61242e98
[2017-06-15 16:19:36,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 448c161c-1df4-4ea0-817e-fcba61242e98
[2017-06-15 16:19:36,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 448c161c-1df4-4ea0-817e-fcba61242e98
[2017-06-15 16:19:37,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 16:19:37,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 16:19:37,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 16:19:37,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 16:19:37,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 741dfe8f-5137-420a-b468-d15b74ca73a9
[2017-06-15 16:19:37,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 741dfe8f-5137-420a-b468-d15b74ca73a9
[2017-06-15 16:19:37,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 741dfe8f-5137-420a-b468-d15b74ca73a9
[2017-06-15 16:19:37,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 741dfe8f-5137-420a-b468-d15b74ca73a9
[2017-06-15 16:19:37,821: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com install_packages {'repos': [u'iml-agent'], 'packages': [u'zfs', u'zfs-dkms', u'kernel-devel-[0-9]*_lustre', u'kernel-[0-9]*_lustre', u'lustre-osd-zfs-mount', u'kmod-lustre-osd-ldiskfs', u'lustre-dkms', u'lustre', u'chroma-agent-management']}
[2017-06-15 16:19:37,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3c9946b9-a6c7-416a-9fcb-4827b9d6dda8
[2017-06-15 16:19:37,856: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3c9946b9-a6c7-416a-9fcb-4827b9d6dda8
[2017-06-15 16:19:38,421: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com install_packages {'repos': [u'iml-agent'], 'packages': [u'zfs', u'zfs-dkms', u'kernel-devel-[0-9]*_lustre', u'kernel-[0-9]*_lustre', u'lustre-osd-zfs-mount', u'kmod-lustre-osd-ldiskfs', u'lustre-dkms', u'lustre', u'chroma-agent-management']}
[2017-06-15 16:19:38,436: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4586ef08-5288-4183-9e11-1e8735cf9a08
[2017-06-15 16:19:38,436: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4586ef08-5288-4183-9e11-1e8735cf9a08
[2017-06-15 16:19:57,000: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 16:19:57,000: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 16:19:57,000: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 16:19:57,000: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 16:19:57,000: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ee7d71e4-eb0e-4c6b-b14e-0d79115fdab4
[2017-06-15 16:19:57,000: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ee7d71e4-eb0e-4c6b-b14e-0d79115fdab4
[2017-06-15 16:19:57,011: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ee7d71e4-eb0e-4c6b-b14e-0d79115fdab4
[2017-06-15 16:19:57,011: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ee7d71e4-eb0e-4c6b-b14e-0d79115fdab4
[2017-06-15 16:19:57,445: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com install_packages {'repos': [u'iml-agent'], 'packages': [u'zfs', u'zfs-dkms', u'kernel-devel-[0-9]*_lustre', u'kernel-[0-9]*_lustre', u'lustre-osd-zfs-mount', u'kmod-lustre-osd-ldiskfs', u'lustre-dkms', u'lustre', u'chroma-agent-management']}
[2017-06-15 16:19:57,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 16d1ee6b-20d5-4afa-a4f8-8f22844ddc0e
[2017-06-15 16:19:57,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 16d1ee6b-20d5-4afa-a4f8-8f22844ddc0e
[2017-06-15 16:19:58,877: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 16:19:58,877: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 16:19:58,877: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 16:19:58,877: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 16:19:58,877: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2912cce4-638d-49a4-a828-3bba4bb0c7c6
[2017-06-15 16:19:58,877: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2912cce4-638d-49a4-a828-3bba4bb0c7c6
[2017-06-15 16:19:58,880: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2912cce4-638d-49a4-a828-3bba4bb0c7c6
[2017-06-15 16:19:58,880: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2912cce4-638d-49a4-a828-3bba4bb0c7c6
[2017-06-15 16:19:59,527: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com install_packages {'repos': [u'iml-agent'], 'packages': [u'zfs', u'zfs-dkms', u'kernel-devel-[0-9]*_lustre', u'kernel-[0-9]*_lustre', u'lustre-osd-zfs-mount', u'kmod-lustre-osd-ldiskfs', u'lustre-dkms', u'lustre', u'chroma-agent-management']}
[2017-06-15 16:19:59,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d6f03e75-36f1-4ff7-a938-c7307cb5ca27
[2017-06-15 16:19:59,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d6f03e75-36f1-4ff7-a938-c7307cb5ca27
[2017-06-15 16:45:19,958: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 16:45:19,958: INFO/job_scheduler]   Job 265 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-15 16:45:19,958: INFO/job_scheduler]   Job 266 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 16:45:19,958: INFO/job_scheduler]   Job 267 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 16:45:19,958: INFO/job_scheduler]   Job 268 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 16:45:19,958: INFO/job_scheduler]   Job 269 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 16:45:19,958: INFO/job_scheduler]   Job 272 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 16:45:19,958: INFO/job_scheduler]   Job 275 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 16:45:19,959: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-15 16:45:29,841: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 16:45:29,841: INFO/job_scheduler]   Job 232 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-15 16:45:29,841: INFO/job_scheduler]   Job 233 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 16:45:29,841: INFO/job_scheduler]   Job 234 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 16:45:29,841: INFO/job_scheduler]   Job 235 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 16:45:29,841: INFO/job_scheduler]   Job 236 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 16:45:29,841: INFO/job_scheduler]   Job 240 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 16:45:29,842: INFO/job_scheduler]   Job 242 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 16:45:29,842: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-15 16:45:56,462: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 16:45:56,462: INFO/job_scheduler]   Job 254 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-15 16:45:56,462: INFO/job_scheduler]   Job 255 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 16:45:56,462: INFO/job_scheduler]   Job 256 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 16:45:56,462: INFO/job_scheduler]   Job 257 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 16:45:56,462: INFO/job_scheduler]   Job 258 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 16:45:56,462: INFO/job_scheduler]   Job 262 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 16:45:56,463: INFO/job_scheduler]   Job 264 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 16:45:56,463: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-15 16:46:24,251: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 16:46:24,251: INFO/job_scheduler]   Job 243 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-15 16:46:24,251: INFO/job_scheduler]   Job 244 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 16:46:24,251: INFO/job_scheduler]   Job 245 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 16:46:24,251: INFO/job_scheduler]   Job 246 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 16:46:24,251: INFO/job_scheduler]   Job 247 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 16:46:24,252: INFO/job_scheduler]   Job 251 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 16:46:24,252: INFO/job_scheduler]   Job 253 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 16:46:24,252: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-15 16:46:26,068: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:46:26,068: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:46:26,124: INFO/command_plan] add_jobs: created Job 276 (Trigger plugin poll for linux plugins)
[2017-06-15 16:46:26,152: INFO/job_scheduler] run_next: 1 runnable jobs of (41 pending, 4 tasked)
[2017-06-15 16:46:26,197: INFO/job_scheduler] Job 276: RunJobThread.run
[2017-06-15 16:46:26,245: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-15 16:46:26,268: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bc55ea21-5b36-442b-b292-bf05f7d0461b
[2017-06-15 16:46:26,268: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bc55ea21-5b36-442b-b292-bf05f7d0461b
[2017-06-15 16:46:26,269: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-15 16:46:26,295: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 66b7ce82-3011-43b9-9342-436cd6a5d7a8
[2017-06-15 16:46:26,295: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 66b7ce82-3011-43b9-9342-436cd6a5d7a8
[2017-06-15 16:46:26,308: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-15 16:46:26,348: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5a709c8a-836a-407e-9bac-78b045e3b3d0
[2017-06-15 16:46:26,348: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5a709c8a-836a-407e-9bac-78b045e3b3d0
[2017-06-15 16:46:26,383: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 16:46:26,383: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 16:46:26,384: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 16:46:26,384: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 16:46:26,384: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bc55ea21-5b36-442b-b292-bf05f7d0461b
[2017-06-15 16:46:26,384: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bc55ea21-5b36-442b-b292-bf05f7d0461b
[2017-06-15 16:46:26,397: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 16:46:26,397: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 16:46:26,397: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 16:46:26,397: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 16:46:26,397: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 66b7ce82-3011-43b9-9342-436cd6a5d7a8
[2017-06-15 16:46:26,397: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 66b7ce82-3011-43b9-9342-436cd6a5d7a8
[2017-06-15 16:46:26,404: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bc55ea21-5b36-442b-b292-bf05f7d0461b
[2017-06-15 16:46:26,404: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bc55ea21-5b36-442b-b292-bf05f7d0461b
[2017-06-15 16:46:26,404: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-15 16:46:26,415: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 16:46:26,415: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 16:46:26,415: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 16:46:26,415: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 16:46:26,415: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5a709c8a-836a-407e-9bac-78b045e3b3d0
[2017-06-15 16:46:26,415: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5a709c8a-836a-407e-9bac-78b045e3b3d0
[2017-06-15 16:46:26,418: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5a709c8a-836a-407e-9bac-78b045e3b3d0
[2017-06-15 16:46:26,418: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5a709c8a-836a-407e-9bac-78b045e3b3d0
[2017-06-15 16:46:26,418: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-15 16:46:26,433: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 66b7ce82-3011-43b9-9342-436cd6a5d7a8
[2017-06-15 16:46:26,433: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 66b7ce82-3011-43b9-9342-436cd6a5d7a8
[2017-06-15 16:46:26,433: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-15 16:46:26,434: INFO/job_scheduler] Job 276 finished 1 steps successfully
[2017-06-15 16:46:26,647: INFO/job_scheduler] Job 276: complete_job: Updating cache
[2017-06-15 16:46:26,647: INFO/job_scheduler] Job 276 completing (errored=False, cancelled=False)
[2017-06-15 16:46:26,756: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-15 16:46:31,668: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:46:31,668: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:46:31,904: INFO/command_plan] add_jobs: created Job 277 (Trigger plugin poll for linux plugins)
[2017-06-15 16:46:31,983: INFO/job_scheduler] run_next: 1 runnable jobs of (41 pending, 4 tasked)
[2017-06-15 16:46:32,015: INFO/job_scheduler] Job 277: RunJobThread.run
[2017-06-15 16:46:32,037: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-15 16:46:32,078: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 40076b1c-88e1-4d21-bef3-d7d34002a2c7
[2017-06-15 16:46:32,078: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 40076b1c-88e1-4d21-bef3-d7d34002a2c7
[2017-06-15 16:46:32,080: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-15 16:46:32,102: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-15 16:46:32,113: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc efa7bf99-9be2-47a6-8fdc-afc1c9566641
[2017-06-15 16:46:32,113: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc efa7bf99-9be2-47a6-8fdc-afc1c9566641
[2017-06-15 16:46:32,146: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 45a96781-08e1-4841-8a82-7f072f0ec038
[2017-06-15 16:46:32,146: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 45a96781-08e1-4841-8a82-7f072f0ec038
[2017-06-15 16:46:32,147: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 16:46:32,147: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 16:46:32,147: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 16:46:32,147: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 16:46:32,147: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 40076b1c-88e1-4d21-bef3-d7d34002a2c7
[2017-06-15 16:46:32,147: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 40076b1c-88e1-4d21-bef3-d7d34002a2c7
[2017-06-15 16:46:32,154: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 40076b1c-88e1-4d21-bef3-d7d34002a2c7
[2017-06-15 16:46:32,154: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 40076b1c-88e1-4d21-bef3-d7d34002a2c7
[2017-06-15 16:46:32,154: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-15 16:46:32,194: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 16:46:32,194: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 16:46:32,194: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 16:46:32,194: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 16:46:32,194: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc efa7bf99-9be2-47a6-8fdc-afc1c9566641
[2017-06-15 16:46:32,194: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc efa7bf99-9be2-47a6-8fdc-afc1c9566641
[2017-06-15 16:46:32,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc efa7bf99-9be2-47a6-8fdc-afc1c9566641
[2017-06-15 16:46:32,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc efa7bf99-9be2-47a6-8fdc-afc1c9566641
[2017-06-15 16:46:32,199: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-15 16:46:32,224: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:46:32,224: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:46:32,224: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:46:32,224: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:46:32,224: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 45a96781-08e1-4841-8a82-7f072f0ec038
[2017-06-15 16:46:32,224: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 45a96781-08e1-4841-8a82-7f072f0ec038
[2017-06-15 16:46:32,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 45a96781-08e1-4841-8a82-7f072f0ec038
[2017-06-15 16:46:32,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 45a96781-08e1-4841-8a82-7f072f0ec038
[2017-06-15 16:46:32,270: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-15 16:46:32,271: INFO/job_scheduler] Job 277 finished 1 steps successfully
[2017-06-15 16:46:32,339: INFO/job_scheduler] Job 277: complete_job: Updating cache
[2017-06-15 16:46:32,339: INFO/job_scheduler] Job 277 completing (errored=False, cancelled=False)
[2017-06-15 16:46:32,467: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-15 16:47:04,139: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:47:04,139: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:47:04,348: INFO/command_plan] add_jobs: created Job 278 (Trigger plugin poll for linux plugins)
[2017-06-15 16:47:04,431: INFO/job_scheduler] run_next: 1 runnable jobs of (41 pending, 4 tasked)
[2017-06-15 16:47:04,453: INFO/job_scheduler] Job 278: RunJobThread.run
[2017-06-15 16:47:04,474: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-15 16:47:04,504: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a4ad4a76-7950-4b07-aa6f-ec083a9d5b47
[2017-06-15 16:47:04,504: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a4ad4a76-7950-4b07-aa6f-ec083a9d5b47
[2017-06-15 16:47:04,504: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-15 16:47:04,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4b4604ca-4690-4516-b0f8-2e38d4f15c30
[2017-06-15 16:47:04,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4b4604ca-4690-4516-b0f8-2e38d4f15c30
[2017-06-15 16:47:04,572: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-15 16:47:04,621: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6a9928d5-b5c8-40a3-9c6c-da37a90aa946
[2017-06-15 16:47:04,621: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6a9928d5-b5c8-40a3-9c6c-da37a90aa946
[2017-06-15 16:47:04,622: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 16:47:04,622: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 16:47:04,622: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 16:47:04,622: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 16:47:04,622: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a4ad4a76-7950-4b07-aa6f-ec083a9d5b47
[2017-06-15 16:47:04,622: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a4ad4a76-7950-4b07-aa6f-ec083a9d5b47
[2017-06-15 16:47:04,622: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 16:47:04,622: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 16:47:04,623: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 16:47:04,623: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 16:47:04,623: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4b4604ca-4690-4516-b0f8-2e38d4f15c30
[2017-06-15 16:47:04,623: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4b4604ca-4690-4516-b0f8-2e38d4f15c30
[2017-06-15 16:47:04,638: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a4ad4a76-7950-4b07-aa6f-ec083a9d5b47
[2017-06-15 16:47:04,638: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a4ad4a76-7950-4b07-aa6f-ec083a9d5b47
[2017-06-15 16:47:04,638: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-15 16:47:04,668: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4b4604ca-4690-4516-b0f8-2e38d4f15c30
[2017-06-15 16:47:04,668: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4b4604ca-4690-4516-b0f8-2e38d4f15c30
[2017-06-15 16:47:04,669: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-15 16:47:04,694: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:47:04,694: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:47:04,695: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:47:04,695: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:47:04,695: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6a9928d5-b5c8-40a3-9c6c-da37a90aa946
[2017-06-15 16:47:04,695: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6a9928d5-b5c8-40a3-9c6c-da37a90aa946
[2017-06-15 16:47:04,695: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6a9928d5-b5c8-40a3-9c6c-da37a90aa946
[2017-06-15 16:47:04,695: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6a9928d5-b5c8-40a3-9c6c-da37a90aa946
[2017-06-15 16:47:04,695: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-15 16:47:04,696: INFO/job_scheduler] Job 278 finished 1 steps successfully
[2017-06-15 16:47:04,775: INFO/job_scheduler] Job 278: complete_job: Updating cache
[2017-06-15 16:47:04,775: INFO/job_scheduler] Job 278 completing (errored=False, cancelled=False)
[2017-06-15 16:47:04,930: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-15 16:47:40,753: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 16:47:40,753: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-15 16:47:40,782: INFO/command_plan] add_jobs: created Job 279 (Trigger plugin poll for linux plugins)
[2017-06-15 16:47:40,802: INFO/job_scheduler] run_next: 1 runnable jobs of (41 pending, 4 tasked)
[2017-06-15 16:47:40,824: INFO/job_scheduler] Job 279: RunJobThread.run
[2017-06-15 16:47:40,839: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-15 16:47:40,865: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc eed0bfd9-b720-4a2b-8790-04f03f2eed39
[2017-06-15 16:47:40,865: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc eed0bfd9-b720-4a2b-8790-04f03f2eed39
[2017-06-15 16:47:40,877: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-15 16:47:40,896: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-15 16:47:40,918: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 01c996d5-6f05-4739-9b82-569906504483
[2017-06-15 16:47:40,918: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 01c996d5-6f05-4739-9b82-569906504483
[2017-06-15 16:47:40,956: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d5f0cb0f-9c69-454e-9a45-266510f3fac4
[2017-06-15 16:47:40,956: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d5f0cb0f-9c69-454e-9a45-266510f3fac4
[2017-06-15 16:47:40,993: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 16:47:40,993: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 16:47:40,993: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 16:47:40,993: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 16:47:40,994: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc eed0bfd9-b720-4a2b-8790-04f03f2eed39
[2017-06-15 16:47:40,994: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc eed0bfd9-b720-4a2b-8790-04f03f2eed39
[2017-06-15 16:47:40,996: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 16:47:40,996: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 16:47:40,996: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 16:47:40,996: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 16:47:40,996: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 01c996d5-6f05-4739-9b82-569906504483
[2017-06-15 16:47:40,996: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 01c996d5-6f05-4739-9b82-569906504483
[2017-06-15 16:47:41,038: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:47:41,038: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:47:41,038: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:47:41,038: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:47:41,039: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d5f0cb0f-9c69-454e-9a45-266510f3fac4
[2017-06-15 16:47:41,039: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d5f0cb0f-9c69-454e-9a45-266510f3fac4
[2017-06-15 16:47:41,039: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc eed0bfd9-b720-4a2b-8790-04f03f2eed39
[2017-06-15 16:47:41,039: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc eed0bfd9-b720-4a2b-8790-04f03f2eed39
[2017-06-15 16:47:41,084: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-15 16:47:41,042: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 01c996d5-6f05-4739-9b82-569906504483
[2017-06-15 16:47:41,042: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 01c996d5-6f05-4739-9b82-569906504483
[2017-06-15 16:47:41,085: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-15 16:47:41,077: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d5f0cb0f-9c69-454e-9a45-266510f3fac4
[2017-06-15 16:47:41,077: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d5f0cb0f-9c69-454e-9a45-266510f3fac4
[2017-06-15 16:47:41,086: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-15 16:47:41,086: INFO/job_scheduler] Job 279 finished 1 steps successfully
[2017-06-15 16:47:41,561: INFO/job_scheduler] Job 279: complete_job: Updating cache
[2017-06-15 16:47:41,561: INFO/job_scheduler] Job 279 completing (errored=False, cancelled=False)
[2017-06-15 16:47:41,815: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-15 16:49:51,952: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 16:49:51,952: INFO/job_scheduler]   Job 265 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-15 16:49:51,952: INFO/job_scheduler]   Job 266 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 16:49:51,952: INFO/job_scheduler]   Job 267 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 16:49:51,952: INFO/job_scheduler]   Job 268 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 16:49:51,953: INFO/job_scheduler]   Job 269 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 16:49:51,953: INFO/job_scheduler]   Job 272 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 16:49:51,953: INFO/job_scheduler]   Job 275 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 16:49:51,953: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-15 16:49:55,530: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 16:49:55,531: INFO/job_scheduler]   Job 243 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-15 16:49:55,531: INFO/job_scheduler]   Job 244 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 16:49:55,531: INFO/job_scheduler]   Job 245 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 16:49:55,531: INFO/job_scheduler]   Job 246 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 16:49:55,531: INFO/job_scheduler]   Job 247 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 16:49:55,531: INFO/job_scheduler]   Job 251 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 16:49:55,532: INFO/job_scheduler]   Job 253 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 16:49:55,532: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-15 16:50:08,650: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 16:50:08,651: INFO/job_scheduler]   Job 254 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-15 16:50:08,651: INFO/job_scheduler]   Job 255 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 16:50:08,651: INFO/job_scheduler]   Job 256 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 16:50:08,651: INFO/job_scheduler]   Job 257 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 16:50:08,651: INFO/job_scheduler]   Job 258 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 16:50:08,651: INFO/job_scheduler]   Job 262 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 16:50:08,651: INFO/job_scheduler]   Job 264 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 16:50:08,652: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-15 16:50:16,378: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 16:50:16,378: INFO/job_scheduler]   Job 232 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-15 16:50:16,379: INFO/job_scheduler]   Job 233 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 16:50:16,379: INFO/job_scheduler]   Job 234 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 16:50:16,379: INFO/job_scheduler]   Job 235 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 16:50:16,379: INFO/job_scheduler]   Job 236 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 16:50:16,379: INFO/job_scheduler]   Job 240 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 16:50:16,379: INFO/job_scheduler]   Job 242 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 16:50:16,379: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-15 16:59:17,195: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:59:17,195: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:59:17,195: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:59:17,195: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:59:17,195: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3c9946b9-a6c7-416a-9fcb-4827b9d6dda8
[2017-06-15 16:59:17,195: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3c9946b9-a6c7-416a-9fcb-4827b9d6dda8
[2017-06-15 16:59:17,219: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3c9946b9-a6c7-416a-9fcb-4827b9d6dda8
[2017-06-15 16:59:17,219: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3c9946b9-a6c7-416a-9fcb-4827b9d6dda8
[2017-06-15 16:59:17,381: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com kernel_status {}
[2017-06-15 16:59:17,410: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5233f33e-1d33-4b60-b91c-77b9e7cdde18
[2017-06-15 16:59:17,410: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5233f33e-1d33-4b60-b91c-77b9e7cdde18
[2017-06-15 16:59:20,470: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:59:20,470: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:59:20,471: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:59:20,471: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:59:20,471: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5233f33e-1d33-4b60-b91c-77b9e7cdde18
[2017-06-15 16:59:20,471: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5233f33e-1d33-4b60-b91c-77b9e7cdde18
[2017-06-15 16:59:20,477: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5233f33e-1d33-4b60-b91c-77b9e7cdde18
[2017-06-15 16:59:20,477: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5233f33e-1d33-4b60-b91c-77b9e7cdde18
[2017-06-15 16:59:20,477: INFO/job] Job 265 RebootIfNeededStep: Reboot of lotus-32vm8.lotus.hpdd.lab.intel.com required to switch from running kernel kernel-3.10.0-514.10.2.el7_lustre.x86_64 to required kernel-3.10.0-514.2.2.el7_lustre.x86_64
[2017-06-15 16:59:20,477: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com reboot_server {}
[2017-06-15 16:59:20,492: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 58ce8fad-77b9-4df0-9134-2c32e521af2a
[2017-06-15 16:59:20,492: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 58ce8fad-77b9-4df0-9134-2c32e521af2a
[2017-06-15 16:59:20,588: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:59:20,588: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:59:20,588: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:59:20,588: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 16:59:20,588: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 58ce8fad-77b9-4df0-9134-2c32e521af2a
[2017-06-15 16:59:20,588: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 58ce8fad-77b9-4df0-9134-2c32e521af2a
[2017-06-15 16:59:20,607: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 58ce8fad-77b9-4df0-9134-2c32e521af2a
[2017-06-15 16:59:20,607: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 58ce8fad-77b9-4df0-9134-2c32e521af2a
[2017-06-15 16:59:20,607: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm8.lotus.hpdd.lab.intel.com (old 49f71cbf-b939-439b-b341-b4a971c1fb70)
[2017-06-15 16:59:20,607: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm8.lotus.hpdd.lab.intel.com (old 49f71cbf-b939-439b-b341-b4a971c1fb70)
[2017-06-15 17:00:07,792: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:00:07,792: INFO/job_scheduler]   Job 243 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-15 17:00:07,792: INFO/job_scheduler]   Job 244 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 17:00:07,792: INFO/job_scheduler]   Job 245 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 17:00:07,793: INFO/job_scheduler]   Job 246 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 17:00:07,793: INFO/job_scheduler]   Job 247 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 17:00:07,793: INFO/job_scheduler]   Job 251 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 17:00:07,793: INFO/job_scheduler]   Job 253 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:00:07,793: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-15 17:00:22,827: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:00:22,827: INFO/job_scheduler]   Job 254 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-15 17:00:22,827: INFO/job_scheduler]   Job 255 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 17:00:22,827: INFO/job_scheduler]   Job 256 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 17:00:22,828: INFO/job_scheduler]   Job 257 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 17:00:22,828: INFO/job_scheduler]   Job 258 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 17:00:22,828: INFO/job_scheduler]   Job 262 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 17:00:22,828: INFO/job_scheduler]   Job 264 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:00:22,828: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-15 17:00:23,931: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 17:00:23,931: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/49f71cbf-b939-439b-b341-b4a971c1fb70
[2017-06-15 17:00:23,931: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 49f71cbf-b939-439b-b341-b4a971c1fb70 because session terminated
[2017-06-15 17:00:23,931: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 49f71cbf-b939-439b-b341-b4a971c1fb70 because session terminated
[2017-06-15 17:00:29,642: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:00:29,643: INFO/job_scheduler]   Job 232 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-15 17:00:29,643: INFO/job_scheduler]   Job 233 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 17:00:29,643: INFO/job_scheduler]   Job 234 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 17:00:29,643: INFO/job_scheduler]   Job 235 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 17:00:29,643: INFO/job_scheduler]   Job 236 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 17:00:29,643: INFO/job_scheduler]   Job 240 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 17:00:29,644: INFO/job_scheduler]   Job 242 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:00:29,644: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-15 17:00:48,086: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:00:48,086: INFO/job_scheduler]   Job 265 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-15 17:00:48,086: INFO/job_scheduler]   Job 266 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 17:00:48,086: INFO/job_scheduler]   Job 267 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 17:00:48,087: INFO/job_scheduler]   Job 268 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 17:00:48,087: INFO/job_scheduler]   Job 269 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 17:00:48,087: INFO/job_scheduler]   Job 272 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 17:00:48,087: INFO/job_scheduler]   Job 275 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:00:48,087: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-15 17:00:48,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:00:48,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:00:48,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm8.lotus.hpdd.lab.intel.com new 1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:00:48,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm8.lotus.hpdd.lab.intel.com new 1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:00:48,733: INFO/job_scheduler] Job 265 finished 4 steps successfully
[2017-06-15 17:00:48,775: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 140623538076688
[2017-06-15 17:00:48,779: INFO/job] Job 265: StateChangeJob complete, setting state packages_installed on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 17:00:48,779: INFO/job] Job 265: not_deleted=True
[2017-06-15 17:00:48,779: INFO/job_scheduler] Job 265: complete_job: Updating cache
[2017-06-15 17:00:48,779: INFO/job_scheduler] Job 265: locked_item 140623538154896 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-15 17:00:48,782: INFO/job_scheduler] Job 265 completing (errored=False, cancelled=False)
[2017-06-15 17:00:48,863: INFO/job_scheduler] run_next: 4 runnable jobs of (40 pending, 3 tasked)
[2017-06-15 17:00:48,929: INFO/job_scheduler] Job 266: RunJobThread.run
[2017-06-15 17:00:48,930: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-06-15 17:00:48,938: INFO/job_scheduler] Job 267: open DB connection during completion
[2017-06-15 17:00:48,988: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc efd9b7aa-e1e6-47fe-8cb5-b438c3ccb1f7
[2017-06-15 17:00:48,988: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc efd9b7aa-e1e6-47fe-8cb5-b438c3ccb1f7
[2017-06-15 17:00:48,999: INFO/job_scheduler] Job 268: RunJobThread.run
[2017-06-15 17:00:48,999: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-06-15 17:00:49,037: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c38436a6-754e-4646-a20b-0a8c0a019e02
[2017-06-15 17:00:49,037: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c38436a6-754e-4646-a20b-0a8c0a019e02
[2017-06-15 17:00:49,048: INFO/job_scheduler] Job 269: RunJobThread.run
[2017-06-15 17:00:49,056: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 140623739791184
[2017-06-15 17:00:49,070: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-06-15 17:00:49,104: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-06-15 17:00:49,112: INFO/job] Job 267: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-15 17:00:49,112: INFO/job] Job 267: not_deleted=True
[2017-06-15 17:00:49,112: INFO/job_scheduler] Job 267: complete_job: Updating cache
[2017-06-15 17:00:49,112: INFO/job_scheduler] Job 267: locked_item 140623739791184 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-15 17:00:49,122: INFO/job_scheduler] Job 267 completing (errored=False, cancelled=False)
[2017-06-15 17:00:49,134: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5a40a23a-8c1a-43e3-ae68-c08e4579b8c9
[2017-06-15 17:00:49,134: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5a40a23a-8c1a-43e3-ae68-c08e4579b8c9
[2017-06-15 17:00:49,171: INFO/job_scheduler] run_next: 1 runnable jobs of (36 pending, 6 tasked)
[2017-06-15 17:00:49,216: INFO/job_scheduler] Job 270: RunJobThread.run
[2017-06-15 17:00:49,217: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com load_lnet {}
[2017-06-15 17:00:49,256: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 815764be-f603-4550-baa9-e1ca7e454076
[2017-06-15 17:00:49,256: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 815764be-f603-4550-baa9-e1ca7e454076
[2017-06-15 17:00:49,485: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:00:49,485: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:00:49,485: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:00:49,485: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:00:49,486: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 815764be-f603-4550-baa9-e1ca7e454076
[2017-06-15 17:00:49,486: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 815764be-f603-4550-baa9-e1ca7e454076
[2017-06-15 17:00:49,530: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 815764be-f603-4550-baa9-e1ca7e454076
[2017-06-15 17:00:49,530: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 815764be-f603-4550-baa9-e1ca7e454076
[2017-06-15 17:00:49,531: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-15 17:00:49,548: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8d93c729-20a9-4008-8f89-8e7c87fe1ff4
[2017-06-15 17:00:49,548: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8d93c729-20a9-4008-8f89-8e7c87fe1ff4
[2017-06-15 17:00:49,653: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:00:49,653: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:00:49,654: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:00:49,654: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:00:49,654: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8d93c729-20a9-4008-8f89-8e7c87fe1ff4
[2017-06-15 17:00:49,654: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8d93c729-20a9-4008-8f89-8e7c87fe1ff4
[2017-06-15 17:00:49,670: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8d93c729-20a9-4008-8f89-8e7c87fe1ff4
[2017-06-15 17:00:49,670: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8d93c729-20a9-4008-8f89-8e7c87fe1ff4
[2017-06-15 17:00:50,556: INFO/job_scheduler] Job 270 finished 2 steps successfully
[2017-06-15 17:00:50,693: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 140623739791184
[2017-06-15 17:00:50,698: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-06-15 17:00:50,702: INFO/job] Job 270: StateChangeJob complete, setting state lnet_down on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-15 17:00:50,703: INFO/job] Job 270: not_deleted=True
[2017-06-15 17:00:50,703: INFO/job_scheduler] Job 270: complete_job: Updating cache
[2017-06-15 17:00:50,703: INFO/job_scheduler] Job 270: locked_item 140623739791184 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-15 17:00:50,735: INFO/job_scheduler] Job 270 completing (errored=False, cancelled=False)
[2017-06-15 17:00:50,807: INFO/job_scheduler] run_next: 1 runnable jobs of (35 pending, 6 tasked)
[2017-06-15 17:00:50,818: INFO/job_scheduler] Job 273: RunJobThread.run
[2017-06-15 17:00:50,818: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_lnet {}
[2017-06-15 17:00:50,857: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a5183f8f-ba62-4f85-ae87-20de260c78eb
[2017-06-15 17:00:50,857: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a5183f8f-ba62-4f85-ae87-20de260c78eb
[2017-06-15 17:00:53,816: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:00:53,816: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:00:53,816: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:00:53,816: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:00:53,816: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a5183f8f-ba62-4f85-ae87-20de260c78eb
[2017-06-15 17:00:53,816: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a5183f8f-ba62-4f85-ae87-20de260c78eb
[2017-06-15 17:00:53,834: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a5183f8f-ba62-4f85-ae87-20de260c78eb
[2017-06-15 17:00:53,834: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a5183f8f-ba62-4f85-ae87-20de260c78eb
[2017-06-15 17:00:53,835: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-15 17:00:53,861: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a861eb5c-6273-4a09-9216-ef0938c159cd
[2017-06-15 17:00:53,861: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a861eb5c-6273-4a09-9216-ef0938c159cd
[2017-06-15 17:00:55,291: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:00:55,291: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:00:55,291: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:00:55,291: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:00:55,291: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5a40a23a-8c1a-43e3-ae68-c08e4579b8c9
[2017-06-15 17:00:55,291: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5a40a23a-8c1a-43e3-ae68-c08e4579b8c9
[2017-06-15 17:00:55,304: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5a40a23a-8c1a-43e3-ae68-c08e4579b8c9
[2017-06-15 17:00:55,304: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5a40a23a-8c1a-43e3-ae68-c08e4579b8c9
[2017-06-15 17:00:55,304: INFO/job_scheduler] Job 269 finished 1 steps successfully
[2017-06-15 17:00:55,335: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:00:55,335: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:00:55,335: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:00:55,335: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:00:55,336: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a861eb5c-6273-4a09-9216-ef0938c159cd
[2017-06-15 17:00:55,336: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a861eb5c-6273-4a09-9216-ef0938c159cd
[2017-06-15 17:00:55,369: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:00:55,369: INFO/job_scheduler]   Job 266 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 17:00:55,369: INFO/job_scheduler]   Job 268 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 17:00:55,369: INFO/job_scheduler]   Job 269 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 17:00:55,369: INFO/job_scheduler]   Job 272 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 17:00:55,369: INFO/job_scheduler]   Job 275 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:00:55,370: INFO/job_scheduler] run_next: 0 runnable jobs of (34 pending, 7 tasked)
[2017-06-15 17:00:55,384: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a861eb5c-6273-4a09-9216-ef0938c159cd
[2017-06-15 17:00:55,384: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a861eb5c-6273-4a09-9216-ef0938c159cd
[2017-06-15 17:00:55,443: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:00:55,443: INFO/job_scheduler]   Job 266 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 17:00:55,443: INFO/job_scheduler]   Job 268 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 17:00:55,443: INFO/job_scheduler]   Job 269 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 17:00:55,443: INFO/job_scheduler]   Job 272 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 17:00:55,443: INFO/job_scheduler]   Job 275 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:00:55,444: INFO/job_scheduler] run_next: 0 runnable jobs of (34 pending, 7 tasked)
[2017-06-15 17:00:55,590: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 140623740273744
[2017-06-15 17:00:55,594: INFO/job] Job 269: StateChangeJob complete, setting state configured on lotus-32vm8.lotus.hpdd.lab.intel.com NTP configuration
[2017-06-15 17:00:55,594: INFO/job] Job 269: not_deleted=True
[2017-06-15 17:00:55,594: INFO/job_scheduler] Job 269: complete_job: Updating cache
[2017-06-15 17:00:55,594: INFO/job_scheduler] Job 269: locked_item 140623740273744 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-06-15 17:00:55,598: INFO/job_scheduler] Job 269 completing (errored=False, cancelled=False)
[2017-06-15 17:00:55,609: INFO/job_scheduler] Job 273 finished 2 steps successfully
[2017-06-15 17:00:55,656: INFO/job_scheduler] run_next: 0 runnable jobs of (34 pending, 6 tasked)
[2017-06-15 17:00:55,741: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 140623739791184
[2017-06-15 17:00:55,746: INFO/job] Job 273: StateChangeJob complete, setting state lnet_up on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-15 17:00:55,746: INFO/job] Job 273: not_deleted=True
[2017-06-15 17:00:55,746: INFO/job_scheduler] Job 273: complete_job: Updating cache
[2017-06-15 17:00:55,746: INFO/job_scheduler] Job 273: locked_item 140623739791184 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-15 17:00:55,751: INFO/job_scheduler] Job 273 completing (errored=False, cancelled=False)
[2017-06-15 17:00:55,783: INFO/job_scheduler] run_next: 0 runnable jobs of (34 pending, 5 tasked)
[2017-06-15 17:00:58,319: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:00:58,319: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:00:58,319: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:00:58,319: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:00:58,319: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c38436a6-754e-4646-a20b-0a8c0a019e02
[2017-06-15 17:00:58,319: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c38436a6-754e-4646-a20b-0a8c0a019e02
[2017-06-15 17:00:58,366: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c38436a6-754e-4646-a20b-0a8c0a019e02
[2017-06-15 17:00:58,366: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c38436a6-754e-4646-a20b-0a8c0a019e02
[2017-06-15 17:00:58,367: INFO/job_scheduler] Job 268 finished 1 steps successfully
[2017-06-15 17:00:58,657: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 140623538039120
[2017-06-15 17:00:58,661: INFO/job] Job 268: StateChangeJob complete, setting state configured on lotus-32vm8.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-06-15 17:00:58,661: INFO/job] Job 268: not_deleted=True
[2017-06-15 17:00:58,661: INFO/job_scheduler] Job 268: complete_job: Updating cache
[2017-06-15 17:00:58,661: INFO/job_scheduler] Job 268: locked_item 140623538039120 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-06-15 17:00:58,664: INFO/job_scheduler] Job 268 completing (errored=False, cancelled=False)
[2017-06-15 17:00:58,739: INFO/job_scheduler] run_next: 0 runnable jobs of (34 pending, 4 tasked)
[2017-06-15 17:01:18,206: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:01:18,206: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:01:18,207: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:01:18,207: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:01:18,207: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d6f03e75-36f1-4ff7-a938-c7307cb5ca27
[2017-06-15 17:01:18,207: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d6f03e75-36f1-4ff7-a938-c7307cb5ca27
[2017-06-15 17:01:18,207: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d6f03e75-36f1-4ff7-a938-c7307cb5ca27
[2017-06-15 17:01:18,207: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d6f03e75-36f1-4ff7-a938-c7307cb5ca27
[2017-06-15 17:01:18,303: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com kernel_status {}
[2017-06-15 17:01:18,336: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3216a862-4466-4ac7-870c-04ffe1063a8a
[2017-06-15 17:01:18,336: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3216a862-4466-4ac7-870c-04ffe1063a8a
[2017-06-15 17:01:21,448: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:01:21,448: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:01:21,449: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:01:21,449: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:01:21,449: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3216a862-4466-4ac7-870c-04ffe1063a8a
[2017-06-15 17:01:21,449: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3216a862-4466-4ac7-870c-04ffe1063a8a
[2017-06-15 17:01:21,453: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3216a862-4466-4ac7-870c-04ffe1063a8a
[2017-06-15 17:01:21,453: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3216a862-4466-4ac7-870c-04ffe1063a8a
[2017-06-15 17:01:21,454: INFO/job_scheduler] Job 254 finished 4 steps successfully
[2017-06-15 17:01:21,682: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 140623740276176
[2017-06-15 17:01:21,686: INFO/job] Job 254: StateChangeJob complete, setting state packages_installed on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 17:01:21,687: INFO/job] Job 254: not_deleted=True
[2017-06-15 17:01:21,687: INFO/job_scheduler] Job 254: complete_job: Updating cache
[2017-06-15 17:01:21,687: INFO/job_scheduler] Job 254: locked_item 140623739344336 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-15 17:01:21,690: INFO/job_scheduler] Job 254 completing (errored=False, cancelled=False)
[2017-06-15 17:01:21,779: INFO/job_scheduler] run_next: 4 runnable jobs of (34 pending, 3 tasked)
[2017-06-15 17:01:21,868: INFO/job_scheduler] Job 256: RunJobThread.run
[2017-06-15 17:01:21,869: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-06-15 17:01:21,890: INFO/job_scheduler] Job 257: RunJobThread.run
[2017-06-15 17:01:21,890: INFO/job_scheduler] Job 258: open DB connection during completion
[2017-06-15 17:01:21,894: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-06-15 17:01:21,904: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 79bec816-7728-4d2d-8cb7-c42f8c698580
[2017-06-15 17:01:21,904: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 79bec816-7728-4d2d-8cb7-c42f8c698580
[2017-06-15 17:01:21,924: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cecf974e-c1ff-4c39-acef-56baf882a7d4
[2017-06-15 17:01:21,924: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cecf974e-c1ff-4c39-acef-56baf882a7d4
[2017-06-15 17:01:22,022: INFO/job_scheduler] Job 255: RunJobThread.run
[2017-06-15 17:01:22,023: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-06-15 17:01:22,040: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 617df8a5-bce3-417b-927d-08629c2d97ce
[2017-06-15 17:01:22,040: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 617df8a5-bce3-417b-927d-08629c2d97ce
[2017-06-15 17:01:22,161: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 140623739789968
[2017-06-15 17:01:22,172: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-06-15 17:01:22,198: INFO/job] Job 258: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-15 17:01:22,199: INFO/job] Job 258: not_deleted=True
[2017-06-15 17:01:22,199: INFO/job_scheduler] Job 258: complete_job: Updating cache
[2017-06-15 17:01:22,199: INFO/job_scheduler] Job 258: locked_item 140623739789968 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-15 17:01:22,202: INFO/job_scheduler] Job 258 completing (errored=False, cancelled=False)
[2017-06-15 17:01:22,256: INFO/job_scheduler] run_next: 1 runnable jobs of (30 pending, 6 tasked)
[2017-06-15 17:01:22,307: INFO/job_scheduler] Job 260: RunJobThread.run
[2017-06-15 17:01:22,308: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com load_lnet {}
[2017-06-15 17:01:22,345: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2796aea5-d673-4ea2-a3dd-4c7138556477
[2017-06-15 17:01:22,345: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2796aea5-d673-4ea2-a3dd-4c7138556477
[2017-06-15 17:01:23,594: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:01:23,594: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:01:23,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:01:23,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:01:23,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2796aea5-d673-4ea2-a3dd-4c7138556477
[2017-06-15 17:01:23,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2796aea5-d673-4ea2-a3dd-4c7138556477
[2017-06-15 17:01:23,617: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2796aea5-d673-4ea2-a3dd-4c7138556477
[2017-06-15 17:01:23,617: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2796aea5-d673-4ea2-a3dd-4c7138556477
[2017-06-15 17:01:23,617: INFO/job] Job 260 LoadLNetStep: Error (1) running 'modprobe lnet': '' 'modprobe: ERROR: could not insert 'lnet': Invalid argument
'
[2017-06-15 17:01:23,617: ERROR/job_scheduler] Job 260 step 0 encountered an agent error: Error (1) running 'modprobe lnet': '' 'modprobe: ERROR: could not insert 'lnet': Invalid argument
'
[2017-06-15 17:01:23,778: INFO/job_scheduler] Job 260: complete_job: Updating cache
[2017-06-15 17:01:23,778: INFO/job_scheduler] Job 260: locked_item 140623739789968 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-15 17:01:23,781: INFO/job_scheduler] Job 260 completing (errored=True, cancelled=False)
[2017-06-15 17:01:23,953: INFO/job_scheduler] run_next: 1 runnable jobs of (29 pending, 6 tasked)
[2017-06-15 17:01:23,973: WARNING/job] DependOn not satisfied: lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration in state lnet_unloaded, not one of ['lnet_down'] (preferred lnet_down)
[2017-06-15 17:01:23,977: WARNING/job] DependOn not satisfied: lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration in state lnet_unloaded, not one of ['lnet_down'] (preferred lnet_down)
[2017-06-15 17:01:23,977: WARNING/job_scheduler] Job 261: cancelling because of failed dependency
[2017-06-15 17:01:23,977: INFO/job_scheduler] Job 261 completing (errored=False, cancelled=True)
[2017-06-15 17:01:23,992: INFO/job_scheduler] run_next: 0 runnable jobs of (28 pending, 6 tasked)
[2017-06-15 17:01:28,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:01:28,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:01:28,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:01:28,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:01:28,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cecf974e-c1ff-4c39-acef-56baf882a7d4
[2017-06-15 17:01:28,780: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cecf974e-c1ff-4c39-acef-56baf882a7d4
[2017-06-15 17:01:28,815: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cecf974e-c1ff-4c39-acef-56baf882a7d4
[2017-06-15 17:01:28,815: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cecf974e-c1ff-4c39-acef-56baf882a7d4
[2017-06-15 17:01:28,815: INFO/job_scheduler] Job 257 finished 1 steps successfully
[2017-06-15 17:01:28,952: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 140623739745040
[2017-06-15 17:01:28,956: INFO/job] Job 257: StateChangeJob complete, setting state configured on lotus-32vm7.lotus.hpdd.lab.intel.com NTP configuration
[2017-06-15 17:01:28,956: INFO/job] Job 257: not_deleted=True
[2017-06-15 17:01:28,956: INFO/job_scheduler] Job 257: complete_job: Updating cache
[2017-06-15 17:01:28,956: INFO/job_scheduler] Job 257: locked_item 140623739745040 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-06-15 17:01:28,959: INFO/job_scheduler] Job 257 completing (errored=False, cancelled=False)
[2017-06-15 17:01:28,991: INFO/job_scheduler] run_next: 0 runnable jobs of (28 pending, 5 tasked)
[2017-06-15 17:01:29,763: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:01:29,763: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:01:29,763: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:01:29,763: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:01:29,763: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 617df8a5-bce3-417b-927d-08629c2d97ce
[2017-06-15 17:01:29,763: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 617df8a5-bce3-417b-927d-08629c2d97ce
[2017-06-15 17:01:29,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 617df8a5-bce3-417b-927d-08629c2d97ce
[2017-06-15 17:01:29,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 617df8a5-bce3-417b-927d-08629c2d97ce
[2017-06-15 17:01:29,781: INFO/job_scheduler] Job 255 finished 1 steps successfully
[2017-06-15 17:01:29,852: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 140623739745232
[2017-06-15 17:01:29,856: INFO/job] Job 255: StateChangeJob complete, setting state configured on lotus-32vm7.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-06-15 17:01:29,856: INFO/job] Job 255: not_deleted=True
[2017-06-15 17:01:29,856: INFO/job_scheduler] Job 255: complete_job: Updating cache
[2017-06-15 17:01:29,856: INFO/job_scheduler] Job 255: locked_item 140623739745232 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-06-15 17:01:29,859: INFO/job_scheduler] Job 255 completing (errored=False, cancelled=False)
[2017-06-15 17:01:29,879: INFO/job_scheduler] run_next: 0 runnable jobs of (28 pending, 4 tasked)
[2017-06-15 17:01:30,928: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 17:01:30,928: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 17:01:30,928: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 17:01:30,928: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 17:01:30,929: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 16d1ee6b-20d5-4afa-a4f8-8f22844ddc0e
[2017-06-15 17:01:30,929: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 16d1ee6b-20d5-4afa-a4f8-8f22844ddc0e
[2017-06-15 17:01:30,958: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 16d1ee6b-20d5-4afa-a4f8-8f22844ddc0e
[2017-06-15 17:01:30,958: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 16d1ee6b-20d5-4afa-a4f8-8f22844ddc0e
[2017-06-15 17:01:31,086: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com kernel_status {}
[2017-06-15 17:01:31,120: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7742590d-0b0b-483b-9468-d82f721702f9
[2017-06-15 17:01:31,120: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7742590d-0b0b-483b-9468-d82f721702f9
[2017-06-15 17:01:34,253: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 17:01:34,253: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 17:01:34,254: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 17:01:34,254: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 17:01:34,254: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7742590d-0b0b-483b-9468-d82f721702f9
[2017-06-15 17:01:34,254: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7742590d-0b0b-483b-9468-d82f721702f9
[2017-06-15 17:01:34,287: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7742590d-0b0b-483b-9468-d82f721702f9
[2017-06-15 17:01:34,287: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7742590d-0b0b-483b-9468-d82f721702f9
[2017-06-15 17:01:34,288: INFO/job] Job 232 RebootIfNeededStep: Reboot of lotus-32vm5.lotus.hpdd.lab.intel.com required to switch from running kernel kernel-3.10.0-514.10.2.el7_lustre.x86_64 to required kernel-3.10.0-514.2.2.el7_lustre.x86_64
[2017-06-15 17:01:34,288: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com reboot_server {}
[2017-06-15 17:01:34,301: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0d1e8b26-7790-49a2-9303-d82530dada31
[2017-06-15 17:01:34,301: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0d1e8b26-7790-49a2-9303-d82530dada31
[2017-06-15 17:01:34,375: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 17:01:34,375: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 17:01:34,375: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 17:01:34,375: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 17:01:34,375: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0d1e8b26-7790-49a2-9303-d82530dada31
[2017-06-15 17:01:34,375: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0d1e8b26-7790-49a2-9303-d82530dada31
[2017-06-15 17:01:34,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0d1e8b26-7790-49a2-9303-d82530dada31
[2017-06-15 17:01:34,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0d1e8b26-7790-49a2-9303-d82530dada31
[2017-06-15 17:01:34,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old d0869604-e83e-4ab3-abcd-b66079073e07)
[2017-06-15 17:01:34,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old d0869604-e83e-4ab3-abcd-b66079073e07)
[2017-06-15 17:01:35,507: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:01:35,507: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:01:35,507: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:01:35,507: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:01:35,507: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc efd9b7aa-e1e6-47fe-8cb5-b438c3ccb1f7
[2017-06-15 17:01:35,507: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc efd9b7aa-e1e6-47fe-8cb5-b438c3ccb1f7
[2017-06-15 17:01:35,550: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc efd9b7aa-e1e6-47fe-8cb5-b438c3ccb1f7
[2017-06-15 17:01:35,550: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc efd9b7aa-e1e6-47fe-8cb5-b438c3ccb1f7
[2017-06-15 17:01:35,551: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.227', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-06-15 17:01:35,568: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 764d1c98-2dec-4ce8-a3c3-3896d8ed3032
[2017-06-15 17:01:35,568: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 764d1c98-2dec-4ce8-a3c3-3896d8ed3032
[2017-06-15 17:01:35,679: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:01:35,679: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:01:35,679: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:01:35,679: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:01:35,680: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 764d1c98-2dec-4ce8-a3c3-3896d8ed3032
[2017-06-15 17:01:35,680: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 764d1c98-2dec-4ce8-a3c3-3896d8ed3032
[2017-06-15 17:01:35,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 764d1c98-2dec-4ce8-a3c3-3896d8ed3032
[2017-06-15 17:01:35,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 764d1c98-2dec-4ce8-a3c3-3896d8ed3032
[2017-06-15 17:01:35,731: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 52617, 'pcs_password': 'Ln41rg0FyONifYwcTRRY'}
[2017-06-15 17:01:35,755: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 62a3afb1-a559-4d10-9adc-a91eb3fde737
[2017-06-15 17:01:35,755: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 62a3afb1-a559-4d10-9adc-a91eb3fde737
[2017-06-15 17:01:39,025: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:01:39,025: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:01:39,025: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:01:39,025: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:01:39,025: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 79bec816-7728-4d2d-8cb7-c42f8c698580
[2017-06-15 17:01:39,025: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 79bec816-7728-4d2d-8cb7-c42f8c698580
[2017-06-15 17:01:39,032: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 79bec816-7728-4d2d-8cb7-c42f8c698580
[2017-06-15 17:01:39,032: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 79bec816-7728-4d2d-8cb7-c42f8c698580
[2017-06-15 17:01:39,032: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.226', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-06-15 17:01:39,050: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9f3b41b1-7581-4ea6-965a-ea6de925105e
[2017-06-15 17:01:39,050: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9f3b41b1-7581-4ea6-965a-ea6de925105e
[2017-06-15 17:01:39,151: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:01:39,151: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:01:39,151: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:01:39,151: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:01:39,151: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9f3b41b1-7581-4ea6-965a-ea6de925105e
[2017-06-15 17:01:39,151: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9f3b41b1-7581-4ea6-965a-ea6de925105e
[2017-06-15 17:01:39,164: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9f3b41b1-7581-4ea6-965a-ea6de925105e
[2017-06-15 17:01:39,164: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9f3b41b1-7581-4ea6-965a-ea6de925105e
[2017-06-15 17:01:48,913: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:01:48,913: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:01:48,914: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:01:48,914: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:01:48,914: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 62a3afb1-a559-4d10-9adc-a91eb3fde737
[2017-06-15 17:01:48,914: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 62a3afb1-a559-4d10-9adc-a91eb3fde737
[2017-06-15 17:01:48,941: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 62a3afb1-a559-4d10-9adc-a91eb3fde737
[2017-06-15 17:01:48,941: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 62a3afb1-a559-4d10-9adc-a91eb3fde737
[2017-06-15 17:01:48,941: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': True, 'mcast_port': 52617, 'new_node_fqdn': u'lotus-32vm8.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'Ln41rg0FyONifYwcTRRY'}
[2017-06-15 17:01:48,959: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c81a243d-262f-442c-b62f-bb7bd281d49c
[2017-06-15 17:01:48,959: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c81a243d-262f-442c-b62f-bb7bd281d49c
[2017-06-15 17:01:57,023: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:01:57,023: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:01:57,023: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:01:57,023: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:01:57,023: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c81a243d-262f-442c-b62f-bb7bd281d49c
[2017-06-15 17:01:57,023: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c81a243d-262f-442c-b62f-bb7bd281d49c
[2017-06-15 17:01:57,028: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c81a243d-262f-442c-b62f-bb7bd281d49c
[2017-06-15 17:01:57,028: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c81a243d-262f-442c-b62f-bb7bd281d49c
[2017-06-15 17:01:57,029: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-06-15 17:01:57.029110+00:00:
[2017-06-15 17:01:57,029: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-06-15 17:01:57.029110+00:00:
[2017-06-15 17:01:57,029: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->52617
[2017-06-15 17:01:57,029: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->52617
[2017-06-15 17:01:57,035: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-06-15 17:01:57,035: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-06-15 17:01:57,062: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:01:57,062: INFO/job_scheduler]   Job 266 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-06-15 17:01:57,062: INFO/job_scheduler]   Job 271 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-06-15 17:01:57,063: INFO/job_scheduler]   Job 272 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:01:57,063: INFO/job_scheduler]   Job 274 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:01:57,063: INFO/job_scheduler] run_next: 0 runnable jobs of (28 pending, 4 tasked)
[2017-06-15 17:01:57,065: INFO/job_scheduler] Job 266 finished 1 steps successfully
[2017-06-15 17:01:57,066: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 52617, 'pcs_password': 'Yg3ygNBEXfOTEFZ7SDeZ'}
[2017-06-15 17:01:57,079: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cd025f01-eadb-4f09-8843-ba63b578bd63
[2017-06-15 17:01:57,079: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cd025f01-eadb-4f09-8843-ba63b578bd63
[2017-06-15 17:01:57,287: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 140623739749904
[2017-06-15 17:01:57,294: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-06-15 17:01:57,324: INFO/job] Job 266: StateChangeJob complete, setting state stopped on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:01:57,324: INFO/job] Job 266: not_deleted=True
[2017-06-15 17:01:57,324: INFO/job_scheduler] Job 266: complete_job: Updating cache
[2017-06-15 17:01:57,325: INFO/job_scheduler] Job 266: locked_item 140623739749904 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-15 17:01:57,329: INFO/job_scheduler] Job 266 completing (errored=False, cancelled=False)
[2017-06-15 17:01:57,451: INFO/job_scheduler] run_next: 1 runnable jobs of (28 pending, 3 tasked)
[2017-06-15 17:01:57,460: INFO/job_scheduler] Job 271: RunJobThread.run
[2017-06-15 17:01:57,460: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-06-15 17:01:57,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5e327d5b-9eec-4573-8484-edc4f9db977d
[2017-06-15 17:01:57,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5e327d5b-9eec-4573-8484-edc4f9db977d
[2017-06-15 17:02:02,691: INFO/job_scheduler] run_next: 0 runnable jobs of (27 pending, 4 tasked)
[2017-06-15 17:02:02,708: INFO/job_scheduler] run_next: 0 runnable jobs of (27 pending, 4 tasked)
[2017-06-15 17:02:05,585: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:02:05,585: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:02:05,585: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:02:05,585: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:02:05,585: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5e327d5b-9eec-4573-8484-edc4f9db977d
[2017-06-15 17:02:05,585: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5e327d5b-9eec-4573-8484-edc4f9db977d
[2017-06-15 17:02:05,620: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5e327d5b-9eec-4573-8484-edc4f9db977d
[2017-06-15 17:02:05,620: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5e327d5b-9eec-4573-8484-edc4f9db977d
[2017-06-15 17:02:05,620: INFO/job_scheduler] Job 271 finished 1 steps successfully
[2017-06-15 17:02:05,675: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 140623739749904
[2017-06-15 17:02:05,734: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm8.lotus.hpdd.lab.intel.com at severity 30
[2017-06-15 17:02:05,764: INFO/job] Job 271: StateChangeJob complete, setting state started on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:02:05,764: INFO/job] Job 271: not_deleted=True
[2017-06-15 17:02:05,764: INFO/job_scheduler] Job 271: complete_job: Updating cache
[2017-06-15 17:02:05,764: INFO/job_scheduler] Job 271: locked_item 140623739749904 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-15 17:02:05,768: INFO/job_scheduler] Job 271 completing (errored=False, cancelled=False)
[2017-06-15 17:02:05,923: INFO/job_scheduler] run_next: 1 runnable jobs of (27 pending, 3 tasked)
[2017-06-15 17:02:05,963: INFO/job_scheduler] Job 272: RunJobThread.run
[2017-06-15 17:02:05,964: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-06-15 17:02:06,027: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc af976418-6eaf-4b60-a9ee-a1698bea8c79
[2017-06-15 17:02:06,027: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc af976418-6eaf-4b60-a9ee-a1698bea8c79
[2017-06-15 17:02:06,326: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:02:06,326: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:02:06,326: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:02:06,326: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:02:06,326: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4586ef08-5288-4183-9e11-1e8735cf9a08
[2017-06-15 17:02:06,326: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4586ef08-5288-4183-9e11-1e8735cf9a08
[2017-06-15 17:02:06,335: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4586ef08-5288-4183-9e11-1e8735cf9a08
[2017-06-15 17:02:06,335: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4586ef08-5288-4183-9e11-1e8735cf9a08
[2017-06-15 17:02:06,446: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com kernel_status {}
[2017-06-15 17:02:06,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 854747cb-8e79-4c27-a47a-0f4c457e0b75
[2017-06-15 17:02:06,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 854747cb-8e79-4c27-a47a-0f4c457e0b75
[2017-06-15 17:02:08,742: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:02:08,742: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:02:08,742: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:02:08,742: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:02:08,742: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 854747cb-8e79-4c27-a47a-0f4c457e0b75
[2017-06-15 17:02:08,742: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 854747cb-8e79-4c27-a47a-0f4c457e0b75
[2017-06-15 17:02:08,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 854747cb-8e79-4c27-a47a-0f4c457e0b75
[2017-06-15 17:02:08,781: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 854747cb-8e79-4c27-a47a-0f4c457e0b75
[2017-06-15 17:02:08,781: INFO/job_scheduler] Job 243 finished 4 steps successfully
[2017-06-15 17:02:08,835: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 140623739923152
[2017-06-15 17:02:08,840: INFO/job] Job 243: StateChangeJob complete, setting state packages_installed on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 17:02:08,840: INFO/job] Job 243: not_deleted=True
[2017-06-15 17:02:08,840: INFO/job_scheduler] Job 243: complete_job: Updating cache
[2017-06-15 17:02:08,840: INFO/job_scheduler] Job 243: locked_item 140623740090832 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-15 17:02:08,843: INFO/job_scheduler] Job 243 completing (errored=False, cancelled=False)
[2017-06-15 17:02:08,859: INFO/job_scheduler] run_next: 4 runnable jobs of (26 pending, 3 tasked)
[2017-06-15 17:02:08,883: INFO/job_scheduler] Job 244: open DB connection during completion
[2017-06-15 17:02:08,895: INFO/job_scheduler] Job 245: RunJobThread.run
[2017-06-15 17:02:08,896: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-06-15 17:02:08,916: INFO/job_scheduler] Job 246: RunJobThread.run
[2017-06-15 17:02:08,920: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-06-15 17:02:08,925: INFO/job_scheduler] Job 247: RunJobThread.run
[2017-06-15 17:02:08,926: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-06-15 17:02:08,926: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 140623739734032
[2017-06-15 17:02:08,933: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-06-15 17:02:08,975: INFO/job] Job 244: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-15 17:02:08,976: INFO/job] Job 244: not_deleted=True
[2017-06-15 17:02:08,976: INFO/job_scheduler] Job 244: complete_job: Updating cache
[2017-06-15 17:02:08,976: INFO/job_scheduler] Job 244: locked_item 140623739734032 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-15 17:02:08,979: INFO/job_scheduler] Job 244 completing (errored=False, cancelled=False)
[2017-06-15 17:02:08,982: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0ad9392c-9cb8-436f-88ec-6f7c7c31bb8f
[2017-06-15 17:02:08,982: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0ad9392c-9cb8-436f-88ec-6f7c7c31bb8f
[2017-06-15 17:02:09,005: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 076722e2-b051-4642-85a0-789518a5a4b9
[2017-06-15 17:02:09,005: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 076722e2-b051-4642-85a0-789518a5a4b9
[2017-06-15 17:02:09,023: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5dac384f-53af-49c0-be51-2a0891bce939
[2017-06-15 17:02:09,023: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5dac384f-53af-49c0-be51-2a0891bce939
[2017-06-15 17:02:09,047: INFO/job_scheduler] run_next: 1 runnable jobs of (22 pending, 6 tasked)
[2017-06-15 17:02:09,102: INFO/job_scheduler] Job 248: RunJobThread.run
[2017-06-15 17:02:09,103: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com load_lnet {}
[2017-06-15 17:02:09,150: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e7f47983-5fad-480e-b864-2de3fd372bc0
[2017-06-15 17:02:09,150: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e7f47983-5fad-480e-b864-2de3fd372bc0
[2017-06-15 17:02:10,214: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:02:10,214: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:02:10,215: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:02:10,215: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:02:10,215: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e7f47983-5fad-480e-b864-2de3fd372bc0
[2017-06-15 17:02:10,215: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e7f47983-5fad-480e-b864-2de3fd372bc0
[2017-06-15 17:02:10,225: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e7f47983-5fad-480e-b864-2de3fd372bc0
[2017-06-15 17:02:10,225: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e7f47983-5fad-480e-b864-2de3fd372bc0
[2017-06-15 17:02:10,226: INFO/job] Job 248 LoadLNetStep: Error (1) running 'modprobe lnet': '' 'modprobe: ERROR: could not insert 'lnet': Invalid argument
'
[2017-06-15 17:02:10,226: ERROR/job_scheduler] Job 248 step 0 encountered an agent error: Error (1) running 'modprobe lnet': '' 'modprobe: ERROR: could not insert 'lnet': Invalid argument
'
[2017-06-15 17:02:10,272: INFO/job_scheduler] Job 248: complete_job: Updating cache
[2017-06-15 17:02:10,272: INFO/job_scheduler] Job 248: locked_item 140623739734032 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-15 17:02:10,275: INFO/job_scheduler] Job 248 completing (errored=True, cancelled=False)
[2017-06-15 17:02:10,300: INFO/job_scheduler] run_next: 1 runnable jobs of (21 pending, 6 tasked)
[2017-06-15 17:02:10,305: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration in state lnet_unloaded, not one of ['lnet_down'] (preferred lnet_down)
[2017-06-15 17:02:10,311: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration in state lnet_unloaded, not one of ['lnet_down'] (preferred lnet_down)
[2017-06-15 17:02:10,311: WARNING/job_scheduler] Job 250: cancelling because of failed dependency
[2017-06-15 17:02:10,311: INFO/job_scheduler] Job 250 completing (errored=False, cancelled=True)
[2017-06-15 17:02:10,325: INFO/job_scheduler] run_next: 0 runnable jobs of (20 pending, 6 tasked)
[2017-06-15 17:02:12,718: INFO/job_scheduler] run_next: 0 runnable jobs of (20 pending, 6 tasked)
[2017-06-15 17:02:12,734: INFO/job_scheduler] run_next: 0 runnable jobs of (20 pending, 6 tasked)
[2017-06-15 17:02:13,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:02:13,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:02:13,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:02:13,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:02:13,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc af976418-6eaf-4b60-a9ee-a1698bea8c79
[2017-06-15 17:02:13,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc af976418-6eaf-4b60-a9ee-a1698bea8c79
[2017-06-15 17:02:13,099: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc af976418-6eaf-4b60-a9ee-a1698bea8c79
[2017-06-15 17:02:13,099: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc af976418-6eaf-4b60-a9ee-a1698bea8c79
[2017-06-15 17:02:13,100: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-06-15 17:02:13,116: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ac497b82-584d-4d3f-a3ad-650e14caef1c
[2017-06-15 17:02:13,116: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ac497b82-584d-4d3f-a3ad-650e14caef1c
[2017-06-15 17:02:14,162: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:02:14,162: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:02:14,162: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:02:14,162: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:02:14,162: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 076722e2-b051-4642-85a0-789518a5a4b9
[2017-06-15 17:02:14,162: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 076722e2-b051-4642-85a0-789518a5a4b9
[2017-06-15 17:02:14,182: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 076722e2-b051-4642-85a0-789518a5a4b9
[2017-06-15 17:02:14,182: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 076722e2-b051-4642-85a0-789518a5a4b9
[2017-06-15 17:02:14,183: INFO/job_scheduler] Job 246 finished 1 steps successfully
[2017-06-15 17:02:14,234: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:02:14,234: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:02:14,234: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:02:14,234: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:02:14,234: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cd025f01-eadb-4f09-8843-ba63b578bd63
[2017-06-15 17:02:14,234: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cd025f01-eadb-4f09-8843-ba63b578bd63
[2017-06-15 17:02:14,263: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cd025f01-eadb-4f09-8843-ba63b578bd63
[2017-06-15 17:02:14,263: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cd025f01-eadb-4f09-8843-ba63b578bd63
[2017-06-15 17:02:14,263: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': False, 'mcast_port': 52617, 'new_node_fqdn': u'lotus-32vm7.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'Yg3ygNBEXfOTEFZ7SDeZ'}
[2017-06-15 17:02:14,280: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8ebbf7a0-8291-4473-9fe5-cc6fb33fdcfd
[2017-06-15 17:02:14,280: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8ebbf7a0-8291-4473-9fe5-cc6fb33fdcfd
[2017-06-15 17:02:14,295: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 140623739921488
[2017-06-15 17:02:14,300: INFO/job] Job 246: StateChangeJob complete, setting state configured on lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration
[2017-06-15 17:02:14,300: INFO/job] Job 246: not_deleted=True
[2017-06-15 17:02:14,300: INFO/job_scheduler] Job 246: complete_job: Updating cache
[2017-06-15 17:02:14,300: INFO/job_scheduler] Job 246: locked_item 140623739921488 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-06-15 17:02:14,303: INFO/job_scheduler] Job 246 completing (errored=False, cancelled=False)
[2017-06-15 17:02:14,321: INFO/job_scheduler] run_next: 0 runnable jobs of (20 pending, 5 tasked)
[2017-06-15 17:02:16,165: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:02:16,165: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:02:16,165: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:02:16,165: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:02:16,165: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5dac384f-53af-49c0-be51-2a0891bce939
[2017-06-15 17:02:16,165: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5dac384f-53af-49c0-be51-2a0891bce939
[2017-06-15 17:02:16,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5dac384f-53af-49c0-be51-2a0891bce939
[2017-06-15 17:02:16,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5dac384f-53af-49c0-be51-2a0891bce939
[2017-06-15 17:02:16,199: INFO/job_scheduler] Job 247 finished 1 steps successfully
[2017-06-15 17:02:16,279: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 140623739921232
[2017-06-15 17:02:16,282: INFO/job] Job 247: StateChangeJob complete, setting state configured on lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-06-15 17:02:16,282: INFO/job] Job 247: not_deleted=True
[2017-06-15 17:02:16,282: INFO/job_scheduler] Job 247: complete_job: Updating cache
[2017-06-15 17:02:16,283: INFO/job_scheduler] Job 247: locked_item 140623739921232 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-06-15 17:02:16,286: INFO/job_scheduler] Job 247 completing (errored=False, cancelled=False)
[2017-06-15 17:02:16,312: INFO/job_scheduler] run_next: 0 runnable jobs of (20 pending, 4 tasked)
[2017-06-15 17:02:20,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:02:20,399: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:02:20,400: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:02:20,400: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:02:20,400: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8ebbf7a0-8291-4473-9fe5-cc6fb33fdcfd
[2017-06-15 17:02:20,400: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8ebbf7a0-8291-4473-9fe5-cc6fb33fdcfd
[2017-06-15 17:02:20,400: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8ebbf7a0-8291-4473-9fe5-cc6fb33fdcfd
[2017-06-15 17:02:20,400: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8ebbf7a0-8291-4473-9fe5-cc6fb33fdcfd
[2017-06-15 17:02:20,401: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-06-15 17:02:20.400992+00:00:
[2017-06-15 17:02:20,401: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-06-15 17:02:20.400992+00:00:
[2017-06-15 17:02:20,401: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->52617
[2017-06-15 17:02:20,401: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->52617
[2017-06-15 17:02:20,403: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-06-15 17:02:20,403: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-06-15 17:02:20,415: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:02:20,416: INFO/job_scheduler]   Job 256 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-06-15 17:02:20,416: INFO/job_scheduler]   Job 259 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-06-15 17:02:20,416: INFO/job_scheduler]   Job 262 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:02:20,416: INFO/job_scheduler]   Job 263 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:02:20,416: INFO/job_scheduler] run_next: 0 runnable jobs of (20 pending, 4 tasked)
[2017-06-15 17:02:20,418: INFO/job_scheduler] Job 256 finished 1 steps successfully
[2017-06-15 17:02:20,461: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 140623739811344
[2017-06-15 17:02:20,468: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-06-15 17:02:20,492: INFO/job] Job 256: StateChangeJob complete, setting state stopped on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:02:20,493: INFO/job] Job 256: not_deleted=True
[2017-06-15 17:02:20,493: INFO/job_scheduler] Job 256: complete_job: Updating cache
[2017-06-15 17:02:20,493: INFO/job_scheduler] Job 256: locked_item 140623739811344 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-15 17:02:20,497: INFO/job_scheduler] Job 256 completing (errored=False, cancelled=False)
[2017-06-15 17:02:20,531: INFO/job_scheduler] run_next: 1 runnable jobs of (20 pending, 3 tasked)
[2017-06-15 17:02:20,560: INFO/job_scheduler] Job 259: RunJobThread.run
[2017-06-15 17:02:20,560: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-06-15 17:02:20,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e8859647-85d8-4be3-937d-9d93da036c5f
[2017-06-15 17:02:20,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e8859647-85d8-4be3-937d-9d93da036c5f
[2017-06-15 17:02:22,754: INFO/job_scheduler] run_next: 0 runnable jobs of (19 pending, 4 tasked)
[2017-06-15 17:02:22,802: INFO/job_scheduler] run_next: 0 runnable jobs of (19 pending, 4 tasked)
[2017-06-15 17:02:28,696: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:02:28,696: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:02:28,697: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:02:28,697: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:02:28,697: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e8859647-85d8-4be3-937d-9d93da036c5f
[2017-06-15 17:02:28,697: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e8859647-85d8-4be3-937d-9d93da036c5f
[2017-06-15 17:02:28,728: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e8859647-85d8-4be3-937d-9d93da036c5f
[2017-06-15 17:02:28,728: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e8859647-85d8-4be3-937d-9d93da036c5f
[2017-06-15 17:02:28,728: INFO/job_scheduler] Job 259 finished 1 steps successfully
[2017-06-15 17:02:28,810: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 140623739811344
[2017-06-15 17:02:28,930: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 30
[2017-06-15 17:02:28,942: INFO/job] Job 259: StateChangeJob complete, setting state started on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:02:28,942: INFO/job] Job 259: not_deleted=True
[2017-06-15 17:02:28,943: INFO/job_scheduler] Job 259: complete_job: Updating cache
[2017-06-15 17:02:28,944: INFO/job_scheduler] Job 259: locked_item 140623739811344 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-15 17:02:28,952: INFO/job_scheduler] Job 259 completing (errored=False, cancelled=False)
[2017-06-15 17:02:29,094: INFO/job_scheduler] run_next: 1 runnable jobs of (19 pending, 3 tasked)
[2017-06-15 17:02:29,108: INFO/job_scheduler] Job 262: RunJobThread.run
[2017-06-15 17:02:29,109: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-06-15 17:02:29,133: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 66772173-fc44-4c43-8816-f0eeeba57211
[2017-06-15 17:02:29,133: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 66772173-fc44-4c43-8816-f0eeeba57211
[2017-06-15 17:02:29,782: INFO/job_scheduler] run_next: 0 runnable jobs of (18 pending, 4 tasked)
[2017-06-15 17:02:29,813: INFO/job_scheduler] run_next: 0 runnable jobs of (18 pending, 4 tasked)
[2017-06-15 17:02:32,762: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:02:32,762: INFO/job_scheduler]   Job 262 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:02:32,763: INFO/job_scheduler]   Job 263 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:02:32,763: INFO/job_scheduler] run_next: 0 runnable jobs of (18 pending, 4 tasked)
[2017-06-15 17:02:32,775: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:02:32,775: INFO/job_scheduler]   Job 262 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 17:02:32,775: INFO/job_scheduler]   Job 264 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:02:32,776: INFO/job_scheduler] run_next: 0 runnable jobs of (18 pending, 4 tasked)
[2017-06-15 17:02:32,953: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:02:32,954: INFO/job_scheduler]   Job 272 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:02:32,954: INFO/job_scheduler]   Job 274 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:02:32,954: INFO/job_scheduler] run_next: 0 runnable jobs of (18 pending, 4 tasked)
[2017-06-15 17:02:32,981: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:02:32,981: INFO/job_scheduler]   Job 272 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 17:02:32,981: INFO/job_scheduler]   Job 275 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:02:32,982: INFO/job_scheduler] run_next: 0 runnable jobs of (18 pending, 4 tasked)
[2017-06-15 17:02:36,206: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:02:36,206: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:02:36,206: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:02:36,206: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:02:36,207: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 66772173-fc44-4c43-8816-f0eeeba57211
[2017-06-15 17:02:36,207: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 66772173-fc44-4c43-8816-f0eeeba57211
[2017-06-15 17:02:36,253: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 66772173-fc44-4c43-8816-f0eeeba57211
[2017-06-15 17:02:36,253: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 66772173-fc44-4c43-8816-f0eeeba57211
[2017-06-15 17:02:36,255: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-06-15 17:02:36,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc db255ea5-0b33-4e01-af91-82613683e6ad
[2017-06-15 17:02:36,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc db255ea5-0b33-4e01-af91-82613683e6ad
[2017-06-15 17:02:39,811: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:02:39,811: INFO/job_scheduler]   Job 262 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 17:02:39,811: INFO/job_scheduler]   Job 264 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:02:39,812: INFO/job_scheduler] run_next: 0 runnable jobs of (18 pending, 4 tasked)
[2017-06-15 17:02:39,831: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:02:39,831: INFO/job_scheduler]   Job 272 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 17:02:39,831: INFO/job_scheduler]   Job 275 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:02:39,831: INFO/job_scheduler] run_next: 0 runnable jobs of (18 pending, 4 tasked)
[2017-06-15 17:02:42,768: INFO/job_scheduler] run_next: 0 runnable jobs of (18 pending, 4 tasked)
[2017-06-15 17:02:42,782: INFO/job_scheduler] run_next: 0 runnable jobs of (18 pending, 4 tasked)
[2017-06-15 17:02:42,809: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:02:42,809: INFO/job_scheduler]   Job 262 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 17:02:42,809: INFO/job_scheduler]   Job 264 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:02:42,810: INFO/job_scheduler] run_next: 0 runnable jobs of (18 pending, 4 tasked)
[2017-06-15 17:02:42,829: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:02:42,830: INFO/job_scheduler]   Job 272 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 17:02:42,830: INFO/job_scheduler]   Job 275 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:02:42,830: INFO/job_scheduler] run_next: 0 runnable jobs of (18 pending, 4 tasked)
[2017-06-15 17:02:44,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 17:02:44,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/d0869604-e83e-4ab3-abcd-b66079073e07
[2017-06-15 17:02:44,085: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session d0869604-e83e-4ab3-abcd-b66079073e07 because session terminated
[2017-06-15 17:02:44,085: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session d0869604-e83e-4ab3-abcd-b66079073e07 because session terminated
[2017-06-15 17:02:49,858: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:02:49,858: INFO/job_scheduler]   Job 262 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 17:02:49,858: INFO/job_scheduler]   Job 264 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:02:49,859: INFO/job_scheduler] run_next: 0 runnable jobs of (18 pending, 4 tasked)
[2017-06-15 17:02:49,903: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:02:49,903: INFO/job_scheduler]   Job 272 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 17:02:49,903: INFO/job_scheduler]   Job 275 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:02:49,904: INFO/job_scheduler] run_next: 0 runnable jobs of (18 pending, 4 tasked)
[2017-06-15 17:02:52,839: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:02:52,839: INFO/job_scheduler]   Job 262 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 17:02:52,839: INFO/job_scheduler]   Job 264 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:02:52,839: INFO/job_scheduler] run_next: 0 runnable jobs of (18 pending, 4 tasked)
[2017-06-15 17:02:52,861: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:02:52,861: INFO/job_scheduler]   Job 272 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 17:02:52,861: INFO/job_scheduler]   Job 275 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:02:52,861: INFO/job_scheduler] run_next: 0 runnable jobs of (18 pending, 4 tasked)
[2017-06-15 17:02:56,276: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:02:56,276: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:02:56,277: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:02:56,277: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:02:56,277: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0ad9392c-9cb8-436f-88ec-6f7c7c31bb8f
[2017-06-15 17:02:56,277: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0ad9392c-9cb8-436f-88ec-6f7c7c31bb8f
[2017-06-15 17:02:56,295: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0ad9392c-9cb8-436f-88ec-6f7c7c31bb8f
[2017-06-15 17:02:56,295: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0ad9392c-9cb8-436f-88ec-6f7c7c31bb8f
[2017-06-15 17:02:56,296: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.225', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-06-15 17:02:56,309: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8b207dec-40ac-4376-900e-92b6304635db
[2017-06-15 17:02:56,309: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8b207dec-40ac-4376-900e-92b6304635db
[2017-06-15 17:02:56,348: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:02:56,348: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:02:56,349: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:02:56,349: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:02:56,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8b207dec-40ac-4376-900e-92b6304635db
[2017-06-15 17:02:56,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8b207dec-40ac-4376-900e-92b6304635db
[2017-06-15 17:02:56,375: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8b207dec-40ac-4376-900e-92b6304635db
[2017-06-15 17:02:56,375: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8b207dec-40ac-4376-900e-92b6304635db
[2017-06-15 17:02:56,395: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 62665, 'pcs_password': 'Bx0iVatrrxm0qYcrWRI8'}
[2017-06-15 17:02:56,408: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bc0345c5-d9e9-4e94-a4ca-182615854112
[2017-06-15 17:02:56,408: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bc0345c5-d9e9-4e94-a4ca-182615854112
[2017-06-15 17:02:59,852: INFO/job_scheduler] run_next: 0 runnable jobs of (18 pending, 4 tasked)
[2017-06-15 17:02:59,868: INFO/job_scheduler] run_next: 0 runnable jobs of (18 pending, 4 tasked)
[2017-06-15 17:03:02,191: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:03:02,191: INFO/job_scheduler]   Job 232 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-15 17:03:02,192: INFO/job_scheduler]   Job 233 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 17:03:02,192: INFO/job_scheduler]   Job 234 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 17:03:02,192: INFO/job_scheduler]   Job 235 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 17:03:02,192: INFO/job_scheduler]   Job 236 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 17:03:02,192: INFO/job_scheduler]   Job 240 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 17:03:02,192: INFO/job_scheduler]   Job 242 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:03:02,193: INFO/job_scheduler] run_next: 0 runnable jobs of (18 pending, 4 tasked)
[2017-06-15 17:03:02,267: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:02,267: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:02,509: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:02,509: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:02,509: INFO/job_scheduler] Job 232 finished 4 steps successfully
[2017-06-15 17:03:02,560: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 140623538066576
[2017-06-15 17:03:02,564: INFO/job] Job 232: StateChangeJob complete, setting state packages_installed on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 17:03:02,564: INFO/job] Job 232: not_deleted=True
[2017-06-15 17:03:02,565: INFO/job_scheduler] Job 232: complete_job: Updating cache
[2017-06-15 17:03:02,565: INFO/job_scheduler] Job 232: locked_item 140623538052624 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-15 17:03:02,567: INFO/job_scheduler] Job 232 completing (errored=False, cancelled=False)
[2017-06-15 17:03:02,610: INFO/job_scheduler] run_next: 4 runnable jobs of (18 pending, 3 tasked)
[2017-06-15 17:03:02,692: INFO/job_scheduler] Job 233: RunJobThread.run
[2017-06-15 17:03:02,693: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-06-15 17:03:02,701: INFO/job_scheduler] Job 234: open DB connection during completion
[2017-06-15 17:03:02,718: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e2a73b5f-8540-402d-8439-2b145afb2f7d
[2017-06-15 17:03:02,718: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e2a73b5f-8540-402d-8439-2b145afb2f7d
[2017-06-15 17:03:02,865: INFO/job_scheduler] Job 235: RunJobThread.run
[2017-06-15 17:03:02,869: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-06-15 17:03:02,897: INFO/job_scheduler] Job 236: RunJobThread.run
[2017-06-15 17:03:02,898: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-06-15 17:03:02,906: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 389f20dd-dcd8-4164-b77c-ac4496c4cba6
[2017-06-15 17:03:02,906: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 389f20dd-dcd8-4164-b77c-ac4496c4cba6
[2017-06-15 17:03:02,934: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cae2ee27-4544-4df0-888a-345dc5abed11
[2017-06-15 17:03:02,934: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cae2ee27-4544-4df0-888a-345dc5abed11
[2017-06-15 17:03:02,957: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 140623740246672
[2017-06-15 17:03:02,964: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-06-15 17:03:02,970: INFO/job] Job 234: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-15 17:03:02,970: INFO/job] Job 234: not_deleted=True
[2017-06-15 17:03:02,970: INFO/job_scheduler] Job 234: complete_job: Updating cache
[2017-06-15 17:03:02,970: INFO/job_scheduler] Job 234: locked_item 140623740246672 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-15 17:03:02,976: INFO/job_scheduler] Job 234 completing (errored=False, cancelled=False)
[2017-06-15 17:03:03,098: INFO/job_scheduler] run_next: 1 runnable jobs of (14 pending, 6 tasked)
[2017-06-15 17:03:03,138: INFO/job_scheduler] Job 238: RunJobThread.run
[2017-06-15 17:03:03,139: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com load_lnet {}
[2017-06-15 17:03:03,220: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:03:03,220: INFO/job_scheduler]   Job 262 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:03:03,221: INFO/job_scheduler]   Job 263 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:03:03,221: INFO/job_scheduler] run_next: 0 runnable jobs of (13 pending, 7 tasked)
[2017-06-15 17:03:03,227: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:03:03,228: INFO/job_scheduler]   Job 262 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 17:03:03,228: INFO/job_scheduler]   Job 264 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:03:03,228: INFO/job_scheduler] run_next: 0 runnable jobs of (13 pending, 7 tasked)
[2017-06-15 17:03:03,230: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:03:03,230: INFO/job_scheduler]   Job 272 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:03:03,230: INFO/job_scheduler]   Job 274 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:03:03,231: INFO/job_scheduler] run_next: 0 runnable jobs of (13 pending, 7 tasked)
[2017-06-15 17:03:03,293: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:03:03,293: INFO/job_scheduler]   Job 272 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 17:03:03,294: INFO/job_scheduler]   Job 275 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:03:03,294: INFO/job_scheduler] run_next: 0 runnable jobs of (13 pending, 7 tasked)
[2017-06-15 17:03:03,300: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 23dad80d-e236-463a-9791-6b9a5b9b9160
[2017-06-15 17:03:03,300: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 23dad80d-e236-463a-9791-6b9a5b9b9160
[2017-06-15 17:03:03,386: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:03,386: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:03,387: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:03,387: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:03,387: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 23dad80d-e236-463a-9791-6b9a5b9b9160
[2017-06-15 17:03:03,387: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 23dad80d-e236-463a-9791-6b9a5b9b9160
[2017-06-15 17:03:03,429: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 23dad80d-e236-463a-9791-6b9a5b9b9160
[2017-06-15 17:03:03,429: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 23dad80d-e236-463a-9791-6b9a5b9b9160
[2017-06-15 17:03:03,430: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-15 17:03:03,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 86de45c4-6195-44f1-bdbd-98472c56e33d
[2017-06-15 17:03:03,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 86de45c4-6195-44f1-bdbd-98472c56e33d
[2017-06-15 17:03:03,764: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:03,764: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:03,765: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:03,765: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:03,765: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 86de45c4-6195-44f1-bdbd-98472c56e33d
[2017-06-15 17:03:03,765: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 86de45c4-6195-44f1-bdbd-98472c56e33d
[2017-06-15 17:03:03,783: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 86de45c4-6195-44f1-bdbd-98472c56e33d
[2017-06-15 17:03:03,783: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 86de45c4-6195-44f1-bdbd-98472c56e33d
[2017-06-15 17:03:04,192: INFO/job_scheduler] Job 238 finished 2 steps successfully
[2017-06-15 17:03:04,235: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 140623740246672
[2017-06-15 17:03:04,241: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-06-15 17:03:04,245: INFO/job] Job 238: StateChangeJob complete, setting state lnet_down on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-15 17:03:04,246: INFO/job] Job 238: not_deleted=True
[2017-06-15 17:03:04,246: INFO/job_scheduler] Job 238: complete_job: Updating cache
[2017-06-15 17:03:04,246: INFO/job_scheduler] Job 238: locked_item 140623740246672 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-15 17:03:04,259: INFO/job_scheduler] Job 238 completing (errored=False, cancelled=False)
[2017-06-15 17:03:04,297: INFO/job_scheduler] run_next: 1 runnable jobs of (13 pending, 6 tasked)
[2017-06-15 17:03:04,317: INFO/job_scheduler] Job 239: RunJobThread.run
[2017-06-15 17:03:04,318: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_lnet {}
[2017-06-15 17:03:04,369: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 74a58199-0402-493b-9b86-7aa1fe095b5a
[2017-06-15 17:03:04,369: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 74a58199-0402-493b-9b86-7aa1fe095b5a
[2017-06-15 17:03:05,842: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:05,842: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:05,842: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:05,842: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:05,842: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 74a58199-0402-493b-9b86-7aa1fe095b5a
[2017-06-15 17:03:05,842: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 74a58199-0402-493b-9b86-7aa1fe095b5a
[2017-06-15 17:03:05,842: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 74a58199-0402-493b-9b86-7aa1fe095b5a
[2017-06-15 17:03:05,842: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 74a58199-0402-493b-9b86-7aa1fe095b5a
[2017-06-15 17:03:05,843: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-15 17:03:05,881: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e3a48c28-9fa1-42e4-b338-75bfda4dbdc5
[2017-06-15 17:03:05,881: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e3a48c28-9fa1-42e4-b338-75bfda4dbdc5
[2017-06-15 17:03:05,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:05,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:05,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:05,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:05,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e3a48c28-9fa1-42e4-b338-75bfda4dbdc5
[2017-06-15 17:03:05,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e3a48c28-9fa1-42e4-b338-75bfda4dbdc5
[2017-06-15 17:03:05,923: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e3a48c28-9fa1-42e4-b338-75bfda4dbdc5
[2017-06-15 17:03:05,923: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e3a48c28-9fa1-42e4-b338-75bfda4dbdc5
[2017-06-15 17:03:06,101: INFO/job_scheduler] Job 239 finished 2 steps successfully
[2017-06-15 17:03:06,151: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 140623740246672
[2017-06-15 17:03:06,173: INFO/job] Job 239: StateChangeJob complete, setting state lnet_up on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-15 17:03:06,174: INFO/job] Job 239: not_deleted=True
[2017-06-15 17:03:06,176: INFO/job_scheduler] Job 239: complete_job: Updating cache
[2017-06-15 17:03:06,176: INFO/job_scheduler] Job 239: locked_item 140623740246672 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-15 17:03:06,185: INFO/job_scheduler] Job 239 completing (errored=False, cancelled=False)
[2017-06-15 17:03:06,221: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 6 tasked)
[2017-06-15 17:03:06,834: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:03:06,834: INFO/job_scheduler]   Job 233 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 17:03:06,834: INFO/job_scheduler]   Job 235 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 17:03:06,835: INFO/job_scheduler]   Job 236 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 17:03:06,835: INFO/job_scheduler]   Job 240 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 17:03:06,835: INFO/job_scheduler]   Job 242 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:03:06,835: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 6 tasked)
[2017-06-15 17:03:06,973: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:03:06,973: INFO/job_scheduler]   Job 233 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 17:03:06,973: INFO/job_scheduler]   Job 235 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 17:03:06,973: INFO/job_scheduler]   Job 236 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 17:03:06,973: INFO/job_scheduler]   Job 240 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 17:03:06,973: INFO/job_scheduler]   Job 242 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:03:06,974: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 6 tasked)
[2017-06-15 17:03:09,321: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:09,321: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:09,321: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:09,321: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:09,321: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e2a73b5f-8540-402d-8439-2b145afb2f7d
[2017-06-15 17:03:09,321: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e2a73b5f-8540-402d-8439-2b145afb2f7d
[2017-06-15 17:03:09,329: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:09,329: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:09,329: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:09,329: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:09,329: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 389f20dd-dcd8-4164-b77c-ac4496c4cba6
[2017-06-15 17:03:09,329: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 389f20dd-dcd8-4164-b77c-ac4496c4cba6
[2017-06-15 17:03:09,336: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 389f20dd-dcd8-4164-b77c-ac4496c4cba6
[2017-06-15 17:03:09,336: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 389f20dd-dcd8-4164-b77c-ac4496c4cba6
[2017-06-15 17:03:09,336: INFO/job_scheduler] Job 235 finished 1 steps successfully
[2017-06-15 17:03:09,349: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e2a73b5f-8540-402d-8439-2b145afb2f7d
[2017-06-15 17:03:09,349: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e2a73b5f-8540-402d-8439-2b145afb2f7d
[2017-06-15 17:03:09,350: INFO/job_scheduler] Job 233 finished 1 steps successfully
[2017-06-15 17:03:09,447: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 140623740246736
[2017-06-15 17:03:09,450: INFO/job] Job 235: StateChangeJob complete, setting state configured on lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration
[2017-06-15 17:03:09,450: INFO/job] Job 235: not_deleted=True
[2017-06-15 17:03:09,451: INFO/job_scheduler] Job 235: complete_job: Updating cache
[2017-06-15 17:03:09,451: INFO/job_scheduler] Job 235: locked_item 140623740246736 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-06-15 17:03:09,454: INFO/job_scheduler] Job 235 completing (errored=False, cancelled=False)
[2017-06-15 17:03:09,481: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 5 tasked)
[2017-06-15 17:03:09,505: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:03:09,505: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:03:09,506: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:03:09,506: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:03:09,506: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc db255ea5-0b33-4e01-af91-82613683e6ad
[2017-06-15 17:03:09,506: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc db255ea5-0b33-4e01-af91-82613683e6ad
[2017-06-15 17:03:09,535: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc db255ea5-0b33-4e01-af91-82613683e6ad
[2017-06-15 17:03:09,535: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc db255ea5-0b33-4e01-af91-82613683e6ad
[2017-06-15 17:03:09,537: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-06-15 17:03:09,584: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8975bfe7-8432-4f4e-be48-15feddc059b4
[2017-06-15 17:03:09,584: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8975bfe7-8432-4f4e-be48-15feddc059b4
[2017-06-15 17:03:09,585: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:03:09,585: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:03:09,585: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:03:09,585: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:03:09,585: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bc0345c5-d9e9-4e94-a4ca-182615854112
[2017-06-15 17:03:09,585: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bc0345c5-d9e9-4e94-a4ca-182615854112
[2017-06-15 17:03:09,596: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bc0345c5-d9e9-4e94-a4ca-182615854112
[2017-06-15 17:03:09,596: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bc0345c5-d9e9-4e94-a4ca-182615854112
[2017-06-15 17:03:09,597: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': True, 'mcast_port': 62665, 'new_node_fqdn': u'lotus-32vm6.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'Bx0iVatrrxm0qYcrWRI8'}
[2017-06-15 17:03:09,625: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bf0df779-ca89-4c26-ac33-7ca83ecc15fb
[2017-06-15 17:03:09,625: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bf0df779-ca89-4c26-ac33-7ca83ecc15fb
[2017-06-15 17:03:09,639: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 140623740275600
[2017-06-15 17:03:09,643: INFO/job] Job 233: StateChangeJob complete, setting state configured on lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-06-15 17:03:09,643: INFO/job] Job 233: not_deleted=True
[2017-06-15 17:03:09,643: INFO/job_scheduler] Job 233: complete_job: Updating cache
[2017-06-15 17:03:09,647: INFO/job_scheduler] Job 233: locked_item 140623740275600 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-06-15 17:03:09,649: INFO/job_scheduler] Job 233 completing (errored=False, cancelled=False)
[2017-06-15 17:03:09,698: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-06-15 17:03:09,989: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:03:09,990: INFO/job_scheduler]   Job 262 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:03:09,990: INFO/job_scheduler]   Job 263 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:03:09,990: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-06-15 17:03:10,023: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:03:10,023: INFO/job_scheduler]   Job 262 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 17:03:10,023: INFO/job_scheduler]   Job 264 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:03:10,023: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-06-15 17:03:10,099: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:03:10,099: INFO/job_scheduler]   Job 272 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:03:10,099: INFO/job_scheduler]   Job 274 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:03:10,100: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-06-15 17:03:10,128: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:03:10,128: INFO/job_scheduler]   Job 272 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 17:03:10,129: INFO/job_scheduler]   Job 275 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:03:10,129: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-06-15 17:03:12,876: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:03:12,876: INFO/job_scheduler]   Job 262 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:03:12,877: INFO/job_scheduler]   Job 263 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:03:12,877: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-06-15 17:03:12,891: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:03:12,891: INFO/job_scheduler]   Job 262 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-15 17:03:12,891: INFO/job_scheduler]   Job 264 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:03:12,891: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-06-15 17:03:12,909: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:03:12,909: INFO/job_scheduler]   Job 272 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:03:12,909: INFO/job_scheduler]   Job 274 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:03:12,910: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-06-15 17:03:12,926: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:03:12,926: INFO/job_scheduler]   Job 272 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 17:03:12,926: INFO/job_scheduler]   Job 275 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:03:12,926: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 4 tasked)
[2017-06-15 17:03:16,307: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:16,307: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:16,308: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:16,308: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:16,308: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cae2ee27-4544-4df0-888a-345dc5abed11
[2017-06-15 17:03:16,308: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cae2ee27-4544-4df0-888a-345dc5abed11
[2017-06-15 17:03:16,323: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cae2ee27-4544-4df0-888a-345dc5abed11
[2017-06-15 17:03:16,323: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cae2ee27-4544-4df0-888a-345dc5abed11
[2017-06-15 17:03:16,324: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.224', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-06-15 17:03:16,343: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7a347569-4ced-4538-bfc1-d85caff656e3
[2017-06-15 17:03:16,343: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7a347569-4ced-4538-bfc1-d85caff656e3
[2017-06-15 17:03:16,401: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:16,401: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:16,402: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:16,402: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:16,402: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7a347569-4ced-4538-bfc1-d85caff656e3
[2017-06-15 17:03:16,402: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7a347569-4ced-4538-bfc1-d85caff656e3
[2017-06-15 17:03:16,415: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7a347569-4ced-4538-bfc1-d85caff656e3
[2017-06-15 17:03:16,415: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7a347569-4ced-4538-bfc1-d85caff656e3
[2017-06-15 17:03:16,688: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:03:16,688: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:03:16,688: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:03:16,688: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:03:16,688: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8975bfe7-8432-4f4e-be48-15feddc059b4
[2017-06-15 17:03:16,688: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8975bfe7-8432-4f4e-be48-15feddc059b4
[2017-06-15 17:03:16,714: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8975bfe7-8432-4f4e-be48-15feddc059b4
[2017-06-15 17:03:16,714: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8975bfe7-8432-4f4e-be48-15feddc059b4
[2017-06-15 17:03:16,714: INFO/job_scheduler] Job 262 finished 3 steps successfully
[2017-06-15 17:03:16,751: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 140623739791312
[2017-06-15 17:03:16,756: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:03:16,756: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:03:16,756: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:03:16,756: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:03:16,756: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bf0df779-ca89-4c26-ac33-7ca83ecc15fb
[2017-06-15 17:03:16,756: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bf0df779-ca89-4c26-ac33-7ca83ecc15fb
[2017-06-15 17:03:16,762: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-06-15 17:03:16,766: INFO/job] Job 262: StateChangeJob complete, setting state stopped on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-15 17:03:16,767: INFO/job] Job 262: not_deleted=True
[2017-06-15 17:03:16,767: INFO/job_scheduler] Job 262: complete_job: Updating cache
[2017-06-15 17:03:16,767: INFO/job_scheduler] Job 262: locked_item 140623739791312 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-15 17:03:16,770: INFO/job_scheduler] Job 262 completing (errored=False, cancelled=False)
[2017-06-15 17:03:16,803: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bf0df779-ca89-4c26-ac33-7ca83ecc15fb
[2017-06-15 17:03:16,803: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bf0df779-ca89-4c26-ac33-7ca83ecc15fb
[2017-06-15 17:03:16,804: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-06-15 17:03:16.804074+00:00:
[2017-06-15 17:03:16,804: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-06-15 17:03:16.804074+00:00:
[2017-06-15 17:03:16,804: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->62665
[2017-06-15 17:03:16,804: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->62665
[2017-06-15 17:03:16,805: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-06-15 17:03:16,805: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-06-15 17:03:16,841: INFO/job_scheduler] Job 245 finished 1 steps successfully
[2017-06-15 17:03:16,845: INFO/job_scheduler] run_next: 1 runnable jobs of (12 pending, 3 tasked)
[2017-06-15 17:03:16,875: INFO/job_scheduler] Job 263: RunJobThread.run
[2017-06-15 17:03:16,876: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-06-15 17:03:16,897: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:03:16,900: INFO/job_scheduler]   Job 245 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-06-15 17:03:16,900: INFO/job_scheduler]   Job 249 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-06-15 17:03:16,900: INFO/job_scheduler]   Job 251 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:03:16,900: INFO/job_scheduler]   Job 252 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:03:16,901: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 4 tasked)
[2017-06-15 17:03:16,907: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 78e82840-8d19-4b23-bd47-34f606ae367a
[2017-06-15 17:03:16,907: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 78e82840-8d19-4b23-bd47-34f606ae367a
[2017-06-15 17:03:16,920: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 62665, 'pcs_password': 'C0gHxK0noB3vekqfMw1B'}
[2017-06-15 17:03:16,933: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 140623740083472
[2017-06-15 17:03:16,961: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-06-15 17:03:16,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9f35deaf-7858-496a-9eec-07fcef234c00
[2017-06-15 17:03:16,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9f35deaf-7858-496a-9eec-07fcef234c00
[2017-06-15 17:03:16,990: INFO/job] Job 245: StateChangeJob complete, setting state stopped on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:03:16,990: INFO/job] Job 245: not_deleted=True
[2017-06-15 17:03:16,991: INFO/job_scheduler] Job 245: complete_job: Updating cache
[2017-06-15 17:03:16,991: INFO/job_scheduler] Job 245: locked_item 140623740083472 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-15 17:03:17,021: INFO/job_scheduler] Job 245 completing (errored=False, cancelled=False)
[2017-06-15 17:03:17,115: INFO/job_scheduler] run_next: 1 runnable jobs of (11 pending, 3 tasked)
[2017-06-15 17:03:17,156: INFO/job_scheduler] Job 249: RunJobThread.run
[2017-06-15 17:03:17,257: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-06-15 17:03:17,289: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2934da88-8869-4add-a769-54f8074d2339
[2017-06-15 17:03:17,289: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2934da88-8869-4add-a769-54f8074d2339
[2017-06-15 17:03:22,414: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:03:22,414: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:03:22,415: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:03:22,415: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:03:22,415: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ac497b82-584d-4d3f-a3ad-650e14caef1c
[2017-06-15 17:03:22,415: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ac497b82-584d-4d3f-a3ad-650e14caef1c
[2017-06-15 17:03:22,436: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ac497b82-584d-4d3f-a3ad-650e14caef1c
[2017-06-15 17:03:22,436: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ac497b82-584d-4d3f-a3ad-650e14caef1c
[2017-06-15 17:03:22,438: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-06-15 17:03:22,456: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 094f92e1-55e2-4bc5-879f-dc6dcbf4143e
[2017-06-15 17:03:22,456: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 094f92e1-55e2-4bc5-879f-dc6dcbf4143e
[2017-06-15 17:03:22,925: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:03:22,925: INFO/job_scheduler]   Job 263 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:03:22,926: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 4 tasked)
[2017-06-15 17:03:22,941: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:03:22,941: INFO/job_scheduler]   Job 264 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:03:22,941: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 4 tasked)
[2017-06-15 17:03:22,961: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:03:22,962: INFO/job_scheduler]   Job 272 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:03:22,962: INFO/job_scheduler]   Job 274 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:03:22,962: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 4 tasked)
[2017-06-15 17:03:22,980: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:03:22,980: INFO/job_scheduler]   Job 272 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 17:03:22,980: INFO/job_scheduler]   Job 275 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:03:22,980: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 4 tasked)
[2017-06-15 17:03:24,023: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:03:24,023: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:03:24,024: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:03:24,024: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/f8302218-50aa-4da4-af28-4543ed4a1019
[2017-06-15 17:03:24,024: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 78e82840-8d19-4b23-bd47-34f606ae367a
[2017-06-15 17:03:24,024: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 78e82840-8d19-4b23-bd47-34f606ae367a
[2017-06-15 17:03:24,029: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 78e82840-8d19-4b23-bd47-34f606ae367a
[2017-06-15 17:03:24,029: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 78e82840-8d19-4b23-bd47-34f606ae367a
[2017-06-15 17:03:24,030: INFO/job_scheduler] Job 263 finished 1 steps successfully
[2017-06-15 17:03:24,097: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 140623739791312
[2017-06-15 17:03:24,135: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 30
[2017-06-15 17:03:24,142: INFO/job] Job 263: StateChangeJob complete, setting state started on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-15 17:03:24,142: INFO/job] Job 263: not_deleted=True
[2017-06-15 17:03:24,142: INFO/job_scheduler] Job 263: complete_job: Updating cache
[2017-06-15 17:03:24,142: INFO/job_scheduler] Job 263: locked_item 140623739791312 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-15 17:03:24,145: INFO/job_scheduler] Job 263 completing (errored=False, cancelled=False)
[2017-06-15 17:03:24,186: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-7
[2017-06-15 17:03:24,186: INFO/job_scheduler] _notify: Updating .mcast_port of item 7 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 52617
[2017-06-15 17:03:24,190: INFO/job_scheduler] _notify: Set mcast_port=52617 on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-7) and saved
[2017-06-15 17:03:24,192: INFO/job_scheduler] _notify: Updating .network_interfaces of item 7 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-06-15 17:03:24,215: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-7) and saved
[2017-06-15 17:03:24,221: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 7 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-06-15 17:03:24,233: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-7) and saved
[2017-06-15 17:03:24,251: INFO/job_scheduler] run_next: 1 runnable jobs of (10 pending, 3 tasked)
[2017-06-15 17:03:24,261: WARNING/job] DependOn not satisfied: lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration in state lnet_unloaded, not one of ['lnet_up'] (preferred lnet_up)
[2017-06-15 17:03:24,268: WARNING/job] DependOn not satisfied: lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration in state lnet_unloaded, not one of ['lnet_up'] (preferred lnet_up)
[2017-06-15 17:03:24,276: WARNING/job_scheduler] Job 264: cancelling because of failed dependency
[2017-06-15 17:03:24,276: INFO/job_scheduler] Job 264 completing (errored=False, cancelled=True)
[2017-06-15 17:03:24,374: INFO/job_scheduler] run_next: 0 runnable jobs of (9 pending, 3 tasked)
[2017-06-15 17:03:25,378: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:03:25,378: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:03:25,378: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:03:25,378: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:03:25,379: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2934da88-8869-4add-a769-54f8074d2339
[2017-06-15 17:03:25,379: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2934da88-8869-4add-a769-54f8074d2339
[2017-06-15 17:03:25,425: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2934da88-8869-4add-a769-54f8074d2339
[2017-06-15 17:03:25,425: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2934da88-8869-4add-a769-54f8074d2339
[2017-06-15 17:03:25,426: INFO/job_scheduler] Job 249 finished 1 steps successfully
[2017-06-15 17:03:25,480: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 140623740083472
[2017-06-15 17:03:25,537: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm6.lotus.hpdd.lab.intel.com at severity 30
[2017-06-15 17:03:25,586: INFO/job] Job 249: StateChangeJob complete, setting state started on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:03:25,586: INFO/job] Job 249: not_deleted=True
[2017-06-15 17:03:25,586: INFO/job_scheduler] Job 249: complete_job: Updating cache
[2017-06-15 17:03:25,587: INFO/job_scheduler] Job 249: locked_item 140623740083472 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-15 17:03:25,601: INFO/job_scheduler] Job 249 completing (errored=False, cancelled=False)
[2017-06-15 17:03:25,661: INFO/job_scheduler] Replaying 5 buffered notifications for ManagedHost-7
[2017-06-15 17:03:25,680: INFO/job_scheduler] _notify: Updating .immutable_state of item 7 (lotus-32vm7.lotus.hpdd.lab.intel.com) from True to False
[2017-06-15 17:03:25,695: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-7) and saved
[2017-06-15 17:03:25,738: INFO/job_scheduler] _notify: Updating .properties of item 7 (lotus-32vm7.lotus.hpdd.lab.intel.com) from {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": false, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-06-15 17:03:25,749: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-7) and saved
[2017-06-15 17:03:25,765: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 7 (lotus-32vm7.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7fe58806a310> to [8]
[2017-06-15 17:03:25,799: INFO/job_scheduler] _notify: Set ha_cluster_peers=[8] on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-7) and saved
[2017-06-15 17:03:25,943: INFO/job_scheduler] run_next: 1 runnable jobs of (9 pending, 2 tasked)
[2017-06-15 17:03:25,960: INFO/job_scheduler] Job 251: RunJobThread.run
[2017-06-15 17:03:25,961: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-06-15 17:03:25,991: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3d2da1b8-3c40-4d7a-8bad-0895804c789b
[2017-06-15 17:03:25,991: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3d2da1b8-3c40-4d7a-8bad-0895804c789b
[2017-06-15 17:03:28,409: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 3 tasked)
[2017-06-15 17:03:28,435: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 3 tasked)
[2017-06-15 17:03:29,958: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:03:29,958: INFO/job_scheduler]   Job 272 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:03:29,958: INFO/job_scheduler]   Job 274 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:03:29,959: INFO/job_scheduler] run_next: 0 runnable jobs of (8 pending, 3 tasked)
[2017-06-15 17:03:30,579: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:03:30,579: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:03:30,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:03:30,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:03:30,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 094f92e1-55e2-4bc5-879f-dc6dcbf4143e
[2017-06-15 17:03:30,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 094f92e1-55e2-4bc5-879f-dc6dcbf4143e
[2017-06-15 17:03:30,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 094f92e1-55e2-4bc5-879f-dc6dcbf4143e
[2017-06-15 17:03:30,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 094f92e1-55e2-4bc5-879f-dc6dcbf4143e
[2017-06-15 17:03:30,581: INFO/job_scheduler] Job 272 finished 3 steps successfully
[2017-06-15 17:03:30,662: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 140623740273872
[2017-06-15 17:03:30,669: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-06-15 17:03:30,672: INFO/job] Job 272: StateChangeJob complete, setting state stopped on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-15 17:03:30,672: INFO/job] Job 272: not_deleted=True
[2017-06-15 17:03:30,672: INFO/job_scheduler] Job 272: complete_job: Updating cache
[2017-06-15 17:03:30,672: INFO/job_scheduler] Job 272: locked_item 140623740273872 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-15 17:03:30,675: INFO/job_scheduler] Job 272 completing (errored=False, cancelled=False)
[2017-06-15 17:03:30,714: INFO/job_scheduler] run_next: 1 runnable jobs of (8 pending, 2 tasked)
[2017-06-15 17:03:30,789: INFO/job_scheduler] Job 274: RunJobThread.run
[2017-06-15 17:03:30,790: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-06-15 17:03:30,850: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 63b83902-6728-4f47-895e-30939781bb3b
[2017-06-15 17:03:30,850: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 63b83902-6728-4f47-895e-30939781bb3b
[2017-06-15 17:03:31,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:31,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:31,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:31,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:31,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9f35deaf-7858-496a-9eec-07fcef234c00
[2017-06-15 17:03:31,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9f35deaf-7858-496a-9eec-07fcef234c00
[2017-06-15 17:03:31,171: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9f35deaf-7858-496a-9eec-07fcef234c00
[2017-06-15 17:03:31,171: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9f35deaf-7858-496a-9eec-07fcef234c00
[2017-06-15 17:03:31,172: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': False, 'mcast_port': 62665, 'new_node_fqdn': u'lotus-32vm5.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'C0gHxK0noB3vekqfMw1B'}
[2017-06-15 17:03:31,186: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fcbcbc8b-86d2-4870-b3bd-59947634f987
[2017-06-15 17:03:31,186: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fcbcbc8b-86d2-4870-b3bd-59947634f987
[2017-06-15 17:03:33,094: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:03:33,094: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:03:33,094: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:03:33,094: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:03:33,094: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3d2da1b8-3c40-4d7a-8bad-0895804c789b
[2017-06-15 17:03:33,094: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3d2da1b8-3c40-4d7a-8bad-0895804c789b
[2017-06-15 17:03:33,122: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3d2da1b8-3c40-4d7a-8bad-0895804c789b
[2017-06-15 17:03:33,122: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3d2da1b8-3c40-4d7a-8bad-0895804c789b
[2017-06-15 17:03:33,123: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-06-15 17:03:33,151: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 553ce96a-b687-4476-9b90-85c93837bcc2
[2017-06-15 17:03:33,151: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 553ce96a-b687-4476-9b90-85c93837bcc2
[2017-06-15 17:03:37,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:03:37,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:03:37,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:03:37,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:03:37,283: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fcbcbc8b-86d2-4870-b3bd-59947634f987
[2017-06-15 17:03:37,283: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fcbcbc8b-86d2-4870-b3bd-59947634f987
[2017-06-15 17:03:37,316: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fcbcbc8b-86d2-4870-b3bd-59947634f987
[2017-06-15 17:03:37,316: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fcbcbc8b-86d2-4870-b3bd-59947634f987
[2017-06-15 17:03:37,317: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-06-15 17:03:37.317454+00:00:
[2017-06-15 17:03:37,317: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-06-15 17:03:37.317454+00:00:
[2017-06-15 17:03:37,317: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->62665
[2017-06-15 17:03:37,317: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->62665
[2017-06-15 17:03:37,319: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-06-15 17:03:37,319: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-06-15 17:03:37,335: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:03:37,336: INFO/job_scheduler]   Job 236 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-06-15 17:03:37,336: INFO/job_scheduler]   Job 237 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-06-15 17:03:37,336: INFO/job_scheduler]   Job 240 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:03:37,336: INFO/job_scheduler]   Job 241 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:03:37,336: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 3 tasked)
[2017-06-15 17:03:37,339: INFO/job_scheduler] Job 236 finished 1 steps successfully
[2017-06-15 17:03:37,379: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 140623739828304
[2017-06-15 17:03:37,389: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-06-15 17:03:37,397: INFO/job] Job 236: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:03:37,397: INFO/job] Job 236: not_deleted=True
[2017-06-15 17:03:37,397: INFO/job_scheduler] Job 236: complete_job: Updating cache
[2017-06-15 17:03:37,398: INFO/job_scheduler] Job 236: locked_item 140623739828304 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-15 17:03:37,407: INFO/job_scheduler] Job 236 completing (errored=False, cancelled=False)
[2017-06-15 17:03:37,441: INFO/job_scheduler] run_next: 1 runnable jobs of (7 pending, 2 tasked)
[2017-06-15 17:03:37,447: INFO/job_scheduler] Job 237: RunJobThread.run
[2017-06-15 17:03:37,447: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-06-15 17:03:37,474: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 338e89f7-c3e3-4fe6-b40d-c89f087085a6
[2017-06-15 17:03:37,474: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 338e89f7-c3e3-4fe6-b40d-c89f087085a6
[2017-06-15 17:03:38,020: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:03:38,020: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:03:38,020: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:03:38,020: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:03:38,021: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 63b83902-6728-4f47-895e-30939781bb3b
[2017-06-15 17:03:38,021: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 63b83902-6728-4f47-895e-30939781bb3b
[2017-06-15 17:03:38,028: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 63b83902-6728-4f47-895e-30939781bb3b
[2017-06-15 17:03:38,028: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 63b83902-6728-4f47-895e-30939781bb3b
[2017-06-15 17:03:38,029: INFO/job_scheduler] Job 274 finished 1 steps successfully
[2017-06-15 17:03:38,164: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 140623740273872
[2017-06-15 17:03:38,219: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm8.lotus.hpdd.lab.intel.com at severity 30
[2017-06-15 17:03:38,235: INFO/job] Job 274: StateChangeJob complete, setting state started on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-15 17:03:38,235: INFO/job] Job 274: not_deleted=True
[2017-06-15 17:03:38,235: INFO/job_scheduler] Job 274: complete_job: Updating cache
[2017-06-15 17:03:38,235: INFO/job_scheduler] Job 274: locked_item 140623740273872 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-15 17:03:38,242: INFO/job_scheduler] Job 274 completing (errored=False, cancelled=False)
[2017-06-15 17:03:38,296: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-8
[2017-06-15 17:03:38,296: INFO/job_scheduler] _notify: Updating .mcast_port of item 8 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 52617
[2017-06-15 17:03:38,308: INFO/job_scheduler] _notify: Set mcast_port=52617 on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-8) and saved
[2017-06-15 17:03:38,311: INFO/job_scheduler] _notify: Updating .network_interfaces of item 8 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-06-15 17:03:38,345: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-8) and saved
[2017-06-15 17:03:38,363: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 8 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-06-15 17:03:38,382: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-8) and saved
[2017-06-15 17:03:38,407: INFO/job_scheduler] run_next: 1 runnable jobs of (6 pending, 2 tasked)
[2017-06-15 17:03:38,506: INFO/job_scheduler] Job 275: RunJobThread.run
[2017-06-15 17:03:38,508: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-06-15 17:03:38,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 31012b07-866c-4607-bf6f-c6e1e012bcf7
[2017-06-15 17:03:38,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 31012b07-866c-4607-bf6f-c6e1e012bcf7
[2017-06-15 17:03:38,612: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 3 tasked)
[2017-06-15 17:03:38,618: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 3 tasked)
[2017-06-15 17:03:45,575: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:45,575: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:45,576: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:45,576: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:45,576: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 338e89f7-c3e3-4fe6-b40d-c89f087085a6
[2017-06-15 17:03:45,576: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 338e89f7-c3e3-4fe6-b40d-c89f087085a6
[2017-06-15 17:03:45,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 338e89f7-c3e3-4fe6-b40d-c89f087085a6
[2017-06-15 17:03:45,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 338e89f7-c3e3-4fe6-b40d-c89f087085a6
[2017-06-15 17:03:45,600: INFO/job_scheduler] Job 237 finished 1 steps successfully
[2017-06-15 17:03:45,654: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 140623739828304
[2017-06-15 17:03:45,693: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 30
[2017-06-15 17:03:45,703: INFO/job] Job 237: StateChangeJob complete, setting state started on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:03:45,704: INFO/job] Job 237: not_deleted=True
[2017-06-15 17:03:45,704: INFO/job_scheduler] Job 237: complete_job: Updating cache
[2017-06-15 17:03:45,704: INFO/job_scheduler] Job 237: locked_item 140623739828304 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-15 17:03:45,708: INFO/job_scheduler] Job 237 completing (errored=False, cancelled=False)
[2017-06-15 17:03:45,763: INFO/job_scheduler] run_next: 1 runnable jobs of (5 pending, 2 tasked)
[2017-06-15 17:03:45,854: INFO/job_scheduler] Job 240: RunJobThread.run
[2017-06-15 17:03:45,855: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-06-15 17:03:45,906: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ede3a3e7-9cfb-45da-9134-eac216559094
[2017-06-15 17:03:45,906: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ede3a3e7-9cfb-45da-9134-eac216559094
[2017-06-15 17:03:46,369: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 3 tasked)
[2017-06-15 17:03:46,390: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 3 tasked)
[2017-06-15 17:03:53,036: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:53,036: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:53,036: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:53,036: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:03:53,036: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ede3a3e7-9cfb-45da-9134-eac216559094
[2017-06-15 17:03:53,036: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ede3a3e7-9cfb-45da-9134-eac216559094
[2017-06-15 17:03:53,042: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ede3a3e7-9cfb-45da-9134-eac216559094
[2017-06-15 17:03:53,042: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ede3a3e7-9cfb-45da-9134-eac216559094
[2017-06-15 17:03:53,043: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-06-15 17:03:53,075: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 56443211-b97a-41e9-a05d-c068c3e6b04c
[2017-06-15 17:03:53,075: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 56443211-b97a-41e9-a05d-c068c3e6b04c
[2017-06-15 17:03:56,407: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:03:56,407: INFO/job_scheduler]   Job 240 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:03:56,408: INFO/job_scheduler]   Job 241 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:03:56,408: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 3 tasked)
[2017-06-15 17:03:56,430: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:03:56,430: INFO/job_scheduler]   Job 240 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 17:03:56,430: INFO/job_scheduler]   Job 242 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:03:56,431: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 3 tasked)
[2017-06-15 17:03:56,469: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:03:56,469: INFO/job_scheduler]   Job 251 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:03:56,469: INFO/job_scheduler]   Job 252 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:03:56,470: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 3 tasked)
[2017-06-15 17:03:56,489: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:03:56,489: INFO/job_scheduler]   Job 251 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 17:03:56,489: INFO/job_scheduler]   Job 253 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:03:56,490: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 3 tasked)
[2017-06-15 17:03:58,450: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 3 tasked)
[2017-06-15 17:03:58,479: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 3 tasked)
[2017-06-15 17:04:06,412: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 3 tasked)
[2017-06-15 17:04:06,492: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 3 tasked)
[2017-06-15 17:04:06,576: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:04:06,577: INFO/job_scheduler]   Job 240 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 17:04:06,577: INFO/job_scheduler]   Job 242 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:04:06,577: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 3 tasked)
[2017-06-15 17:04:06,605: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:04:06,605: INFO/job_scheduler]   Job 251 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 17:04:06,606: INFO/job_scheduler]   Job 253 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:04:06,606: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 3 tasked)
[2017-06-15 17:04:06,866: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:04:06,866: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:04:06,866: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:04:06,866: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/1d8e35c5-b1ca-4916-b200-ba7a75898e7e
[2017-06-15 17:04:06,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 31012b07-866c-4607-bf6f-c6e1e012bcf7
[2017-06-15 17:04:06,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 31012b07-866c-4607-bf6f-c6e1e012bcf7
[2017-06-15 17:04:06,894: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 31012b07-866c-4607-bf6f-c6e1e012bcf7
[2017-06-15 17:04:06,894: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 31012b07-866c-4607-bf6f-c6e1e012bcf7
[2017-06-15 17:04:06,896: INFO/job_scheduler] Job 275 finished 1 steps successfully
[2017-06-15 17:04:07,352: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 140623538154896
[2017-06-15 17:04:07,357: INFO/job] Job 275: StateChangeJob complete, setting state managed on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-15 17:04:07,358: INFO/job] Job 275: not_deleted=True
[2017-06-15 17:04:07,358: INFO/job_scheduler] Job 275: complete_job: Updating cache
[2017-06-15 17:04:07,358: INFO/job_scheduler] Job 275: locked_item 140623538154896 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-15 17:04:07,361: INFO/job_scheduler] Job 275 completing (errored=False, cancelled=False)
[2017-06-15 17:04:07,434: INFO/job_scheduler] Replaying 6 buffered notifications for ManagedHost-8
[2017-06-15 17:04:07,445: INFO/job_scheduler] _notify: Updating .immutable_state of item 8 (lotus-32vm8.lotus.hpdd.lab.intel.com) from True to False
[2017-06-15 17:04:07,463: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-8) and saved
[2017-06-15 17:04:07,481: INFO/job_scheduler] _notify: Updating .boot_time of item 8 (lotus-32vm8.lotus.hpdd.lab.intel.com) from 2017-06-14 09:00:40+00:00 to 2017-06-15 09:59:41+00:00
[2017-06-15 17:04:07,487: INFO/job_scheduler] _notify: Set boot_time=2017-06-15 09:59:41+00:00 on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-8) and saved
[2017-06-15 17:04:07,531: INFO/job_scheduler] _notify: Updating .properties of item 8 (lotus-32vm8.lotus.hpdd.lab.intel.com) from {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": false, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.21.1.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-06-15 17:04:07,538: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.21.1.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-8) and saved
[2017-06-15 17:04:07,556: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 8 (lotus-32vm8.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7fe58410a510> to [7]
[2017-06-15 17:04:07,586: INFO/job_scheduler] _notify: Set ha_cluster_peers=[7] on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-8) and saved
[2017-06-15 17:04:07,806: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-15 17:04:08,525: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:04:08,525: INFO/job_scheduler]   Job 240 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 17:04:08,526: INFO/job_scheduler]   Job 242 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:04:08,526: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-15 17:04:08,562: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:04:08,562: INFO/job_scheduler]   Job 251 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 17:04:08,562: INFO/job_scheduler]   Job 253 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:04:08,562: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-15 17:04:16,427: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:04:16,427: INFO/job_scheduler]   Job 240 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 17:04:16,427: INFO/job_scheduler]   Job 242 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:04:16,427: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-15 17:04:16,449: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:04:16,449: INFO/job_scheduler]   Job 251 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 17:04:16,449: INFO/job_scheduler]   Job 253 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:04:16,450: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-15 17:04:18,538: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:04:18,538: INFO/job_scheduler]   Job 240 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 17:04:18,538: INFO/job_scheduler]   Job 242 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:04:18,539: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-15 17:04:18,576: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:04:18,576: INFO/job_scheduler]   Job 251 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 17:04:18,576: INFO/job_scheduler]   Job 253 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:04:18,577: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-15 17:04:26,520: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:04:26,521: INFO/job_scheduler]   Job 240 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:04:26,521: INFO/job_scheduler]   Job 241 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:04:26,521: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-15 17:04:26,551: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:04:26,551: INFO/job_scheduler]   Job 240 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 17:04:26,551: INFO/job_scheduler]   Job 242 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:04:26,552: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-15 17:04:26,612: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:04:26,612: INFO/job_scheduler]   Job 251 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:04:26,612: INFO/job_scheduler]   Job 252 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:04:26,612: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-15 17:04:26,638: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:04:26,638: INFO/job_scheduler]   Job 251 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 17:04:26,638: INFO/job_scheduler]   Job 253 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:04:26,639: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-15 17:04:28,562: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:04:28,562: INFO/job_scheduler]   Job 240 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:04:28,563: INFO/job_scheduler]   Job 241 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:04:28,563: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-15 17:04:28,576: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:04:28,576: INFO/job_scheduler]   Job 240 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 17:04:28,576: INFO/job_scheduler]   Job 242 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:04:28,576: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-15 17:04:28,596: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:04:28,597: INFO/job_scheduler]   Job 251 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:04:28,597: INFO/job_scheduler]   Job 252 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:04:28,597: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-15 17:04:28,613: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:04:28,613: INFO/job_scheduler]   Job 251 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 17:04:28,613: INFO/job_scheduler]   Job 253 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:04:28,613: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-15 17:04:36,579: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:04:36,580: INFO/job_scheduler]   Job 240 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:04:36,580: INFO/job_scheduler]   Job 241 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:04:36,580: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-15 17:04:36,622: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:04:36,623: INFO/job_scheduler]   Job 240 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 17:04:36,623: INFO/job_scheduler]   Job 242 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:04:36,623: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-15 17:04:36,652: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:04:36,652: INFO/job_scheduler]   Job 251 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:04:36,652: INFO/job_scheduler]   Job 252 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:04:36,653: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-15 17:04:36,682: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:04:36,682: INFO/job_scheduler]   Job 251 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 17:04:36,682: INFO/job_scheduler]   Job 253 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:04:36,682: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-15 17:04:38,619: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:04:38,619: INFO/job_scheduler]   Job 240 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:04:38,619: INFO/job_scheduler]   Job 241 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:04:38,620: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-15 17:04:38,641: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:04:38,641: INFO/job_scheduler]   Job 240 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 17:04:38,641: INFO/job_scheduler]   Job 242 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:04:38,641: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-15 17:04:38,666: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:04:38,666: INFO/job_scheduler]   Job 251 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:04:38,667: INFO/job_scheduler]   Job 252 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:04:38,667: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-15 17:04:38,680: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:04:38,681: INFO/job_scheduler]   Job 251 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 17:04:38,681: INFO/job_scheduler]   Job 253 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:04:38,681: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-15 17:04:40,329: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:04:40,329: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:04:40,329: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:04:40,329: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:04:40,330: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 56443211-b97a-41e9-a05d-c068c3e6b04c
[2017-06-15 17:04:40,330: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 56443211-b97a-41e9-a05d-c068c3e6b04c
[2017-06-15 17:04:40,331: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 56443211-b97a-41e9-a05d-c068c3e6b04c
[2017-06-15 17:04:40,331: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 56443211-b97a-41e9-a05d-c068c3e6b04c
[2017-06-15 17:04:40,333: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-06-15 17:04:40,346: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e0d6ca35-03c0-41db-982e-b490cbc8dc90
[2017-06-15 17:04:40,346: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e0d6ca35-03c0-41db-982e-b490cbc8dc90
[2017-06-15 17:04:46,501: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-15 17:04:46,517: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-15 17:04:47,448: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:04:47,448: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:04:47,448: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:04:47,448: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:04:47,449: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e0d6ca35-03c0-41db-982e-b490cbc8dc90
[2017-06-15 17:04:47,449: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e0d6ca35-03c0-41db-982e-b490cbc8dc90
[2017-06-15 17:04:47,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e0d6ca35-03c0-41db-982e-b490cbc8dc90
[2017-06-15 17:04:47,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e0d6ca35-03c0-41db-982e-b490cbc8dc90
[2017-06-15 17:04:47,468: INFO/job_scheduler] Job 240 finished 3 steps successfully
[2017-06-15 17:04:47,573: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 140623740244944
[2017-06-15 17:04:47,580: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-06-15 17:04:47,603: INFO/job] Job 240: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-15 17:04:47,603: INFO/job] Job 240: not_deleted=True
[2017-06-15 17:04:47,604: INFO/job_scheduler] Job 240: complete_job: Updating cache
[2017-06-15 17:04:47,604: INFO/job_scheduler] Job 240: locked_item 140623740244944 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-15 17:04:47,606: INFO/job_scheduler] Job 240 completing (errored=False, cancelled=False)
[2017-06-15 17:04:47,630: INFO/job_scheduler] run_next: 1 runnable jobs of (4 pending, 1 tasked)
[2017-06-15 17:04:47,682: INFO/job_scheduler] Job 241: RunJobThread.run
[2017-06-15 17:04:47,683: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-06-15 17:04:47,706: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2315463d-096e-428f-aadf-f7e3bfd9f8e8
[2017-06-15 17:04:47,706: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2315463d-096e-428f-aadf-f7e3bfd9f8e8
[2017-06-15 17:04:48,623: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:04:48,624: INFO/job_scheduler]   Job 241 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:04:48,624: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 2 tasked)
[2017-06-15 17:04:48,640: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:04:48,640: INFO/job_scheduler]   Job 242 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:04:48,640: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 2 tasked)
[2017-06-15 17:04:48,660: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:04:48,661: INFO/job_scheduler]   Job 251 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:04:48,661: INFO/job_scheduler]   Job 252 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:04:48,661: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 2 tasked)
[2017-06-15 17:04:48,678: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:04:48,678: INFO/job_scheduler]   Job 251 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 17:04:48,678: INFO/job_scheduler]   Job 253 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:04:48,678: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 2 tasked)
[2017-06-15 17:04:54,655: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:04:54,655: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:04:54,655: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:04:54,655: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:04:54,656: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 553ce96a-b687-4476-9b90-85c93837bcc2
[2017-06-15 17:04:54,656: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 553ce96a-b687-4476-9b90-85c93837bcc2
[2017-06-15 17:04:54,683: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 553ce96a-b687-4476-9b90-85c93837bcc2
[2017-06-15 17:04:54,683: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 553ce96a-b687-4476-9b90-85c93837bcc2
[2017-06-15 17:04:54,686: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-06-15 17:04:54,724: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f6d2d9d6-c7d2-4905-8674-68c7c2b31370
[2017-06-15 17:04:54,724: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f6d2d9d6-c7d2-4905-8674-68c7c2b31370
[2017-06-15 17:04:54,831: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:04:54,831: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:04:54,831: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:04:54,831: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:04:54,831: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2315463d-096e-428f-aadf-f7e3bfd9f8e8
[2017-06-15 17:04:54,831: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2315463d-096e-428f-aadf-f7e3bfd9f8e8
[2017-06-15 17:04:54,843: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2315463d-096e-428f-aadf-f7e3bfd9f8e8
[2017-06-15 17:04:54,843: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2315463d-096e-428f-aadf-f7e3bfd9f8e8
[2017-06-15 17:04:54,843: INFO/job_scheduler] Job 241 finished 1 steps successfully
[2017-06-15 17:04:55,079: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 140623740244944
[2017-06-15 17:04:55,125: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 30
[2017-06-15 17:04:55,132: INFO/job] Job 241: StateChangeJob complete, setting state started on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-15 17:04:55,132: INFO/job] Job 241: not_deleted=True
[2017-06-15 17:04:55,132: INFO/job_scheduler] Job 241: complete_job: Updating cache
[2017-06-15 17:04:55,132: INFO/job_scheduler] Job 241: locked_item 140623740244944 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-15 17:04:55,137: INFO/job_scheduler] Job 241 completing (errored=False, cancelled=False)
[2017-06-15 17:04:55,160: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-5
[2017-06-15 17:04:55,160: INFO/job_scheduler] _notify: Updating .mcast_port of item 5 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 62665
[2017-06-15 17:04:55,208: INFO/job_scheduler] _notify: Set mcast_port=62665 on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-5) and saved
[2017-06-15 17:04:55,209: INFO/job_scheduler] _notify: Updating .network_interfaces of item 5 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-06-15 17:04:55,223: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-5) and saved
[2017-06-15 17:04:55,234: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 5 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-06-15 17:04:55,238: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-5) and saved
[2017-06-15 17:04:55,244: INFO/job_scheduler] run_next: 1 runnable jobs of (3 pending, 1 tasked)
[2017-06-15 17:04:55,293: INFO/job_scheduler] Job 242: RunJobThread.run
[2017-06-15 17:04:55,295: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-06-15 17:04:55,323: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fafdb525-e13a-4d06-9b13-861fd2e0dbaf
[2017-06-15 17:04:55,323: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fafdb525-e13a-4d06-9b13-861fd2e0dbaf
[2017-06-15 17:04:56,546: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 2 tasked)
[2017-06-15 17:04:56,554: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 2 tasked)
[2017-06-15 17:04:56,579: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:04:56,579: INFO/job_scheduler]   Job 242 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:04:56,579: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 2 tasked)
[2017-06-15 17:04:56,598: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:04:56,599: INFO/job_scheduler]   Job 251 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:04:56,599: INFO/job_scheduler]   Job 252 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:04:56,599: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 2 tasked)
[2017-06-15 17:04:56,611: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:04:56,611: INFO/job_scheduler]   Job 251 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 17:04:56,612: INFO/job_scheduler]   Job 253 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:04:56,612: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 2 tasked)
[2017-06-15 17:04:58,652: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 2 tasked)
[2017-06-15 17:04:58,681: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 2 tasked)
[2017-06-15 17:04:58,726: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:04:58,727: INFO/job_scheduler]   Job 242 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:04:58,727: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 2 tasked)
[2017-06-15 17:04:58,749: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:04:58,749: INFO/job_scheduler]   Job 251 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:04:58,749: INFO/job_scheduler]   Job 252 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:04:58,749: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 2 tasked)
[2017-06-15 17:04:58,765: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:04:58,773: INFO/job_scheduler]   Job 251 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 17:04:58,773: INFO/job_scheduler]   Job 253 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:04:58,773: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 2 tasked)
[2017-06-15 17:05:02,808: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:05:02,808: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:05:02,808: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:05:02,808: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:05:02,809: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f6d2d9d6-c7d2-4905-8674-68c7c2b31370
[2017-06-15 17:05:02,809: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f6d2d9d6-c7d2-4905-8674-68c7c2b31370
[2017-06-15 17:05:02,853: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f6d2d9d6-c7d2-4905-8674-68c7c2b31370
[2017-06-15 17:05:02,853: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f6d2d9d6-c7d2-4905-8674-68c7c2b31370
[2017-06-15 17:05:02,853: INFO/job_scheduler] Job 251 finished 3 steps successfully
[2017-06-15 17:05:02,909: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 140623739921296
[2017-06-15 17:05:02,917: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-06-15 17:05:02,940: INFO/job] Job 251: StateChangeJob complete, setting state stopped on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-15 17:05:02,940: INFO/job] Job 251: not_deleted=True
[2017-06-15 17:05:02,940: INFO/job_scheduler] Job 251: complete_job: Updating cache
[2017-06-15 17:05:02,940: INFO/job_scheduler] Job 251: locked_item 140623739921296 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-15 17:05:02,943: INFO/job_scheduler] Job 251 completing (errored=False, cancelled=False)
[2017-06-15 17:05:02,964: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 1 tasked)
[2017-06-15 17:05:02,975: INFO/job_scheduler] Job 252: RunJobThread.run
[2017-06-15 17:05:02,976: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-06-15 17:05:02,996: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 063bb85a-f654-43a5-a6c0-1aa718d6bad3
[2017-06-15 17:05:02,996: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 063bb85a-f654-43a5-a6c0-1aa718d6bad3
[2017-06-15 17:05:06,620: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:05:06,620: INFO/job_scheduler]   Job 242 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:05:06,621: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-06-15 17:05:06,641: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:05:06,641: INFO/job_scheduler]   Job 252 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:05:06,641: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-06-15 17:05:06,657: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:05:06,657: INFO/job_scheduler]   Job 253 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:05:06,657: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-06-15 17:05:08,702: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-06-15 17:05:08,718: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-06-15 17:05:08,754: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 5 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from True to False
[2017-06-15 17:05:08,764: INFO/job_scheduler] _notify: Set corosync_reported_up=False on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-5) and saved
[2017-06-15 17:05:08,783: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-06-15 17:05:08,793: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:05:08,793: INFO/job_scheduler]   Job 242 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:05:08,794: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-06-15 17:05:08,828: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:05:08,828: INFO/job_scheduler]   Job 253 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:05:08,828: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 2 tasked)
[2017-06-15 17:05:10,078: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:05:10,078: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:05:10,078: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:05:10,078: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:05:10,079: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 063bb85a-f654-43a5-a6c0-1aa718d6bad3
[2017-06-15 17:05:10,079: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 063bb85a-f654-43a5-a6c0-1aa718d6bad3
[2017-06-15 17:05:10,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 063bb85a-f654-43a5-a6c0-1aa718d6bad3
[2017-06-15 17:05:10,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 063bb85a-f654-43a5-a6c0-1aa718d6bad3
[2017-06-15 17:05:10,116: INFO/job_scheduler] Job 252 finished 1 steps successfully
[2017-06-15 17:05:10,158: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 140623739921296
[2017-06-15 17:05:10,202: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm6.lotus.hpdd.lab.intel.com at severity 30
[2017-06-15 17:05:10,217: INFO/job] Job 252: StateChangeJob complete, setting state started on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-15 17:05:10,217: INFO/job] Job 252: not_deleted=True
[2017-06-15 17:05:10,217: INFO/job_scheduler] Job 252: complete_job: Updating cache
[2017-06-15 17:05:10,217: INFO/job_scheduler] Job 252: locked_item 140623739921296 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-15 17:05:10,220: INFO/job_scheduler] Job 252 completing (errored=False, cancelled=False)
[2017-06-15 17:05:10,238: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-6
[2017-06-15 17:05:10,238: INFO/job_scheduler] _notify: Updating .mcast_port of item 6 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 62665
[2017-06-15 17:05:10,247: INFO/job_scheduler] _notify: Set mcast_port=62665 on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-6) and saved
[2017-06-15 17:05:10,248: INFO/job_scheduler] _notify: Updating .network_interfaces of item 6 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-06-15 17:05:10,281: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-6) and saved
[2017-06-15 17:05:10,313: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 6 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-06-15 17:05:10,328: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-6) and saved
[2017-06-15 17:05:10,339: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-15 17:05:10,347: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration in state lnet_unloaded, not one of ['lnet_up'] (preferred lnet_up)
[2017-06-15 17:05:10,355: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration in state lnet_unloaded, not one of ['lnet_up'] (preferred lnet_up)
[2017-06-15 17:05:10,359: WARNING/job_scheduler] Job 253: cancelling because of failed dependency
[2017-06-15 17:05:10,359: INFO/job_scheduler] Job 253 completing (errored=False, cancelled=True)
[2017-06-15 17:05:10,416: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 17:05:16,649: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 5 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-06-15 17:05:16,655: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-5) and saved
[2017-06-15 17:05:16,673: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 17:05:16,684: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-15 17:05:16,684: INFO/job_scheduler]   Job 242 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-15 17:05:16,684: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 17:05:16,719: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 6 (lotus-32vm6.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7fe584217d50> to [5]
[2017-06-15 17:05:16,742: INFO/job_scheduler] _notify: Set ha_cluster_peers=[5] on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-6) and saved
[2017-06-15 17:05:16,888: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-15 17:05:20,579: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:05:20,579: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:05:20,579: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:05:20,579: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 17:05:20,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fafdb525-e13a-4d06-9b13-861fd2e0dbaf
[2017-06-15 17:05:20,580: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fafdb525-e13a-4d06-9b13-861fd2e0dbaf
[2017-06-15 17:05:20,624: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fafdb525-e13a-4d06-9b13-861fd2e0dbaf
[2017-06-15 17:05:20,624: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fafdb525-e13a-4d06-9b13-861fd2e0dbaf
[2017-06-15 17:05:20,626: INFO/job_scheduler] Job 242 finished 1 steps successfully
[2017-06-15 17:05:20,984: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 140623538052624
[2017-06-15 17:05:20,990: INFO/job] Job 242: StateChangeJob complete, setting state managed on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-15 17:05:20,990: INFO/job] Job 242: not_deleted=True
[2017-06-15 17:05:20,990: INFO/job_scheduler] Job 242: complete_job: Updating cache
[2017-06-15 17:05:20,990: INFO/job_scheduler] Job 242: locked_item 140623538052624 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-15 17:05:20,993: INFO/job_scheduler] Job 242 completing (errored=False, cancelled=False)
[2017-06-15 17:05:21,050: INFO/job_scheduler] Replaying 6 buffered notifications for ManagedHost-5
[2017-06-15 17:05:21,080: INFO/job_scheduler] _notify: Updating .immutable_state of item 5 (lotus-32vm5.lotus.hpdd.lab.intel.com) from True to False
[2017-06-15 17:05:21,096: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-5) and saved
[2017-06-15 17:05:21,110: INFO/job_scheduler] _notify: Updating .boot_time of item 5 (lotus-32vm5.lotus.hpdd.lab.intel.com) from 2017-06-14 09:00:19+00:00 to 2017-06-15 10:01:54+00:00
[2017-06-15 17:05:21,127: INFO/job_scheduler] _notify: Set boot_time=2017-06-15 10:01:54+00:00 on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-5) and saved
[2017-06-15 17:05:21,148: INFO/job_scheduler] _notify: Updating .properties of item 5 (lotus-32vm5.lotus.hpdd.lab.intel.com) from {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": false, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.21.1.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-06-15 17:05:21,163: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.21.1.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-5) and saved
[2017-06-15 17:05:21,195: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 5 (lotus-32vm5.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7fe58419b150> to [6]
[2017-06-15 17:05:21,233: INFO/job_scheduler] _notify: Set ha_cluster_peers=[6] on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-5) and saved
[2017-06-15 17:05:21,384: INFO/job_scheduler] Replaying 5 buffered notifications for ManagedHost-6
[2017-06-15 17:05:21,398: INFO/job_scheduler] _notify: Updating .immutable_state of item 6 (lotus-32vm6.lotus.hpdd.lab.intel.com) from True to False
[2017-06-15 17:05:21,414: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-6) and saved
[2017-06-15 17:05:21,452: INFO/job_scheduler] _notify: Updating .properties of item 6 (lotus-32vm6.lotus.hpdd.lab.intel.com) from {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": false, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-06-15 17:05:21,458: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-6) and saved
[2017-06-15 17:05:21,479: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 6 (lotus-32vm6.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7fe5840ea210> to [5]
[2017-06-15 17:05:21,531: INFO/job_scheduler] _notify: Set ha_cluster_peers=[5] on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-6) and saved
[2017-06-15 17:05:21,646: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:10:39,639: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:10:39,671: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:10:51,207: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:10:51,262: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:10:54,206: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:10:54,222: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:13:07,889: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:13:07,919: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:20:51,315: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:20:51,335: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:21:02,945: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:21:02,958: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:21:06,017: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:21:06,039: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:23:19,630: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:23:19,651: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:31:03,008: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:31:03,023: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:31:14,812: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:31:14,830: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:31:17,889: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:31:17,911: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:33:31,338: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:33:31,365: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:41:14,802: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:41:14,825: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:41:26,621: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:41:26,649: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:41:29,685: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:41:29,708: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:43:43,006: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:43:43,022: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:51:26,553: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:51:26,588: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:51:38,717: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:51:38,734: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:51:41,433: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:51:41,458: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:53:54,716: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:53:54,766: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:58:51,696: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 17:58:51,714: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-6 transitioning from packages_installed to managed
[2017-06-15 17:58:51,762: INFO/command_plan] Created command 149 (Setup managed host lotus-32vm6.lotus.hpdd.lab.intel.com) with 3 jobs
[2017-06-15 17:58:51,785: INFO/job_scheduler] run_next: 1 runnable jobs of (3 pending, 0 tasked)
[2017-06-15 17:58:51,793: INFO/job_scheduler] Job 280: RunJobThread.run
[2017-06-15 17:58:51,794: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com load_lnet {}
[2017-06-15 17:58:51,831: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 15540586-666b-4e51-ad2d-794dc49050d0
[2017-06-15 17:58:51,831: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 15540586-666b-4e51-ad2d-794dc49050d0
[2017-06-15 17:58:52,918: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:58:52,918: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:58:52,918: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:58:52,918: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:58:52,919: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 15540586-666b-4e51-ad2d-794dc49050d0
[2017-06-15 17:58:52,919: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 15540586-666b-4e51-ad2d-794dc49050d0
[2017-06-15 17:58:52,947: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 15540586-666b-4e51-ad2d-794dc49050d0
[2017-06-15 17:58:52,947: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 15540586-666b-4e51-ad2d-794dc49050d0
[2017-06-15 17:58:52,947: INFO/job] Job 280 LoadLNetStep: Error (1) running 'modprobe lnet': '' 'modprobe: ERROR: could not insert 'lnet': Invalid argument
'
[2017-06-15 17:58:52,948: ERROR/job_scheduler] Job 280 step 0 encountered an agent error: Error (1) running 'modprobe lnet': '' 'modprobe: ERROR: could not insert 'lnet': Invalid argument
'
[2017-06-15 17:58:53,010: INFO/job_scheduler] Job 280: complete_job: Updating cache
[2017-06-15 17:58:53,010: INFO/job_scheduler] Job 280: locked_item 140623739946064 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-15 17:58:53,013: INFO/job_scheduler] Job 280 completing (errored=True, cancelled=False)
[2017-06-15 17:58:53,026: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 0 tasked)
[2017-06-15 17:58:53,029: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration in state lnet_unloaded, not one of ['lnet_down'] (preferred lnet_down)
[2017-06-15 17:58:53,035: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration in state lnet_unloaded, not one of ['lnet_down'] (preferred lnet_down)
[2017-06-15 17:58:53,035: WARNING/job_scheduler] Job 281: cancelling because of failed dependency
[2017-06-15 17:58:53,035: INFO/job_scheduler] Job 281 completing (errored=False, cancelled=True)
[2017-06-15 17:58:53,041: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-15 17:58:53,048: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration in state lnet_unloaded, not one of ['lnet_up'] (preferred lnet_up)
[2017-06-15 17:58:53,063: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration in state lnet_unloaded, not one of ['lnet_up'] (preferred lnet_up)
[2017-06-15 17:58:53,074: WARNING/job_scheduler] Job 282: cancelling because of failed dependency
[2017-06-15 17:58:53,074: INFO/job_scheduler] Job 282 completing (errored=False, cancelled=True)
[2017-06-15 17:58:53,123: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:59:23,488: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 17:59:23,491: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-6 transitioning from packages_installed to removed
[2017-06-15 17:59:23,665: INFO/command_plan] Created command 150 (Remove host lotus-32vm6.lotus.hpdd.lab.intel.com from configuration) with 8 jobs
[2017-06-15 17:59:23,715: INFO/job_scheduler] run_next: 2 runnable jobs of (8 pending, 0 tasked)
[2017-06-15 17:59:23,747: INFO/job_scheduler] Job 283: RunJobThread.run
[2017-06-15 17:59:23,748: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-06-15 17:59:23,766: INFO/job_scheduler] Job 284: RunJobThread.run
[2017-06-15 17:59:23,767: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com unconfigure_lnet {}
[2017-06-15 17:59:23,779: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d14bbe05-bbbb-4d83-a781-a77a1a7ac097
[2017-06-15 17:59:23,779: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d14bbe05-bbbb-4d83-a781-a77a1a7ac097
[2017-06-15 17:59:23,797: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0243ded9-47e3-4e1d-8198-700175e6a548
[2017-06-15 17:59:23,797: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0243ded9-47e3-4e1d-8198-700175e6a548
[2017-06-15 17:59:23,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:59:23,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:59:23,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:59:23,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:59:23,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0243ded9-47e3-4e1d-8198-700175e6a548
[2017-06-15 17:59:23,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0243ded9-47e3-4e1d-8198-700175e6a548
[2017-06-15 17:59:23,916: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0243ded9-47e3-4e1d-8198-700175e6a548
[2017-06-15 17:59:23,916: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0243ded9-47e3-4e1d-8198-700175e6a548
[2017-06-15 17:59:23,917: INFO/job] Invalid result None fixed up on called to unconfigure_lnet with args {}
[2017-06-15 17:59:23,917: INFO/job_scheduler] Job 284 finished 1 steps successfully
[2017-06-15 17:59:23,970: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->unconfigured (intentional=True) 140623740651088
[2017-06-15 17:59:23,978: INFO/job] Job 284: StateChangeJob complete, setting state unconfigured on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-15 17:59:23,978: INFO/job] Job 284: not_deleted=True
[2017-06-15 17:59:23,978: INFO/job_scheduler] Job 284: complete_job: Updating cache
[2017-06-15 17:59:23,979: INFO/job_scheduler] Job 284: locked_item 140623740651088 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-15 17:59:23,982: INFO/job_scheduler] Job 284 completing (errored=False, cancelled=False)
[2017-06-15 17:59:24,003: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 1 tasked)
[2017-06-15 17:59:27,787: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 1 tasked)
[2017-06-15 17:59:27,803: INFO/job_scheduler] run_next: 0 runnable jobs of (6 pending, 1 tasked)
[2017-06-15 17:59:30,907: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:59:30,907: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:59:30,907: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:59:30,907: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:59:30,907: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d14bbe05-bbbb-4d83-a781-a77a1a7ac097
[2017-06-15 17:59:30,907: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d14bbe05-bbbb-4d83-a781-a77a1a7ac097
[2017-06-15 17:59:30,948: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d14bbe05-bbbb-4d83-a781-a77a1a7ac097
[2017-06-15 17:59:30,948: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d14bbe05-bbbb-4d83-a781-a77a1a7ac097
[2017-06-15 17:59:30,948: INFO/job_scheduler] Job 283 finished 1 steps successfully
[2017-06-15 17:59:31,033: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration started->stopped (intentional=True) 140623740691536
[2017-06-15 17:59:31,041: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-06-15 17:59:31,046: INFO/job] Job 283: StateChangeJob complete, setting state stopped on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-15 17:59:31,046: INFO/job] Job 283: not_deleted=True
[2017-06-15 17:59:31,046: INFO/job_scheduler] Job 283: complete_job: Updating cache
[2017-06-15 17:59:31,048: INFO/job_scheduler] Job 283: locked_item 140623740691536 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-15 17:59:31,056: INFO/job_scheduler] Job 283 completing (errored=False, cancelled=False)
[2017-06-15 17:59:31,159: INFO/job_scheduler] run_next: 2 runnable jobs of (6 pending, 0 tasked)
[2017-06-15 17:59:31,171: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration in state unconfigured, not one of ['lnet_unloaded', 'lnet_down', 'lnet_up'] (preferred lnet_unloaded)
[2017-06-15 17:59:31,182: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration in state unconfigured, not one of ['lnet_unloaded', 'lnet_down', 'lnet_up'] (preferred lnet_unloaded)
[2017-06-15 17:59:31,185: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration in state stopped, not one of ['started'] (preferred started)
[2017-06-15 17:59:31,192: WARNING/job_scheduler] Job 285: cancelling because of failed dependency
[2017-06-15 17:59:31,192: INFO/job_scheduler] Job 285 completing (errored=False, cancelled=True)
[2017-06-15 17:59:31,198: INFO/job_scheduler] Job 288: RunJobThread.run
[2017-06-15 17:59:31,199: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com stop_corosync2 {}
[2017-06-15 17:59:31,207: INFO/job_scheduler] run_next: 2 runnable jobs of (4 pending, 1 tasked)
[2017-06-15 17:59:31,216: INFO/job_scheduler] Job 286: RunJobThread.run
[2017-06-15 17:59:31,217: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com unconfigure_rsyslog {}
[2017-06-15 17:59:31,220: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc debc2bf2-1082-4a87-82d2-a723c3481be5
[2017-06-15 17:59:31,220: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc debc2bf2-1082-4a87-82d2-a723c3481be5
[2017-06-15 17:59:31,235: INFO/job_scheduler] Job 287: RunJobThread.run
[2017-06-15 17:59:31,236: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com unconfigure_ntp {}
[2017-06-15 17:59:31,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc aa12f515-1569-435d-adb7-6722f305371f
[2017-06-15 17:59:31,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc aa12f515-1569-435d-adb7-6722f305371f
[2017-06-15 17:59:31,275: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9c36f7cd-e96a-4a19-9316-0ccd76bbb844
[2017-06-15 17:59:31,275: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9c36f7cd-e96a-4a19-9316-0ccd76bbb844
[2017-06-15 17:59:35,668: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:59:35,668: INFO/job_scheduler]   Job 288 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration started->stopped
[2017-06-15 17:59:35,668: INFO/job_scheduler]   Job 289 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->unconfigured
[2017-06-15 17:59:35,669: INFO/job_scheduler]   Job 290 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:59:35,669: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 3 tasked)
[2017-06-15 17:59:37,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:59:37,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:59:37,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:59:37,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:59:37,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9c36f7cd-e96a-4a19-9316-0ccd76bbb844
[2017-06-15 17:59:37,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9c36f7cd-e96a-4a19-9316-0ccd76bbb844
[2017-06-15 17:59:37,393: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9c36f7cd-e96a-4a19-9316-0ccd76bbb844
[2017-06-15 17:59:37,393: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9c36f7cd-e96a-4a19-9316-0ccd76bbb844
[2017-06-15 17:59:37,393: INFO/job_scheduler] Job 287 finished 1 steps successfully
[2017-06-15 17:59:37,466: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration configured->unconfigured (intentional=True) 140623739813456
[2017-06-15 17:59:37,469: INFO/job] Job 287: StateChangeJob complete, setting state unconfigured on lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration
[2017-06-15 17:59:37,469: INFO/job] Job 287: not_deleted=True
[2017-06-15 17:59:37,469: INFO/job_scheduler] Job 287: complete_job: Updating cache
[2017-06-15 17:59:37,470: INFO/job_scheduler] Job 287: locked_item 140623739813456 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-06-15 17:59:37,472: INFO/job_scheduler] Job 287 completing (errored=False, cancelled=False)
[2017-06-15 17:59:37,484: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 2 tasked)
[2017-06-15 17:59:37,821: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 2 tasked)
[2017-06-15 17:59:37,838: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 2 tasked)
[2017-06-15 17:59:38,329: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:59:38,329: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:59:38,330: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:59:38,330: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:59:38,330: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc aa12f515-1569-435d-adb7-6722f305371f
[2017-06-15 17:59:38,330: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc aa12f515-1569-435d-adb7-6722f305371f
[2017-06-15 17:59:38,360: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc aa12f515-1569-435d-adb7-6722f305371f
[2017-06-15 17:59:38,360: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc aa12f515-1569-435d-adb7-6722f305371f
[2017-06-15 17:59:38,361: INFO/job_scheduler] Job 286 finished 1 steps successfully
[2017-06-15 17:59:38,507: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration configured->unconfigured (intentional=True) 140623740572496
[2017-06-15 17:59:38,511: INFO/job] Job 286: StateChangeJob complete, setting state unconfigured on lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-06-15 17:59:38,511: INFO/job] Job 286: not_deleted=True
[2017-06-15 17:59:38,511: INFO/job_scheduler] Job 286: complete_job: Updating cache
[2017-06-15 17:59:38,511: INFO/job_scheduler] Job 286: locked_item 140623740572496 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-06-15 17:59:38,516: INFO/job_scheduler] Job 286 completing (errored=False, cancelled=False)
[2017-06-15 17:59:38,559: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-15 17:59:39,309: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:59:39,309: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:59:39,309: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:59:39,309: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:59:39,310: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc debc2bf2-1082-4a87-82d2-a723c3481be5
[2017-06-15 17:59:39,310: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc debc2bf2-1082-4a87-82d2-a723c3481be5
[2017-06-15 17:59:39,336: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc debc2bf2-1082-4a87-82d2-a723c3481be5
[2017-06-15 17:59:39,336: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc debc2bf2-1082-4a87-82d2-a723c3481be5
[2017-06-15 17:59:39,337: INFO/job_scheduler] Job 288 finished 1 steps successfully
[2017-06-15 17:59:39,516: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration started->stopped (intentional=True) 140623739435216
[2017-06-15 17:59:39,523: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-06-15 17:59:39,527: INFO/job] Job 288: StateChangeJob complete, setting state stopped on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:59:39,529: INFO/job] Job 288: not_deleted=True
[2017-06-15 17:59:39,530: INFO/job_scheduler] Job 288: complete_job: Updating cache
[2017-06-15 17:59:39,531: INFO/job_scheduler] Job 288: locked_item 140623739435216 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-15 17:59:39,542: INFO/job_scheduler] Job 288 completing (errored=False, cancelled=False)
[2017-06-15 17:59:39,576: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 0 tasked)
[2017-06-15 17:59:39,581: INFO/job_scheduler] Job 289: RunJobThread.run
[2017-06-15 17:59:39,582: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com unconfigure_corosync2 {'mcast_port': 62665, 'host_fqdn': u'lotus-32vm6.lotus.hpdd.lab.intel.com'}
[2017-06-15 17:59:39,597: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d9c2fe1f-1d18-49ab-930c-7b5616e73c9e
[2017-06-15 17:59:39,597: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d9c2fe1f-1d18-49ab-930c-7b5616e73c9e
[2017-06-15 17:59:45,733: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 17:59:45,733: INFO/job_scheduler]   Job 289 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->unconfigured
[2017-06-15 17:59:45,733: INFO/job_scheduler]   Job 290 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:59:45,733: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-06-15 17:59:46,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:59:46,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:59:46,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:59:46,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 17:59:46,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d9c2fe1f-1d18-49ab-930c-7b5616e73c9e
[2017-06-15 17:59:46,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d9c2fe1f-1d18-49ab-930c-7b5616e73c9e
[2017-06-15 17:59:46,712: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d9c2fe1f-1d18-49ab-930c-7b5616e73c9e
[2017-06-15 17:59:46,712: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d9c2fe1f-1d18-49ab-930c-7b5616e73c9e
[2017-06-15 17:59:46,712: INFO/job_scheduler] Job 289 finished 1 steps successfully
[2017-06-15 17:59:46,831: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->unconfigured (intentional=True) 140623739435216
[2017-06-15 17:59:46,837: INFO/job] Job 289: StateChangeJob complete, setting state unconfigured on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 17:59:46,837: INFO/job] Job 289: not_deleted=True
[2017-06-15 17:59:46,837: INFO/job_scheduler] Job 289: complete_job: Updating cache
[2017-06-15 17:59:46,838: INFO/job_scheduler] Job 289: locked_item 140623739435216 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-15 17:59:46,842: INFO/job_scheduler] Job 289 completing (errored=False, cancelled=False)
[2017-06-15 17:59:46,859: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-15 17:59:46,870: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com in state packages_installed, not one of ['monitored'] (preferred monitored)
[2017-06-15 17:59:46,880: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com in state packages_installed, not one of ['monitored'] (preferred monitored)
[2017-06-15 17:59:46,880: WARNING/job_scheduler] Job 290: cancelling because of failed dependency
[2017-06-15 17:59:46,880: INFO/job_scheduler] Job 290 completing (errored=False, cancelled=True)
[2017-06-15 17:59:46,931: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 17:59:55,764: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 6 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from True to False
[2017-06-15 17:59:55,768: INFO/job_scheduler] _notify: Set corosync_reported_up=False on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-6) and saved
[2017-06-15 17:59:55,777: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:00:40,096: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-15 18:00:40,100: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-6 transitioning from packages_installed to removed
[2017-06-15 18:00:40,209: INFO/command_plan] Created command 151 (Remove host lotus-32vm6.lotus.hpdd.lab.intel.com from configuration) with 8 jobs
[2017-06-15 18:00:40,259: INFO/job_scheduler] run_next: 4 runnable jobs of (8 pending, 0 tasked)
[2017-06-15 18:00:40,288: INFO/job_scheduler] Job 291: open DB connection during completion
[2017-06-15 18:00:40,311: INFO/job_scheduler] Job 292: RunJobThread.run
[2017-06-15 18:00:40,311: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-06-15 18:00:40,339: INFO/job_scheduler] Job 293: RunJobThread.run
[2017-06-15 18:00:40,340: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-06-15 18:00:40,340: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 69152e0e-ffd7-4864-95db-4b5c1d8e8895
[2017-06-15 18:00:40,340: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 69152e0e-ffd7-4864-95db-4b5c1d8e8895
[2017-06-15 18:00:40,352: INFO/job_scheduler] Job 294: RunJobThread.run
[2017-06-15 18:00:40,353: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 140623538119504
[2017-06-15 18:00:40,360: INFO/job] Job 291: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-15 18:00:40,360: INFO/job] Job 291: not_deleted=True
[2017-06-15 18:00:40,360: INFO/job_scheduler] Job 291: complete_job: Updating cache
[2017-06-15 18:00:40,360: INFO/job_scheduler] Job 291: locked_item 140623538119504 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-15 18:00:40,364: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b9dc609d-bb63-4a06-ae62-67664eada9a7
[2017-06-15 18:00:40,364: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b9dc609d-bb63-4a06-ae62-67664eada9a7
[2017-06-15 18:00:40,366: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-06-15 18:00:40,375: INFO/job_scheduler] Job 291 completing (errored=False, cancelled=False)
[2017-06-15 18:00:40,389: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e894a48e-1a15-41aa-91ac-a9a1db591e24
[2017-06-15 18:00:40,389: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e894a48e-1a15-41aa-91ac-a9a1db591e24
[2017-06-15 18:00:40,402: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 3 tasked)
[2017-06-15 18:00:46,501: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 18:00:46,501: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 18:00:46,501: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 18:00:46,501: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 18:00:46,501: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e894a48e-1a15-41aa-91ac-a9a1db591e24
[2017-06-15 18:00:46,501: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e894a48e-1a15-41aa-91ac-a9a1db591e24
[2017-06-15 18:00:46,505: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e894a48e-1a15-41aa-91ac-a9a1db591e24
[2017-06-15 18:00:46,505: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e894a48e-1a15-41aa-91ac-a9a1db591e24
[2017-06-15 18:00:46,506: INFO/job_scheduler] Job 294 finished 1 steps successfully
[2017-06-15 18:00:46,564: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 140623538067472
[2017-06-15 18:00:46,567: INFO/job] Job 294: StateChangeJob complete, setting state configured on lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration
[2017-06-15 18:00:46,568: INFO/job] Job 294: not_deleted=True
[2017-06-15 18:00:46,568: INFO/job_scheduler] Job 294: complete_job: Updating cache
[2017-06-15 18:00:46,568: INFO/job_scheduler] Job 294: locked_item 140623538067472 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-06-15 18:00:46,571: INFO/job_scheduler] Job 294 completing (errored=False, cancelled=False)
[2017-06-15 18:00:46,583: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-15 18:00:47,470: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 18:00:47,470: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 18:00:47,471: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 18:00:47,471: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 18:00:47,471: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 69152e0e-ffd7-4864-95db-4b5c1d8e8895
[2017-06-15 18:00:47,471: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 69152e0e-ffd7-4864-95db-4b5c1d8e8895
[2017-06-15 18:00:47,506: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 69152e0e-ffd7-4864-95db-4b5c1d8e8895
[2017-06-15 18:00:47,506: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 69152e0e-ffd7-4864-95db-4b5c1d8e8895
[2017-06-15 18:00:47,507: INFO/job_scheduler] Job 292 finished 1 steps successfully
[2017-06-15 18:00:47,605: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 140623740989968
[2017-06-15 18:00:47,609: INFO/job] Job 292: StateChangeJob complete, setting state configured on lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-06-15 18:00:47,609: INFO/job] Job 292: not_deleted=True
[2017-06-15 18:00:47,610: INFO/job_scheduler] Job 292: complete_job: Updating cache
[2017-06-15 18:00:47,610: INFO/job_scheduler] Job 292: locked_item 140623740989968 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-06-15 18:00:47,612: INFO/job_scheduler] Job 292 completing (errored=False, cancelled=False)
[2017-06-15 18:00:47,624: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 1 tasked)
[2017-06-15 18:00:57,554: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 18:00:57,554: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 18:00:57,555: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 18:00:57,555: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 18:00:57,555: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b9dc609d-bb63-4a06-ae62-67664eada9a7
[2017-06-15 18:00:57,555: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b9dc609d-bb63-4a06-ae62-67664eada9a7
[2017-06-15 18:00:57,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b9dc609d-bb63-4a06-ae62-67664eada9a7
[2017-06-15 18:00:57,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b9dc609d-bb63-4a06-ae62-67664eada9a7
[2017-06-15 18:00:57,590: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.225', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-06-15 18:00:57,605: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 94dc67b3-fc40-4cc7-8fc6-120db75ce9b1
[2017-06-15 18:00:57,605: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 94dc67b3-fc40-4cc7-8fc6-120db75ce9b1
[2017-06-15 18:00:57,673: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 18:00:57,673: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 18:00:57,673: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 18:00:57,673: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 18:00:57,673: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 94dc67b3-fc40-4cc7-8fc6-120db75ce9b1
[2017-06-15 18:00:57,673: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 94dc67b3-fc40-4cc7-8fc6-120db75ce9b1
[2017-06-15 18:00:57,720: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 94dc67b3-fc40-4cc7-8fc6-120db75ce9b1
[2017-06-15 18:00:57,720: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 94dc67b3-fc40-4cc7-8fc6-120db75ce9b1
[2017-06-15 18:00:57,740: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 62665, 'pcs_password': 'beyJ4qQuu7rNAngkw1Xp'}
[2017-06-15 18:00:57,756: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2c74516b-c40b-4933-bdc5-76717ffb90a0
[2017-06-15 18:00:57,756: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2c74516b-c40b-4933-bdc5-76717ffb90a0
[2017-06-15 18:01:09,880: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 18:01:09,880: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 18:01:09,880: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 18:01:09,880: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 18:01:09,880: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2c74516b-c40b-4933-bdc5-76717ffb90a0
[2017-06-15 18:01:09,880: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2c74516b-c40b-4933-bdc5-76717ffb90a0
[2017-06-15 18:01:09,926: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2c74516b-c40b-4933-bdc5-76717ffb90a0
[2017-06-15 18:01:09,926: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2c74516b-c40b-4933-bdc5-76717ffb90a0
[2017-06-15 18:01:09,927: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': False, 'mcast_port': 62665, 'new_node_fqdn': u'lotus-32vm6.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'beyJ4qQuu7rNAngkw1Xp'}
[2017-06-15 18:01:09,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5b65667a-c845-4368-b098-e48dec5fd014
[2017-06-15 18:01:09,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5b65667a-c845-4368-b098-e48dec5fd014
[2017-06-15 18:01:16,046: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 18:01:16,046: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 18:01:16,046: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 18:01:16,046: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/f8d44926-1882-444f-a432-7e6b1869f211
[2017-06-15 18:01:16,046: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5b65667a-c845-4368-b098-e48dec5fd014
[2017-06-15 18:01:16,046: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5b65667a-c845-4368-b098-e48dec5fd014
[2017-06-15 18:01:16,057: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5b65667a-c845-4368-b098-e48dec5fd014
[2017-06-15 18:01:16,057: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5b65667a-c845-4368-b098-e48dec5fd014
[2017-06-15 18:01:16,058: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-06-15 18:01:16.058080+00:00:
[2017-06-15 18:01:16,058: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-06-15 18:01:16.058080+00:00:
[2017-06-15 18:01:16,058: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port 62665->62665
[2017-06-15 18:01:16,058: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port 62665->62665
[2017-06-15 18:01:16,063: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces [<NetworkInterface: lotus-32vm6.lotus.hpdd.lab.intel.com-eth1>, <NetworkInterface: lotus-32vm6.lotus.hpdd.lab.intel.com-eth0>]->[u'eth0', u'eth1']
[2017-06-15 18:01:16,063: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces [<NetworkInterface: lotus-32vm6.lotus.hpdd.lab.intel.com-eth1>, <NetworkInterface: lotus-32vm6.lotus.hpdd.lab.intel.com-eth0>]->[u'eth0', u'eth1']
[2017-06-15 18:01:16,080: INFO/job_scheduler] Job 293 finished 1 steps successfully
[2017-06-15 18:01:16,080: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 18:01:16,080: INFO/job_scheduler]   Job 293 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-06-15 18:01:16,081: INFO/job_scheduler]   Job 295 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-06-15 18:01:16,081: INFO/job_scheduler]   Job 296 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 18:01:16,083: INFO/job_scheduler]   Job 298 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 18:01:16,083: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 1 tasked)
[2017-06-15 18:01:16,104: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 140623739437584
[2017-06-15 18:01:16,111: INFO/job] Job 293: StateChangeJob complete, setting state stopped on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 18:01:16,111: INFO/job] Job 293: not_deleted=True
[2017-06-15 18:01:16,111: INFO/job_scheduler] Job 293: complete_job: Updating cache
[2017-06-15 18:01:16,111: INFO/job_scheduler] Job 293: locked_item 140623739437584 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-15 18:01:16,116: INFO/job_scheduler] Job 293 completing (errored=False, cancelled=False)
[2017-06-15 18:01:16,130: INFO/job_scheduler] run_next: 1 runnable jobs of (4 pending, 0 tasked)
[2017-06-15 18:01:16,134: INFO/job_scheduler] Job 295: RunJobThread.run
[2017-06-15 18:01:16,135: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-06-15 18:01:16,152: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fa4f2424-b157-4d76-8c6f-d342c03e589d
[2017-06-15 18:01:16,152: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fa4f2424-b157-4d76-8c6f-d342c03e589d
[2017-06-15 18:01:24,252: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 18:01:24,252: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 18:01:24,252: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 18:01:24,252: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 18:01:24,253: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fa4f2424-b157-4d76-8c6f-d342c03e589d
[2017-06-15 18:01:24,253: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fa4f2424-b157-4d76-8c6f-d342c03e589d
[2017-06-15 18:01:24,268: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fa4f2424-b157-4d76-8c6f-d342c03e589d
[2017-06-15 18:01:24,268: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fa4f2424-b157-4d76-8c6f-d342c03e589d
[2017-06-15 18:01:24,268: INFO/job_scheduler] Job 295 finished 1 steps successfully
[2017-06-15 18:01:24,311: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 140623739437584
[2017-06-15 18:01:24,367: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm6.lotus.hpdd.lab.intel.com at severity 30
[2017-06-15 18:01:24,384: INFO/job] Job 295: StateChangeJob complete, setting state started on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 18:01:24,384: INFO/job] Job 295: not_deleted=True
[2017-06-15 18:01:24,384: INFO/job_scheduler] Job 295: complete_job: Updating cache
[2017-06-15 18:01:24,384: INFO/job_scheduler] Job 295: locked_item 140623739437584 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-15 18:01:24,394: INFO/job_scheduler] Job 295 completing (errored=False, cancelled=False)
[2017-06-15 18:01:24,427: INFO/job_scheduler] run_next: 1 runnable jobs of (3 pending, 0 tasked)
[2017-06-15 18:01:24,442: INFO/job_scheduler] Job 296: RunJobThread.run
[2017-06-15 18:01:24,443: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-06-15 18:01:24,484: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 65eb68d3-7d59-45a0-a405-a0104c6044f7
[2017-06-15 18:01:24,484: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 65eb68d3-7d59-45a0-a405-a0104c6044f7
[2017-06-15 18:01:26,035: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-15 18:01:26,035: INFO/job_scheduler]   Job 296 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 18:01:26,035: INFO/job_scheduler]   Job 298 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-15 18:01:26,035: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-15 18:01:29,110: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-15 18:01:29,128: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-15 18:01:29,160: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 5 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from True to False
[2017-06-15 18:01:29,166: INFO/job_scheduler] _notify: Set corosync_reported_up=False on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-5) and saved
[2017-06-15 18:01:29,171: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-15 18:01:31,594: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 18:01:31,594: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 18:01:31,594: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 18:01:31,594: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/d23a9aa9-b4fa-47bb-b576-fd1a180ca8e8
[2017-06-15 18:01:31,594: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 65eb68d3-7d59-45a0-a405-a0104c6044f7
[2017-06-15 18:01:31,594: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 65eb68d3-7d59-45a0-a405-a0104c6044f7
[2017-06-15 18:01:31,600: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 65eb68d3-7d59-45a0-a405-a0104c6044f7
[2017-06-15 18:01:31,600: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 65eb68d3-7d59-45a0-a405-a0104c6044f7
[2017-06-15 18:01:31,600: INFO/job_scheduler] Job 296 finished 1 steps successfully
[2017-06-15 18:01:31,665: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 140623740622928
[2017-06-15 18:01:31,704: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm6.lotus.hpdd.lab.intel.com at severity 30
[2017-06-15 18:01:31,710: INFO/job] Job 296: StateChangeJob complete, setting state started on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-15 18:01:31,710: INFO/job] Job 296: not_deleted=True
[2017-06-15 18:01:31,711: INFO/job_scheduler] Job 296: complete_job: Updating cache
[2017-06-15 18:01:31,711: INFO/job_scheduler] Job 296: locked_item 140623740622928 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-15 18:01:31,722: INFO/job_scheduler] Job 296 completing (errored=False, cancelled=False)
[2017-06-15 18:01:31,734: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 0 tasked)
[2017-06-15 18:01:31,786: INFO/job_scheduler] Job 297: open DB connection during completion
[2017-06-15 18:01:31,791: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->monitored (intentional=True) 140623740090000
[2017-06-15 18:01:31,800: INFO/job] Job 297: StateChangeJob complete, setting state monitored on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-15 18:01:31,800: INFO/job] Job 297: not_deleted=True
[2017-06-15 18:01:31,800: INFO/job_scheduler] Job 297: complete_job: Updating cache
[2017-06-15 18:01:31,800: INFO/job_scheduler] Job 297: locked_item 140623740090000 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-15 18:01:31,807: INFO/job_scheduler] Job 297 completing (errored=False, cancelled=False)
[2017-06-15 18:01:31,838: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-15 18:01:31,847: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration in state lnet_unloaded, not one of ['unconfigured'] (preferred unconfigured)
[2017-06-15 18:01:31,879: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration in state lnet_unloaded, not one of ['unconfigured'] (preferred unconfigured)
[2017-06-15 18:01:31,889: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration in state started, not one of ['unconfigured'] (preferred unconfigured)
[2017-06-15 18:01:31,891: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration in state configured, not one of ['unconfigured'] (preferred unconfigured)
[2017-06-15 18:01:31,897: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration in state configured, not one of ['unconfigured'] (preferred unconfigured)
[2017-06-15 18:01:31,898: WARNING/job_scheduler] Job 298: cancelling because of failed dependency
[2017-06-15 18:01:31,899: INFO/job_scheduler] Job 298 completing (errored=False, cancelled=True)
[2017-06-15 18:01:31,984: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:01:36,070: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 5 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-06-15 18:01:36,076: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-5) and saved
[2017-06-15 18:01:36,082: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:01:36,107: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 6 (lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-06-15 18:01:36,112: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-6) and saved
[2017-06-15 18:01:36,120: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:01:39,160: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:01:39,180: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:01:50,396: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:01:50,409: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:01:53,146: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:01:53,159: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:04:06,472: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:04:06,510: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:11:50,788: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:11:50,813: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:12:02,125: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:12:02,152: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:12:04,861: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:12:04,881: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:14:18,053: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:14:18,075: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:22:02,283: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:22:02,298: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:22:13,883: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:22:13,905: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:22:16,582: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:22:16,604: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:24:29,785: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:24:29,800: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:32:13,835: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:32:13,853: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:32:25,522: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:32:25,536: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:32:28,454: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:32:28,467: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:34:41,536: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:34:41,554: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:42:25,385: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:42:25,403: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:42:37,268: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:42:37,286: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:42:40,220: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:42:40,240: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:44:53,343: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:44:53,364: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:52:36,981: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:52:36,993: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:52:48,913: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:52:48,935: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:52:51,975: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:52:51,991: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:55:05,126: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 18:55:05,140: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:02:48,847: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:02:48,864: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:03:00,699: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:03:00,725: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:03:03,656: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:03:03,677: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:05:16,807: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:05:16,847: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:13:00,536: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:13:00,555: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:13:12,549: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:13:12,571: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:13:15,553: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:13:15,572: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:15:28,475: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:15:28,496: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:23:12,147: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:23:12,163: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:23:24,208: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:23:24,225: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:23:27,283: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:23:27,325: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:25:40,115: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:25:40,141: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:33:23,954: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:33:23,969: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:33:36,013: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:33:36,034: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:33:39,019: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:33:39,039: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:35:51,934: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:35:51,953: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:43:35,584: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:43:35,597: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:43:47,737: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:43:47,753: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:43:50,692: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:43:50,718: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:46:03,743: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:46:03,770: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:53:47,223: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:53:47,240: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:53:59,736: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:53:59,756: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:54:02,341: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:54:02,354: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:56:15,416: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 19:56:15,433: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:03:58,932: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:03:58,958: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:04:11,516: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:04:11,533: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:04:14,327: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:04:14,357: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:06:27,234: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:06:27,251: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:14:10,598: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:14:10,620: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:14:23,207: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:14:23,235: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:14:26,043: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:14:26,059: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:16:38,925: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:16:38,941: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:24:22,219: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:24:22,246: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:24:34,869: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:24:34,889: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:24:37,888: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:24:37,906: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:26:50,512: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:26:50,527: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:34:33,867: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:34:33,896: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:34:46,707: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:34:46,735: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:34:49,638: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:34:49,666: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:37:02,278: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:37:02,295: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:44:45,626: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:44:45,645: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:44:58,537: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:44:58,561: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:45:01,373: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:45:01,404: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:47:14,083: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:47:14,099: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:54:57,279: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:54:57,294: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:55:10,456: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:55:10,475: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:55:13,203: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:55:13,219: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:57:25,809: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 20:57:25,832: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:05:09,057: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:05:09,069: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:05:22,228: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:05:22,244: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:05:24,943: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:05:24,959: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:07:37,471: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:07:37,487: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:15:20,717: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:15:20,734: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:15:33,952: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:15:34,061: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:15:36,555: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:15:36,568: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:17:49,273: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:17:49,297: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:25:32,351: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:25:32,365: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:25:45,593: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:25:45,605: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:25:48,195: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:25:48,212: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:28:00,902: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:28:00,921: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:35:44,037: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:35:44,052: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:35:57,389: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:35:57,408: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:35:59,899: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:35:59,916: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:38:12,672: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:38:12,688: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:45:55,548: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:45:55,577: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:46:09,227: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:46:09,244: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:46:11,662: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:46:11,678: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:48:24,474: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:48:24,485: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:56:07,155: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:56:07,176: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:56:20,873: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:56:20,887: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:56:23,330: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:56:23,345: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:58:36,084: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 21:58:36,109: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:06:18,770: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:06:18,787: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:06:32,686: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:06:32,727: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:06:35,025: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:06:35,044: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:08:47,742: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:08:47,759: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:16:30,435: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:16:30,462: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:16:44,521: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:16:44,536: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:16:46,638: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:16:46,653: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:18:59,493: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:18:59,511: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:26:42,097: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:26:42,117: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:26:56,322: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:26:56,334: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:26:58,464: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:26:58,498: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:29:11,192: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:29:11,207: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:36:53,704: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:36:53,726: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:37:08,093: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:37:08,106: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:37:10,121: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:37:10,133: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:39:22,869: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:39:22,887: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:47:05,375: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:47:05,395: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:47:19,837: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:47:19,850: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:47:21,823: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:47:21,841: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:49:34,474: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:49:34,490: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:57:16,914: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:57:16,933: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:57:31,548: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:57:31,564: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:57:33,643: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:57:33,664: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:59:46,157: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 22:59:46,177: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:07:28,539: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:07:28,551: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:07:43,052: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:07:43,067: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:07:45,270: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:07:45,291: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:09:57,966: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:09:57,985: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:17:40,022: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:17:40,041: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:17:54,538: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:17:54,554: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:17:56,916: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:17:56,937: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:20:09,633: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:20:09,655: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:27:51,446: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:27:51,463: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:28:06,239: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:28:06,293: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:28:08,629: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:28:08,644: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:30:21,256: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:30:21,272: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:38:03,053: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:38:03,070: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:38:17,711: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:38:17,750: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:38:20,458: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:38:20,474: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:40:32,756: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:40:32,778: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:48:14,656: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:48:14,679: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:48:29,350: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:48:29,375: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:48:32,119: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:48:32,140: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:50:44,303: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:50:44,326: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:58:26,278: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:58:26,295: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:58:41,110: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:58:41,140: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:58:43,913: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-15 23:58:43,931: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:00:55,902: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:00:55,921: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:08:37,927: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:08:37,939: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:08:52,868: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:08:52,886: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:08:55,729: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:08:55,750: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:11:07,740: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:11:07,754: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:18:49,597: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:18:49,611: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:19:04,775: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:19:04,789: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:19:07,448: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:19:07,463: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:21:19,496: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:21:19,522: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:29:01,466: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:29:01,498: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:29:16,638: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:29:16,661: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:29:19,030: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:29:19,048: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:31:31,230: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:31:31,247: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:39:13,118: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:39:13,134: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:39:28,236: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:39:28,257: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:39:30,675: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:39:30,692: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:41:42,844: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:41:42,858: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:49:24,645: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:49:24,662: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:49:40,106: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:49:40,128: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:49:42,392: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:49:42,409: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:51:54,540: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:51:54,560: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:59:36,312: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:59:36,335: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:59:51,737: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:59:51,760: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:59:54,177: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 00:59:54,258: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:02:06,074: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:02:06,087: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:09:47,860: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:09:47,888: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:10:03,423: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:10:03,439: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:10:05,866: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:10:05,880: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:12:17,613: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:12:17,628: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:19:59,451: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:19:59,467: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:20:15,202: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:20:15,216: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:20:17,629: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:20:17,665: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:22:29,182: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:22:29,202: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:30:11,048: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:30:11,067: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:30:27,023: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:30:27,045: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:30:29,414: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:30:29,428: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:32:40,931: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:32:40,947: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:40:22,743: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:40:22,770: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:40:38,854: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:40:38,867: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:40:41,077: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:40:41,091: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:42:52,667: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:42:52,684: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:50:34,471: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:50:34,492: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:50:50,666: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:50:50,697: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:50:52,679: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:50:52,722: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:53:04,328: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 01:53:04,354: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:00:46,279: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:00:46,293: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:01:02,304: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:01:02,322: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:01:04,424: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:01:04,456: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:03:16,083: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:03:16,108: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:10:58,160: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:10:58,176: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:11:13,884: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:11:13,905: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:11:16,126: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:11:16,142: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:13:27,730: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:13:27,752: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:21:09,746: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:21:09,762: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:21:25,571: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:21:25,592: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:21:27,741: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:21:27,764: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:23:39,482: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:23:39,497: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:31:21,283: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:31:21,298: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:31:37,266: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:31:37,278: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:31:39,547: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:31:39,560: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:33:51,137: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:33:51,158: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:41:32,796: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:41:32,812: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:41:48,859: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:41:48,878: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:41:51,085: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:41:51,101: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:44:02,732: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:44:02,746: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:51:44,483: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:51:44,499: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:52:00,560: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:52:00,577: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:52:02,713: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:52:02,728: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:54:14,225: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 02:54:14,239: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:01:56,174: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:01:56,204: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:02:12,296: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:02:12,312: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:02:14,446: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:02:14,474: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:04:25,792: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:04:25,806: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:12:07,747: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:12:07,760: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:12:23,979: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:12:23,996: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:12:25,956: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:12:25,975: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:14:37,487: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:14:37,506: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:22:19,276: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:22:19,294: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:22:35,667: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:22:35,684: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:22:37,731: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:22:37,749: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:24:49,129: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:24:49,144: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:32:30,903: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:32:30,919: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:32:47,388: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:32:47,409: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:32:49,412: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:32:49,449: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:35:00,834: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:35:00,853: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:42:42,441: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:42:42,459: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:42:58,915: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:42:58,935: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:43:01,036: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:43:01,051: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:45:12,538: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:45:12,553: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:52:53,989: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:52:54,005: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:53:10,545: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:53:10,564: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:53:12,706: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:53:12,722: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:55:24,221: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 03:55:24,238: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:03:05,564: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:03:05,585: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:03:22,089: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:03:22,107: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:03:24,384: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:03:24,404: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:05:35,885: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:05:35,905: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:13:17,172: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:13:17,188: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:13:33,602: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:13:33,618: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:13:36,180: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:13:36,200: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:15:47,394: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:15:47,413: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:23:28,753: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:23:28,783: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:23:45,326: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:23:45,343: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:23:47,928: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:23:47,941: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:25:58,907: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:25:58,924: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:33:40,401: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:33:40,416: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:33:57,070: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:33:57,111: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:34:00,012: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:34:00,035: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:36:10,508: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:36:10,529: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:43:52,015: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:43:52,038: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:44:08,926: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:44:08,962: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:44:11,493: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:44:11,517: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:46:22,118: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:46:22,182: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:54:03,709: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:54:03,723: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:54:20,732: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:54:20,748: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:54:23,200: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:54:23,214: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:56:33,749: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 04:56:33,765: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:04:15,479: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:04:15,496: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:04:32,481: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:04:32,495: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:04:34,901: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:04:34,917: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:06:45,386: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:06:45,418: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:14:27,105: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:14:27,123: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:14:44,259: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:14:44,280: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:14:46,561: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:14:46,579: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:16:57,061: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:16:57,076: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:24:38,714: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:24:38,731: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:24:55,915: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:24:55,935: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:24:58,399: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:24:58,413: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:27:08,870: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:27:08,885: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:34:50,255: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:34:50,268: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:35:07,464: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:35:07,481: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:35:10,007: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:35:10,024: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:37:20,664: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:37:20,683: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:45:01,895: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:45:01,912: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:45:19,049: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:45:19,064: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:45:21,677: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:45:21,692: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:47:32,300: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:47:32,350: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:55:13,454: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:55:13,468: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:55:30,672: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:55:30,690: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:55:33,415: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:55:33,431: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:57:44,121: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 05:57:44,135: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:05:25,098: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:05:25,114: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:05:42,484: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:05:42,496: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:05:45,054: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:05:45,074: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:07:55,754: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:07:55,773: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:15:36,932: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:15:36,947: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:15:54,255: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:15:54,276: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:15:56,716: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:15:56,728: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:18:07,303: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:18:07,321: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:25:48,581: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:25:48,604: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:26:06,090: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:26:06,139: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:26:08,363: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:26:08,393: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:28:18,920: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:28:18,941: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:36:00,282: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:36:00,295: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:36:17,863: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:36:17,881: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:36:20,151: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:36:20,173: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:38:30,550: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:38:30,564: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:46:11,942: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:46:11,963: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:46:29,623: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:46:29,752: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:46:31,969: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:46:31,985: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:48:42,262: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:48:42,276: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:56:23,508: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:56:23,524: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:56:41,409: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:56:41,428: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:56:43,697: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:56:43,723: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:58:54,059: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 06:58:54,082: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:06:35,160: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:06:35,192: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:06:52,925: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:06:52,952: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:06:55,460: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:06:55,486: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:09:05,757: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:09:05,777: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:16:46,805: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:16:46,820: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:17:04,543: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:17:04,560: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:17:07,134: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:17:07,150: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:19:17,482: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:19:17,501: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:26:58,364: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:26:58,378: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:27:16,089: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:27:16,104: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:27:18,732: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:27:18,746: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:29:29,152: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:29:29,167: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:37:10,051: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:37:10,096: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:37:27,858: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:37:27,870: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:37:30,510: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:37:30,537: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:39:40,817: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:39:40,838: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:47:21,702: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:47:21,723: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:47:39,618: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:47:39,631: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:47:42,087: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:47:42,115: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:49:52,414: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:49:52,429: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:57:33,329: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:57:33,346: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:57:51,350: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:57:51,364: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:57:53,713: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 07:57:53,739: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:00:03,934: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:00:03,952: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:07:44,931: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:07:44,945: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:08:03,063: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:08:03,088: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:08:05,473: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:08:05,498: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:10:15,614: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:10:15,633: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:17:56,283: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:17:56,297: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:18:14,626: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:18:14,665: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:18:17,142: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:18:17,159: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:20:27,242: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:20:27,255: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:28:07,843: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:28:07,860: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:28:26,099: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:28:26,113: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:28:28,712: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:28:28,732: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:30:38,961: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:30:38,981: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:38:19,545: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:38:19,557: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:38:37,679: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:38:37,694: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:38:40,427: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:38:40,444: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:40:50,689: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:40:50,706: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:48:31,184: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:48:31,201: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:48:49,328: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:48:49,342: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:48:51,987: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:48:51,999: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:51:02,253: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:51:02,272: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:58:42,787: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:58:42,805: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:59:01,008: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:59:01,023: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:59:03,762: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 08:59:03,777: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:01:13,912: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:01:13,935: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:08:54,298: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:08:54,315: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:09:12,801: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:09:12,815: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:09:15,496: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:09:15,513: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:11:25,527: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:11:25,545: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:19:05,955: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:19:05,968: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:19:24,502: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:19:24,514: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:19:27,308: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:19:27,323: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:21:36,995: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:21:37,008: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:29:17,520: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:29:17,538: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:29:36,234: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:29:36,249: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:29:38,925: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:29:38,943: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:31:48,518: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:31:48,533: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:39:29,421: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:39:29,437: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:39:47,996: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:39:48,011: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:39:50,570: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:39:50,590: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:42:00,092: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:42:00,106: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:49:40,985: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:49:41,005: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:49:59,780: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:49:59,800: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:50:02,401: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:50:02,525: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:52:11,628: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:52:11,651: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:59:52,668: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 09:59:52,689: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:00:11,506: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:00:11,521: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:00:14,114: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:00:14,134: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:02:23,296: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:02:23,316: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:10:04,356: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:10:04,371: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:10:23,077: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:10:23,091: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:10:25,837: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:10:25,855: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:12:34,896: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:12:34,915: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:20:15,918: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:20:15,940: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:20:34,735: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:20:34,752: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:20:37,710: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:20:37,729: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:22:46,548: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:22:46,565: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:30:27,452: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:30:27,471: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:30:46,514: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:30:46,526: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:30:49,452: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:30:49,468: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:32:58,132: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:32:58,145: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:40:39,127: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:40:39,143: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:40:58,261: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:40:58,276: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:41:01,069: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:41:01,085: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:43:09,828: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:43:09,845: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:50:50,929: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:50:50,943: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:51:10,076: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:51:10,097: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:51:12,911: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:51:12,925: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:53:21,515: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 10:53:21,528: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:01:02,567: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:01:02,580: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:01:21,888: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:01:21,923: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:01:24,622: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:01:24,639: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:03:33,035: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:03:33,053: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:11:14,257: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:11:14,277: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:11:33,629: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:11:33,656: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:11:36,342: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:11:36,355: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:13:44,568: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:13:44,581: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:21:26,042: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:21:26,057: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:21:45,518: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:21:45,538: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:21:48,089: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:21:48,104: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:23:56,088: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:23:56,104: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:31:37,643: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:31:37,667: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:31:57,275: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:31:57,289: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:31:59,780: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:31:59,797: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:34:07,933: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:34:07,953: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:41:49,331: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:41:49,353: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:42:08,923: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:42:08,940: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:42:11,485: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:42:11,498: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:44:19,859: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:44:19,877: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:52:00,793: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:52:00,810: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:52:20,520: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:52:20,542: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:52:23,175: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:52:23,189: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:54:31,688: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 11:54:31,703: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:02:12,368: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:02:12,383: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:02:31,988: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:02:32,005: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:02:34,852: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:02:34,866: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:04:43,447: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:04:43,464: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:12:23,881: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:12:23,903: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:12:43,637: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:12:43,677: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:12:46,458: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:12:46,472: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:14:55,169: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:14:55,198: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:22:35,318: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:22:35,332: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:22:55,305: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:22:55,326: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:22:58,200: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:22:58,213: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:25:06,925: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:25:06,944: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:32:46,760: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:32:46,776: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:33:07,068: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:33:07,082: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:33:09,902: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:33:09,922: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:35:18,613: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:35:18,634: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:42:58,419: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:42:58,432: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:43:18,625: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:43:18,714: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:43:21,431: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:43:21,447: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:45:30,215: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:45:30,230: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:53:09,982: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:53:09,998: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:53:30,425: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:53:30,444: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:53:33,060: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:53:33,078: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:55:41,982: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 12:55:41,993: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:03:21,565: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:03:21,599: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:03:42,107: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:03:42,126: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:03:44,697: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:03:44,713: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:05:53,740: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:05:53,755: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:13:33,268: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:13:33,283: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:13:53,623: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:13:53,638: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:13:56,411: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:13:56,425: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:16:05,562: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:16:05,651: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:23:44,988: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:23:45,012: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:24:05,264: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:24:05,282: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:24:08,082: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:24:08,107: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:26:17,212: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:26:17,227: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:33:56,758: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:33:56,779: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:34:16,923: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:34:16,939: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:34:19,745: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:34:19,762: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:36:29,020: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:36:29,034: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:44:08,419: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:44:08,436: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:44:28,744: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:44:28,757: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:44:31,498: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:44:31,514: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:46:40,631: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:46:40,645: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:54:19,913: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:54:19,929: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:54:40,461: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:54:40,483: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:54:43,231: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:54:43,249: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:56:52,199: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 13:56:52,225: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:04:31,478: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:04:31,492: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:04:52,234: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:04:52,248: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:04:54,932: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:04:54,945: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:07:04,080: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:07:04,102: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:14:42,997: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:14:43,017: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:15:04,003: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:15:04,023: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:15:06,670: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:15:06,694: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:17:15,506: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:17:15,525: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:24:54,714: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:24:54,746: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:25:15,737: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:25:15,754: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:25:18,381: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:25:18,406: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:27:27,111: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:27:27,125: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:35:06,340: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:35:06,367: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:35:27,543: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:35:27,565: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:35:30,070: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:35:30,092: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:37:38,769: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:37:38,790: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:45:17,881: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:45:17,897: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:45:39,346: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:45:39,393: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:45:41,858: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:45:41,876: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:47:50,293: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:47:50,306: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:55:29,601: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:55:29,616: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:55:51,069: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:55:51,103: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:55:53,563: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:55:53,577: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:58:01,962: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 14:58:01,978: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:05:41,223: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:05:41,240: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:06:02,747: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:06:02,765: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:06:05,126: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:06:05,146: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:08:13,646: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:08:13,676: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:15:52,899: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:15:52,916: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:16:14,319: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:16:14,334: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:16:16,763: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:16:16,776: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:18:25,292: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:18:25,314: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:26:04,428: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:26:04,449: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:26:26,229: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:26:26,245: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:26:28,389: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:26:28,404: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:28:36,868: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:28:36,895: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:36:16,063: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:36:16,084: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:36:37,951: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:36:37,966: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:36:40,377: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:36:40,390: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:38:48,397: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:38:48,411: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:46:27,827: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:46:27,841: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:46:49,821: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:46:49,840: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:46:52,117: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:46:52,136: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:48:59,952: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:48:59,974: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:56:39,493: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:56:39,508: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:57:01,631: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:57:01,646: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:57:03,784: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:57:03,799: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:59:11,633: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 15:59:11,652: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:06:51,065: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:06:51,078: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:07:13,350: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:07:13,368: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:07:15,526: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:07:15,545: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:09:23,211: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:09:23,226: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:17:02,829: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:17:02,846: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:17:24,954: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:17:24,968: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:17:27,241: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:17:27,257: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:19:34,777: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:19:34,792: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:27:14,421: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:27:14,440: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:27:36,682: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:27:36,700: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:27:38,918: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:27:38,932: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:29:46,322: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:29:46,338: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:37:26,064: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:37:26,088: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:37:48,326: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:37:48,346: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:37:50,549: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:37:50,568: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:39:57,998: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:39:58,059: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:47:37,633: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:47:37,653: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:47:59,955: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:47:59,982: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:48:02,244: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:48:02,258: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:50:09,748: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:50:09,771: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:57:49,205: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:57:49,226: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:58:11,779: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:58:11,792: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:58:14,027: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 16:58:14,047: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:00:21,478: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:00:21,490: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:08:00,906: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:08:00,923: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:08:23,438: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:08:23,462: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:08:25,807: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:08:25,828: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:10:33,100: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:10:33,116: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:18:12,522: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:18:12,554: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:18:35,159: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:18:35,175: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:18:37,433: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:18:37,450: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:20:44,964: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:20:44,980: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:28:24,078: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:28:24,094: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:28:46,831: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:28:46,850: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:28:49,102: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:28:49,116: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:30:56,631: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:30:56,710: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:38:35,641: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:38:35,656: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:38:58,341: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:38:58,359: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:39:00,785: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:39:00,799: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:41:08,225: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:41:08,259: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:48:47,206: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:48:47,221: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:49:10,071: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:49:10,099: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:49:12,346: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:49:12,372: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:51:19,818: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:51:19,838: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:58:58,803: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:58:58,823: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:59:21,781: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:59:21,795: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:59:24,051: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 17:59:24,076: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:01:31,447: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:01:31,462: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:09:10,527: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:09:10,548: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:09:33,393: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:09:33,414: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:09:35,820: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:09:35,837: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:11:43,243: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:11:43,258: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:19:22,130: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:19:22,148: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:19:44,982: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:19:45,005: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:19:47,441: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:19:47,462: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:21:55,081: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:21:55,097: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:29:33,821: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:29:33,838: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:29:56,558: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:29:56,574: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:29:59,042: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:29:59,057: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:32:06,727: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:32:06,754: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:39:45,526: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:39:45,547: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:40:08,358: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:40:08,375: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:40:10,933: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:40:10,949: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:42:18,425: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:42:18,442: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:49:57,101: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:49:57,119: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:50:20,143: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:50:20,157: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:50:22,669: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:50:22,690: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:52:30,078: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 18:52:30,095: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:00:08,818: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:00:08,836: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:00:31,920: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:00:31,934: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:00:34,342: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:00:34,368: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:02:41,742: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:02:41,765: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:10:20,507: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:10:20,528: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:10:43,774: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:10:43,791: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:10:45,930: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:10:45,951: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:12:53,389: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:12:53,413: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:20:32,229: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:20:32,244: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:20:55,480: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:20:55,494: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:20:57,544: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:20:57,563: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:23:04,938: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:23:04,953: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:30:43,819: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:30:43,844: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:31:07,148: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:31:07,163: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:31:09,314: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:31:09,328: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:33:16,656: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:33:16,671: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:40:55,394: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:40:55,408: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:41:18,805: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:41:18,824: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:41:21,047: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:41:21,063: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:43:28,347: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:43:28,362: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:51:06,966: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:51:06,985: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:51:30,652: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:51:30,673: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:51:32,666: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:51:32,690: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:53:40,007: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 19:53:40,023: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:01:18,711: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:01:18,746: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:01:42,361: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:01:42,376: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:01:44,559: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:01:44,576: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:03:51,727: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:03:51,746: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:11:30,439: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:11:30,455: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:11:54,077: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:11:54,089: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:11:56,275: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:11:56,290: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:14:03,380: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:14:03,396: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:21:42,054: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:21:42,083: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:22:06,081: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:22:06,102: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:22:08,006: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:22:08,030: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:24:14,977: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:24:14,992: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:31:53,602: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:31:53,618: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:32:18,092: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:32:18,105: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:32:19,820: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:32:19,838: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:34:26,538: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:34:26,553: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:42:05,130: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:42:05,143: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:42:29,950: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:42:29,965: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:42:31,492: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:42:31,532: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:44:38,097: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:44:38,117: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:52:16,716: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:52:16,733: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:52:41,662: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:52:41,712: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:52:43,083: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:52:43,104: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:54:49,660: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 20:54:49,675: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:02:28,295: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:02:28,317: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:02:53,344: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:02:53,364: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:02:54,916: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:02:54,931: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:05:01,191: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:05:01,209: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:12:39,953: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:12:39,997: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:13:04,952: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:13:04,968: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:13:06,643: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:13:06,685: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:15:12,594: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:15:12,613: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:22:51,660: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:22:51,686: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:23:16,455: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:23:16,467: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:23:18,262: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:23:18,279: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:25:24,136: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:25:24,151: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:33:03,272: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:33:03,421: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:33:28,086: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:33:28,100: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:33:30,066: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:33:30,085: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:35:35,764: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:35:35,779: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:43:14,831: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:43:14,848: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:43:39,765: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:43:39,784: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:43:41,758: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:43:41,776: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:45:47,464: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:45:47,481: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:53:26,466: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:53:26,481: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:53:51,401: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:53:51,412: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:53:53,469: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:53:53,485: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:55:59,205: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 21:55:59,224: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:03:37,959: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:03:37,979: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:04:03,091: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:04:03,107: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:04:05,156: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:04:05,180: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:06:10,984: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:06:11,001: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:13:49,390: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:13:49,410: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:14:14,780: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:14:14,800: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:14:16,757: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:14:16,774: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:16:22,575: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:16:22,591: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:24:00,901: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:24:00,916: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:24:26,360: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:24:26,380: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:24:28,362: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:24:28,383: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:26:34,128: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:26:34,147: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:34:12,478: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:34:12,497: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:34:37,995: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:34:38,011: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:34:39,992: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:34:40,011: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:36:45,652: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:36:45,668: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:44:24,218: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:44:24,229: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:44:49,658: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:44:49,674: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:44:51,581: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:44:51,598: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:46:57,228: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:46:57,249: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:54:35,788: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:54:35,817: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:55:01,279: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:55:01,295: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:55:03,195: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:55:03,228: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:57:09,154: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 22:57:09,172: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:04:47,553: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:04:47,571: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:05:12,988: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:05:13,006: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:05:14,834: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:05:14,849: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:07:20,899: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:07:20,915: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:14:59,182: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:14:59,203: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:15:24,734: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:15:24,751: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:15:26,499: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:15:26,517: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:17:32,511: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:17:32,526: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:25:10,872: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:25:10,887: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:25:36,495: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:25:36,511: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:25:38,231: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:25:38,244: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:27:44,123: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:27:44,143: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:35:22,560: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:35:22,575: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:35:48,219: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:35:48,236: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:35:49,839: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:35:49,861: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:37:55,753: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:37:55,768: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:45:34,141: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:45:34,172: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:46:00,153: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:46:00,170: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:46:01,401: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:46:01,419: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:48:07,696: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:48:07,717: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:55:45,702: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:55:45,725: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:56:11,962: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:56:11,979: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:56:13,047: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:56:13,063: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:58:19,357: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-16 23:58:19,371: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:05:57,299: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:05:57,322: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:06:23,760: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:06:23,773: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:06:24,713: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:06:24,730: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:08:30,935: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:08:30,955: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:16:09,139: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:16:09,153: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:16:35,598: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:16:35,620: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:16:36,497: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:16:36,512: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:18:42,502: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:18:42,530: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:26:20,885: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:26:20,901: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:26:47,369: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:26:47,387: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:26:48,263: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:26:48,279: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:28:54,024: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:28:54,037: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:36:32,512: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:36:32,527: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:36:59,294: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:36:59,313: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:36:59,983: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:36:59,996: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:39:05,641: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:39:05,679: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:46:44,218: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:46:44,230: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:47:11,048: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:47:11,060: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:47:11,595: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:47:11,609: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:49:17,228: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:49:17,247: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:56:55,908: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:56:55,946: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:57:22,861: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:57:22,882: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:57:23,253: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:57:23,274: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:59:29,098: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 00:59:29,121: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:07:07,523: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:07:07,546: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:07:34,549: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:07:34,567: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:07:35,022: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:07:35,049: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:09:40,863: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:09:40,881: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:17:19,226: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:17:19,248: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:17:46,475: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:17:46,491: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:17:46,845: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:17:46,858: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:19:52,448: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:19:52,461: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:27:30,822: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:27:30,836: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:27:58,144: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:27:58,166: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:27:58,581: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:27:58,599: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:30:04,134: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:30:04,156: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:37:42,461: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:37:42,481: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:38:09,949: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:38:09,972: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:38:10,373: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:38:10,389: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:40:15,810: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:40:15,824: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:47:54,093: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:47:54,109: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:48:21,724: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:48:21,746: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:48:22,117: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:48:22,129: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:50:27,367: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:50:27,386: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:58:05,836: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:58:05,856: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:58:33,345: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:58:33,360: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:58:33,869: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 01:58:33,885: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:00:38,949: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:00:38,971: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:08:17,412: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:08:17,440: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:08:44,973: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:08:44,986: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:08:45,523: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:08:45,534: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:10:50,567: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:10:50,581: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:18:29,042: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:18:29,057: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:18:56,621: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:18:56,635: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:18:57,322: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:18:57,343: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:21:02,315: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:21:02,333: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:28:40,659: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:28:40,673: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:29:08,428: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:29:08,442: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:29:09,072: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:29:09,091: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:31:13,950: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:31:13,965: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:38:52,156: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:38:52,173: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:39:20,330: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:39:20,345: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:39:20,921: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:39:20,939: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:41:25,534: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:41:25,554: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:49:03,676: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:49:03,692: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:49:32,004: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:49:32,020: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:49:32,683: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:49:32,701: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:51:37,323: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:51:37,339: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:59:15,305: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:59:15,316: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:59:43,724: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:59:43,744: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:59:44,340: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 02:59:44,354: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:01:48,981: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:01:48,994: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:09:26,815: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:09:26,936: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:09:55,563: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:09:55,581: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:09:56,123: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:09:56,138: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:12:00,697: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:12:00,712: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:19:38,401: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:19:38,415: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:20:07,283: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:20:07,302: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:20:07,816: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:20:07,834: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:22:12,173: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:22:12,202: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:29:49,907: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:29:49,921: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:30:18,937: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:30:18,957: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:30:19,482: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:30:19,507: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:32:23,610: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:32:23,629: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:40:01,561: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:40:01,575: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:40:30,652: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:40:30,673: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:40:31,313: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:40:31,326: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:42:35,194: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:42:35,209: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:50:13,090: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:50:13,105: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:50:42,526: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:50:42,549: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:50:42,968: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:50:42,984: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:52:46,781: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 03:52:46,797: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:00:24,931: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:00:24,951: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:00:54,327: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:00:54,344: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:00:54,697: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:00:54,711: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:02:58,407: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:02:58,428: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:10:36,674: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:10:36,699: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:11:06,071: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:11:06,094: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:11:06,414: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:11:06,441: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:13:09,935: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:13:09,958: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:20:48,448: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:20:48,463: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:21:17,786: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:21:17,802: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:21:18,110: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:21:18,144: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:23:21,727: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:23:21,744: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:31:00,006: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:31:00,028: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:31:29,520: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:31:29,546: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:31:29,785: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:31:29,810: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:33:33,392: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:33:33,406: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:41:11,553: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:41:11,570: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:41:41,322: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:41:41,341: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:41:41,603: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:41:41,622: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:43:45,042: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:43:45,058: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:51:23,180: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:51:23,201: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:51:52,980: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:51:52,994: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:51:53,256: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:51:53,266: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:53:56,679: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 04:53:56,696: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:01:34,856: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:01:34,874: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:02:04,702: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:02:04,718: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:02:04,865: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:02:04,893: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:04:08,318: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:04:08,332: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:11:46,464: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:11:46,479: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:12:16,343: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:12:16,364: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:12:16,518: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:12:16,598: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:14:19,925: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:14:19,945: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:21:58,120: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:21:58,137: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:22:28,258: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:22:28,293: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:22:28,320: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:22:28,332: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:24:31,831: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:24:31,849: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:32:09,984: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:32:10,007: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:32:40,118: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:32:40,132: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:32:40,227: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:32:40,242: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:34:43,466: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:34:43,482: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:42:21,633: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:42:21,657: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:42:51,820: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:42:51,838: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:42:52,145: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:42:52,157: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:44:55,300: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:44:55,316: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:52:33,268: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:52:33,287: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:53:03,636: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:53:03,653: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:53:04,019: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:53:04,037: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:55:07,043: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 05:55:07,055: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:02:45,020: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:02:45,046: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:03:15,275: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:03:15,290: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:03:15,805: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:03:15,828: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:05:18,845: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:05:18,862: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:12:56,596: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:12:56,619: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:13:26,965: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:13:26,986: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:13:27,492: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:13:27,510: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:15:30,450: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:15:30,469: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:23:08,323: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:23:08,347: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:23:38,678: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:23:38,702: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:23:39,173: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:23:39,191: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:25:42,048: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:25:42,071: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:33:19,951: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:33:19,968: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:33:50,198: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:33:50,214: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:33:50,779: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:33:50,795: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:35:53,860: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:35:53,882: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:43:31,536: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:43:31,551: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:44:01,928: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:44:01,945: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:44:02,806: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:44:02,835: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:46:05,454: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:46:05,474: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:53:43,087: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:53:43,109: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:54:13,682: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:54:13,700: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:54:14,516: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:54:14,559: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:56:17,086: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 06:56:17,107: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:03:54,776: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:03:54,799: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:04:25,434: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:04:25,449: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:04:26,208: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:04:26,224: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:06:28,755: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:06:28,769: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:14:06,415: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:14:06,444: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:14:37,234: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:14:37,249: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:14:38,036: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:14:38,050: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:16:40,385: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:16:40,412: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:24:18,056: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:24:18,071: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:24:49,102: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:24:49,123: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:24:49,733: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:24:49,756: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:26:52,177: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:26:52,195: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:34:29,754: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:34:29,772: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:35:00,780: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:35:00,797: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:35:01,508: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:35:01,532: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:37:04,037: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:37:04,052: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:44:41,418: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:44:41,430: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:45:12,622: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:45:12,635: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:45:13,144: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:45:13,160: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:47:15,925: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:47:15,945: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:54:53,004: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:54:53,019: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:55:24,408: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:55:24,429: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:55:24,781: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:55:24,795: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:57:27,515: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 07:57:27,529: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:05:04,556: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:05:04,569: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:05:36,211: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:05:36,236: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:05:36,443: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:05:36,458: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:07:39,141: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:07:39,158: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:15:16,104: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:15:16,120: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:15:48,076: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:15:48,095: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:15:48,142: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:15:48,160: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:17:50,686: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:17:50,700: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:25:27,710: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:25:27,730: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:25:59,847: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:25:59,861: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:25:59,917: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:25:59,936: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:28:02,266: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:28:02,280: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:35:39,219: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:35:39,236: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:36:11,679: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:36:11,696: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:36:11,735: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:36:11,750: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:38:13,923: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:38:13,944: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:45:50,827: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:45:50,843: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:46:23,456: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:46:23,475: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:46:23,622: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:46:23,635: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:48:25,281: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:48:25,301: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:56:02,572: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:56:02,590: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:56:35,358: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:56:35,378: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:56:35,414: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:56:35,430: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:58:36,861: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 08:58:36,876: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:06:14,177: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:06:14,194: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:06:46,901: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:06:46,914: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:06:47,374: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:06:47,386: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:08:48,577: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:08:48,593: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:16:25,754: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:16:25,769: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:16:58,547: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:16:58,561: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:16:59,072: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:16:59,087: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:19:00,291: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:19:00,306: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:26:37,304: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:26:37,332: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:27:10,195: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:27:10,216: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:27:10,719: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:27:10,733: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:29:11,863: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:29:11,890: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:36:49,127: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:36:49,148: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:37:21,831: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:37:21,856: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:37:22,473: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:37:22,489: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:39:23,422: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:39:23,437: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:47:00,697: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:47:00,720: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:47:33,468: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:47:33,481: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:47:34,146: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:47:34,164: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:49:35,046: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:49:35,059: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:57:12,296: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:57:12,313: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:57:45,277: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:57:45,288: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:57:45,917: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:57:45,939: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:59:46,653: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 09:59:46,666: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:07:24,159: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:07:24,181: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:07:57,113: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:07:57,134: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:07:57,781: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:07:57,796: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:09:58,183: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:09:58,200: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:17:35,866: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:17:35,915: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:18:08,926: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:18:08,942: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:18:09,667: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:18:09,699: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:20:09,805: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:20:09,820: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:27:47,697: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:27:47,710: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:28:20,633: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:28:20,665: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:28:21,412: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:28:21,432: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:30:21,626: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:30:21,667: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:37:59,382: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:37:59,402: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:38:32,326: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:38:32,341: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:38:33,053: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:38:33,070: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:40:33,472: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:40:33,490: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:48:10,980: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:48:11,003: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:48:44,052: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:48:44,067: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:48:44,503: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:48:44,516: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:50:45,161: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:50:45,242: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:58:22,528: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:58:22,542: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:58:55,827: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:58:55,848: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:58:56,194: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 10:58:56,210: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:00:56,934: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:00:56,955: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:08:34,113: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:08:34,130: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:09:07,542: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:09:07,559: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:09:07,911: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:09:07,926: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:11:08,702: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:11:08,757: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:18:45,687: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:18:45,790: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:19:19,229: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:19:19,245: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:19:19,668: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:19:19,682: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:21:20,298: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:21:20,314: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:28:57,321: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:28:57,340: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:29:30,971: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:29:30,988: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:29:31,498: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:29:31,525: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:31:32,125: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:31:32,141: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:39:08,935: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:39:08,951: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:39:42,786: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:39:42,801: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:39:43,257: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:39:43,275: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:41:43,910: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:41:43,930: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:49:20,478: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:49:20,497: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:49:54,578: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:49:54,593: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:49:55,182: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:49:55,199: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:51:55,510: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:51:55,531: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:59:32,033: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 11:59:32,051: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:00:06,511: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:00:06,529: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:00:07,004: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:00:07,020: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:02:07,090: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:02:07,105: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:09:43,663: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:09:43,677: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:10:18,303: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:10:18,318: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:10:18,746: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:10:18,767: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:12:18,755: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:12:18,775: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:19:55,145: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:19:55,162: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:20:30,032: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:20:30,046: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:20:30,650: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:20:30,686: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:22:30,673: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:22:30,701: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:30:06,810: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:30:06,831: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:30:41,688: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:30:41,714: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:30:42,470: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:30:42,485: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:32:42,466: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:32:42,539: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:40:18,456: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:40:18,471: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:40:53,426: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:40:53,492: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:40:54,156: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:40:54,168: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:42:54,140: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:42:54,155: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:50:29,955: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:50:29,965: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:51:05,084: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:51:05,098: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:51:06,329: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:51:06,345: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:53:05,884: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 12:53:05,900: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 13:00:41,687: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 13:00:41,744: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 13:01:16,778: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 13:01:16,798: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 13:01:18,171: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 13:01:18,189: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 13:03:17,631: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 13:03:17,646: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 13:10:53,358: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 13:10:53,380: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-17 13:11:01,763: INFO/chroma_service] Stopping job_scheduler
[2017-06-17 13:11:01,764: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-06-17 13:11:01,764: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-06-17 13:11:01,764: INFO/queue] Stopping ServiceQueue agent_action_runner_rx
[2017-06-17 13:11:02,222: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-06-17 13:11:02,222: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-06-17 13:11:02,223: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-06-17 13:11:02,223: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-06-17 13:11:02,223: INFO/job_scheduler] Stopping...
[2017-06-17 13:11:02,223: INFO/queue] Stopping ServiceQueue job_scheduler_notifications
[2017-06-17 13:11:02,223: INFO/job_scheduler] Joining...
[2017-06-17 13:11:02,767: INFO/job_scheduler] Complete.
[2017-06-17 13:11:02,767: INFO/chroma_service] Joining job_scheduler
[2017-06-17 13:11:02,767: INFO/job_scheduler] Cancelling outstanding jobs...
[2017-06-17 13:12:42,525: INFO/queue] Purged 0 messages from 'job_scheduler_notifications' queue
[2017-06-17 13:12:42,603: INFO/queue] Purged 0 messages from 'agent_action_runner_rx' queue
[2017-06-17 13:12:42,888: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: None/None
[2017-06-17 13:12:42,888: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: None/None
[2017-06-29 09:49:14,395: INFO/chroma_service] Stopping job_scheduler
[2017-06-29 09:49:14,396: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-06-29 09:49:14,396: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-06-29 09:49:14,396: INFO/queue] Stopping ServiceQueue agent_action_runner_rx
[2017-06-29 09:49:14,583: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-06-29 09:49:14,583: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-06-29 09:49:14,583: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-06-29 09:49:14,583: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-06-29 09:49:14,584: INFO/job_scheduler] Stopping...
[2017-06-29 09:49:14,584: INFO/queue] Stopping ServiceQueue job_scheduler_notifications
[2017-06-29 09:49:14,584: INFO/job_scheduler] Joining...
[2017-06-29 09:49:15,582: INFO/job_scheduler] Complete.
[2017-06-29 09:49:15,582: INFO/chroma_service] Joining job_scheduler
[2017-06-29 09:49:15,582: INFO/job_scheduler] Cancelling outstanding jobs...
[2017-06-29 09:50:50,582: INFO/queue] Purged 0 messages from 'job_scheduler_notifications' queue
[2017-06-29 09:50:50,595: INFO/queue] Purged 1 messages from 'agent_action_runner_rx' queue
[2017-06-29 10:24:44,760: INFO/chroma_service] Stopping job_scheduler
[2017-06-29 10:24:44,760: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-06-29 10:24:44,760: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-06-29 10:24:44,767: INFO/queue] Stopping ServiceQueue agent_action_runner_rx
[2017-06-29 10:24:44,821: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-06-29 10:24:44,821: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-06-29 10:24:44,821: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-06-29 10:24:44,821: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-06-29 10:24:44,821: INFO/job_scheduler] Stopping...
[2017-06-29 10:24:44,822: INFO/queue] Stopping ServiceQueue job_scheduler_notifications
[2017-06-29 10:24:44,822: INFO/job_scheduler] Joining...
[2017-06-29 10:24:45,813: INFO/job_scheduler] Complete.
[2017-06-29 10:24:45,813: INFO/chroma_service] Joining job_scheduler
[2017-06-29 10:24:45,813: INFO/job_scheduler] Cancelling outstanding jobs...
[2017-06-29 10:24:45,815: ERROR/chroma_service] Rogue thread still running, exiting hard
[2017-06-29 10:24:45,815: ERROR/chroma_service] ['MainThread', 'Thread-1813', 'Thread-1825', 'Thread-1814', 'Thread-1824', 'Thread-1822', 'Thread-1819', 'Thread-1823']
[2017-06-29 10:34:26,000: INFO/queue] Purged 0 messages from 'job_scheduler_notifications' queue
[2017-06-29 10:34:26,035: INFO/queue] Purged 0 messages from 'agent_action_runner_rx' queue
[2017-06-29 10:34:26,237: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: None/None
[2017-06-29 10:34:26,237: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: None/None
[2017-06-29 10:36:02,189: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 10:36:02,189: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-29 10:36:02,201: INFO/command_plan] add_jobs: created Job 1 (Test for host connectivity)
[2017-06-29 10:36:02,225: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-29 10:36:02,255: INFO/job_scheduler] Job 1: RunJobThread.run
[2017-06-29 10:36:56,325: INFO/job_scheduler] Job 1 finished 1 steps successfully
[2017-06-29 10:36:56,365: INFO/job_scheduler] Job 1: complete_job: Updating cache
[2017-06-29 10:36:56,365: INFO/job_scheduler] Job 1 completing (errored=False, cancelled=False)
[2017-06-29 10:36:56,398: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 10:36:56,567: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 10:36:56,569: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-29 10:36:56,581: INFO/command_plan] add_jobs: created Job 2 (Test for host connectivity)
[2017-06-29 10:36:56,614: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-29 10:36:56,635: INFO/job_scheduler] Job 2: RunJobThread.run
[2017-06-29 10:37:02,567: INFO/job_scheduler] Job 2 finished 1 steps successfully
[2017-06-29 10:37:02,622: INFO/job_scheduler] Job 2: complete_job: Updating cache
[2017-06-29 10:37:02,622: INFO/job_scheduler] Job 2 completing (errored=False, cancelled=False)
[2017-06-29 10:37:02,655: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 10:37:03,804: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 10:37:03,806: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-29 10:37:03,813: INFO/command_plan] add_jobs: created Job 3 (Test for host connectivity)
[2017-06-29 10:37:03,872: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-29 10:37:03,884: INFO/job_scheduler] Job 3: RunJobThread.run
[2017-06-29 10:37:05,583: INFO/job_scheduler] Getting FQDN for 'lotus-32vm5.lotus.hpdd.lab.intel.com': lotus-32vm5
lotus-32vm5.lotus.hpdd.lab.intel.com

[2017-06-29 10:37:05,620: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 10:37:05,623: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-1 transitioning from undeployed to unconfigured
[2017-06-29 10:37:05,678: INFO/command_plan] Created command 4 (Deploying agent to host lotus-32vm5.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-06-29 10:37:05,697: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-29 10:37:05,762: INFO/job_scheduler] Job 4: RunJobThread.run
[2017-06-29 10:37:06,624: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-29 10:37:09,945: INFO/job_scheduler] Job 3 finished 1 steps successfully
[2017-06-29 10:37:09,970: INFO/job_scheduler] Job 3: complete_job: Updating cache
[2017-06-29 10:37:09,970: INFO/job_scheduler] Job 3 completing (errored=False, cancelled=False)
[2017-06-29 10:37:10,006: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-29 10:37:15,060: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 10:37:15,060: INFO/job_scheduler]   Job 4 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-06-29 10:37:15,060: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-29 10:37:15,108: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bf466257-29cb-4bb4-9ee2-5a5902bb98da
[2017-06-29 10:37:15,108: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bf466257-29cb-4bb4-9ee2-5a5902bb98da
[2017-06-29 10:37:15,163: INFO/job_scheduler] Job 4 finished 1 steps successfully
[2017-06-29 10:37:15,184: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 140667359829072
[2017-06-29 10:37:15,190: INFO/job] Job 4: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:37:15,190: INFO/job] Job 4: not_deleted=True
[2017-06-29 10:37:15,190: INFO/job_scheduler] Job 4: complete_job: Updating cache
[2017-06-29 10:37:15,190: INFO/job_scheduler] Job 4: locked_item 140667359829072 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-29 10:37:15,193: INFO/job_scheduler] Job 4 completing (errored=False, cancelled=False)
[2017-06-29 10:37:15,242: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-1
[2017-06-29 10:37:15,242: INFO/job_scheduler] _notify: Updating .boot_time of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com) from None to 2017-06-23 08:35:25+00:00
[2017-06-29 10:37:15,247: INFO/job_scheduler] _notify: Set boot_time=2017-06-23 08:35:25+00:00 on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-1) and saved
[2017-06-29 10:37:15,253: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 10:37:20,823: INFO/job_scheduler] _notify: Updating .properties of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.21.1.el7_lustre.x86_64", "zfs_installed": false, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-06-29 10:37:20,852: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.21.1.el7_lustre.x86_64", "zfs_installed": false, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-1) and saved
[2017-06-29 10:37:20,860: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 10:37:20,934: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 10:37:32,127: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 10:37:32,132: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-1 transitioning from unconfigured to managed
[2017-06-29 10:37:32,295: INFO/command_plan] Created command 5 (Setup managed host lotus-32vm5.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-06-29 10:37:32,358: INFO/job_scheduler] run_next: 1 runnable jobs of (11 pending, 0 tasked)
[2017-06-29 10:37:32,391: INFO/job_scheduler] Job 5: RunJobThread.run
[2017-06-29 10:37:32,392: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-06-29 10:37:32,432: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b5fb2130-04ee-4c2b-b085-c0b2536f0bc5
[2017-06-29 10:37:32,432: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b5fb2130-04ee-4c2b-b085-c0b2536f0bc5
[2017-06-29 10:38:11,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bf466257-29cb-4bb4-9ee2-5a5902bb98da
[2017-06-29 10:38:11,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bf466257-29cb-4bb4-9ee2-5a5902bb98da
[2017-06-29 10:38:11,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bf466257-29cb-4bb4-9ee2-5a5902bb98da
[2017-06-29 10:38:11,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bf466257-29cb-4bb4-9ee2-5a5902bb98da
[2017-06-29 10:38:11,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b5fb2130-04ee-4c2b-b085-c0b2536f0bc5
[2017-06-29 10:38:11,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b5fb2130-04ee-4c2b-b085-c0b2536f0bc5
[2017-06-29 10:38:11,568: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b5fb2130-04ee-4c2b-b085-c0b2536f0bc5
[2017-06-29 10:38:11,568: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b5fb2130-04ee-4c2b-b085-c0b2536f0bc5
[2017-06-29 10:38:11,569: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-06-29 10:38:11.569105+00:00:
[2017-06-29 10:38:11,569: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-06-29 10:38:11.569105+00:00:
[2017-06-29 10:38:11,569: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-06-29 10:38:11,569: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-06-29 10:38:11,583: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-06-29 10:38:11.583942+00:00:
[2017-06-29 10:38:11,583: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-06-29 10:38:11.583942+00:00:
[2017-06-29 10:38:11,584: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-06-29 10:38:11,584: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-06-29 10:38:11,592: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 10:38:11,592: INFO/job_scheduler]   Job 5 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-29 10:38:11,592: INFO/job_scheduler]   Job 6 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:38:11,594: INFO/job_scheduler]   Job 7 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:38:11,594: INFO/job_scheduler]   Job 8 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:38:11,595: INFO/job_scheduler]   Job 9 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:38:11,596: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com restart_agent {}
[2017-06-29 10:38:11,604: INFO/job_scheduler]   Job 12 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:38:11,604: INFO/job_scheduler]   Job 15 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-29 10:38:11,605: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 1 tasked)
[2017-06-29 10:38:11,608: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 10:38:11,613: INFO/job_scheduler]   Job 5 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-29 10:38:11,613: INFO/job_scheduler]   Job 6 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:38:11,613: INFO/job_scheduler]   Job 7 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:38:11,613: INFO/job_scheduler]   Job 8 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:38:11,614: INFO/job_scheduler]   Job 9 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:38:11,614: INFO/job_scheduler]   Job 12 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:38:11,614: INFO/job_scheduler]   Job 15 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-29 10:38:11,614: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 1 tasked)
[2017-06-29 10:38:11,624: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b4259a74-8cf9-45a3-8331-e0c24bcc155d
[2017-06-29 10:38:11,624: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b4259a74-8cf9-45a3-8331-e0c24bcc155d
[2017-06-29 10:38:11,714: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bf466257-29cb-4bb4-9ee2-5a5902bb98da
[2017-06-29 10:38:11,714: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bf466257-29cb-4bb4-9ee2-5a5902bb98da
[2017-06-29 10:38:11,714: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bf466257-29cb-4bb4-9ee2-5a5902bb98da
[2017-06-29 10:38:11,714: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/bf466257-29cb-4bb4-9ee2-5a5902bb98da
[2017-06-29 10:38:11,714: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b4259a74-8cf9-45a3-8331-e0c24bcc155d
[2017-06-29 10:38:11,714: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b4259a74-8cf9-45a3-8331-e0c24bcc155d
[2017-06-29 10:38:11,738: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b4259a74-8cf9-45a3-8331-e0c24bcc155d
[2017-06-29 10:38:11,738: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b4259a74-8cf9-45a3-8331-e0c24bcc155d
[2017-06-29 10:38:11,738: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old bf466257-29cb-4bb4-9ee2-5a5902bb98da)
[2017-06-29 10:38:11,738: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old bf466257-29cb-4bb4-9ee2-5a5902bb98da)
[2017-06-29 10:38:30,415: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bf466257-29cb-4bb4-9ee2-5a5902bb98da
[2017-06-29 10:38:30,415: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/bf466257-29cb-4bb4-9ee2-5a5902bb98da
[2017-06-29 10:38:30,415: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session bf466257-29cb-4bb4-9ee2-5a5902bb98da because session terminated
[2017-06-29 10:38:30,415: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session bf466257-29cb-4bb4-9ee2-5a5902bb98da because session terminated
[2017-06-29 10:38:30,417: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/22b803d1-017d-4318-b441-4119346958ed
[2017-06-29 10:38:30,417: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/22b803d1-017d-4318-b441-4119346958ed
[2017-06-29 10:38:30,759: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new 22b803d1-017d-4318-b441-4119346958ed
[2017-06-29 10:38:30,759: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new 22b803d1-017d-4318-b441-4119346958ed
[2017-06-29 10:38:30,796: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-29 10:38:30,813: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fae711fb-a98c-4ba9-ace3-70b0b8978342
[2017-06-29 10:38:30,813: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fae711fb-a98c-4ba9-ace3-70b0b8978342
[2017-06-29 10:38:30,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/22b803d1-017d-4318-b441-4119346958ed
[2017-06-29 10:38:30,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/22b803d1-017d-4318-b441-4119346958ed
[2017-06-29 10:38:30,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/22b803d1-017d-4318-b441-4119346958ed
[2017-06-29 10:38:30,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/22b803d1-017d-4318-b441-4119346958ed
[2017-06-29 10:38:30,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fae711fb-a98c-4ba9-ace3-70b0b8978342
[2017-06-29 10:38:30,908: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fae711fb-a98c-4ba9-ace3-70b0b8978342
[2017-06-29 10:38:30,927: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fae711fb-a98c-4ba9-ace3-70b0b8978342
[2017-06-29 10:38:30,927: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fae711fb-a98c-4ba9-ace3-70b0b8978342
[2017-06-29 10:38:30,928: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-06-29 10:38:30,949: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 33a2f71c-544d-41de-92f6-b6918d70af8f
[2017-06-29 10:38:30,949: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 33a2f71c-544d-41de-92f6-b6918d70af8f
[2017-06-29 10:38:34,660: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 10:38:34,660: INFO/job_scheduler]   Job 5 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-29 10:38:34,661: INFO/job_scheduler]   Job 6 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:38:34,661: INFO/job_scheduler]   Job 7 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:38:34,661: INFO/job_scheduler]   Job 8 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:38:34,661: INFO/job_scheduler]   Job 9 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:38:34,661: INFO/job_scheduler]   Job 12 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:38:34,661: INFO/job_scheduler]   Job 15 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-29 10:38:34,661: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 1 tasked)
[2017-06-29 10:39:03,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/22b803d1-017d-4318-b441-4119346958ed
[2017-06-29 10:39:03,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/22b803d1-017d-4318-b441-4119346958ed
[2017-06-29 10:39:03,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/22b803d1-017d-4318-b441-4119346958ed
[2017-06-29 10:39:03,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/22b803d1-017d-4318-b441-4119346958ed
[2017-06-29 10:39:03,923: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 33a2f71c-544d-41de-92f6-b6918d70af8f
[2017-06-29 10:39:03,923: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 33a2f71c-544d-41de-92f6-b6918d70af8f
[2017-06-29 10:39:03,931: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 33a2f71c-544d-41de-92f6-b6918d70af8f
[2017-06-29 10:39:03,931: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 33a2f71c-544d-41de-92f6-b6918d70af8f
[2017-06-29 10:39:04,362: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com install_packages {'repos': [u'iml-agent'], 'packages': [u'zfs', u'zfs-dkms', u'kernel-devel-[0-9]*_lustre', u'kernel-[0-9]*_lustre', u'lustre-osd-zfs-mount', u'kmod-lustre-osd-ldiskfs', u'lustre-dkms', u'lustre', u'chroma-agent-management']}
[2017-06-29 10:39:04,378: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 121f5493-8a92-4961-b7f2-19e4df7c8211
[2017-06-29 10:39:04,378: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 121f5493-8a92-4961-b7f2-19e4df7c8211
[2017-06-29 10:50:26,489: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 10:50:26,489: INFO/job_scheduler]   Job 5 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-29 10:50:26,489: INFO/job_scheduler]   Job 6 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:50:26,489: INFO/job_scheduler]   Job 7 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:50:26,489: INFO/job_scheduler]   Job 8 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:50:26,489: INFO/job_scheduler]   Job 9 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:50:26,489: INFO/job_scheduler]   Job 12 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:50:26,490: INFO/job_scheduler]   Job 15 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-29 10:50:26,490: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 1 tasked)
[2017-06-29 10:52:50,597: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/22b803d1-017d-4318-b441-4119346958ed
[2017-06-29 10:52:50,597: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/22b803d1-017d-4318-b441-4119346958ed
[2017-06-29 10:52:50,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/22b803d1-017d-4318-b441-4119346958ed
[2017-06-29 10:52:50,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/22b803d1-017d-4318-b441-4119346958ed
[2017-06-29 10:52:50,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 121f5493-8a92-4961-b7f2-19e4df7c8211
[2017-06-29 10:52:50,598: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 121f5493-8a92-4961-b7f2-19e4df7c8211
[2017-06-29 10:52:50,620: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 121f5493-8a92-4961-b7f2-19e4df7c8211
[2017-06-29 10:52:50,620: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 121f5493-8a92-4961-b7f2-19e4df7c8211
[2017-06-29 10:52:50,665: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com kernel_status {}
[2017-06-29 10:52:50,688: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1a06f685-676f-435e-8c98-428ef8be8cbf
[2017-06-29 10:52:50,688: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1a06f685-676f-435e-8c98-428ef8be8cbf
[2017-06-29 10:52:52,815: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/22b803d1-017d-4318-b441-4119346958ed
[2017-06-29 10:52:52,815: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/22b803d1-017d-4318-b441-4119346958ed
[2017-06-29 10:52:52,815: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/22b803d1-017d-4318-b441-4119346958ed
[2017-06-29 10:52:52,815: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/22b803d1-017d-4318-b441-4119346958ed
[2017-06-29 10:52:52,815: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1a06f685-676f-435e-8c98-428ef8be8cbf
[2017-06-29 10:52:52,815: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1a06f685-676f-435e-8c98-428ef8be8cbf
[2017-06-29 10:52:52,855: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1a06f685-676f-435e-8c98-428ef8be8cbf
[2017-06-29 10:52:52,855: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1a06f685-676f-435e-8c98-428ef8be8cbf
[2017-06-29 10:52:52,855: INFO/job] Job 5 RebootIfNeededStep: Reboot of lotus-32vm5.lotus.hpdd.lab.intel.com required to switch from running kernel kernel-3.10.0-514.21.1.el7_lustre.x86_64 to required kernel-3.10.0-514.10.2.el7_lustre.x86_64
[2017-06-29 10:52:52,855: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com reboot_server {}
[2017-06-29 10:52:52,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 52fed012-294d-43ae-9ba0-f19f10177cba
[2017-06-29 10:52:52,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 52fed012-294d-43ae-9ba0-f19f10177cba
[2017-06-29 10:52:52,942: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/22b803d1-017d-4318-b441-4119346958ed
[2017-06-29 10:52:52,942: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/22b803d1-017d-4318-b441-4119346958ed
[2017-06-29 10:52:52,947: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/22b803d1-017d-4318-b441-4119346958ed
[2017-06-29 10:52:52,947: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/22b803d1-017d-4318-b441-4119346958ed
[2017-06-29 10:52:52,947: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 52fed012-294d-43ae-9ba0-f19f10177cba
[2017-06-29 10:52:52,947: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 52fed012-294d-43ae-9ba0-f19f10177cba
[2017-06-29 10:52:52,988: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 52fed012-294d-43ae-9ba0-f19f10177cba
[2017-06-29 10:52:52,988: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 52fed012-294d-43ae-9ba0-f19f10177cba
[2017-06-29 10:52:52,989: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old 22b803d1-017d-4318-b441-4119346958ed)
[2017-06-29 10:52:52,989: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old 22b803d1-017d-4318-b441-4119346958ed)
[2017-06-29 10:53:55,990: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/22b803d1-017d-4318-b441-4119346958ed
[2017-06-29 10:53:55,990: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/22b803d1-017d-4318-b441-4119346958ed
[2017-06-29 10:53:55,991: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 22b803d1-017d-4318-b441-4119346958ed because session terminated
[2017-06-29 10:53:55,991: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 22b803d1-017d-4318-b441-4119346958ed because session terminated
[2017-06-29 10:54:16,099: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 10:54:16,099: INFO/job_scheduler]   Job 5 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-29 10:54:16,099: INFO/job_scheduler]   Job 6 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:54:16,099: INFO/job_scheduler]   Job 7 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:54:16,099: INFO/job_scheduler]   Job 8 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:54:16,099: INFO/job_scheduler]   Job 9 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:54:16,099: INFO/job_scheduler]   Job 12 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:54:16,099: INFO/job_scheduler]   Job 15 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-29 10:54:16,100: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 1 tasked)
[2017-06-29 10:54:16,155: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:54:16,155: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:54:17,092: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:54:17,092: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:54:17,093: INFO/job_scheduler] Job 5 finished 4 steps successfully
[2017-06-29 10:54:17,149: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 140667829847120
[2017-06-29 10:54:17,157: INFO/job] Job 5: StateChangeJob complete, setting state packages_installed on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:54:17,157: INFO/job] Job 5: not_deleted=True
[2017-06-29 10:54:17,157: INFO/job_scheduler] Job 5: complete_job: Updating cache
[2017-06-29 10:54:17,157: INFO/job_scheduler] Job 5: locked_item 140667359942928 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-29 10:54:17,160: INFO/job_scheduler] Job 5 completing (errored=False, cancelled=False)
[2017-06-29 10:54:17,187: INFO/job_scheduler] run_next: 4 runnable jobs of (10 pending, 0 tasked)
[2017-06-29 10:54:17,251: INFO/job_scheduler] Job 6: RunJobThread.run
[2017-06-29 10:54:17,252: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-06-29 10:54:17,284: INFO/job_scheduler] Job 7: RunJobThread.run
[2017-06-29 10:54:17,285: INFO/job_scheduler] Job 8: open DB connection during completion
[2017-06-29 10:54:17,290: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-06-29 10:54:17,320: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2b9bc7d9-df6c-4ec1-a346-7c236a176bc0
[2017-06-29 10:54:17,320: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2b9bc7d9-df6c-4ec1-a346-7c236a176bc0
[2017-06-29 10:54:17,341: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5e0cf750-64b6-44d9-a27a-5a147ed2cfba
[2017-06-29 10:54:17,341: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5e0cf750-64b6-44d9-a27a-5a147ed2cfba
[2017-06-29 10:54:17,358: INFO/job_scheduler] Job 9: RunJobThread.run
[2017-06-29 10:54:17,358: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-06-29 10:54:17,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5249e0f8-25ca-4e85-bcd6-edad8490e53b
[2017-06-29 10:54:17,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5249e0f8-25ca-4e85-bcd6-edad8490e53b
[2017-06-29 10:54:17,432: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 140667359923408
[2017-06-29 10:54:17,449: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-06-29 10:54:17,454: INFO/job] Job 8: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-29 10:54:17,454: INFO/job] Job 8: not_deleted=True
[2017-06-29 10:54:17,454: INFO/job_scheduler] Job 8: complete_job: Updating cache
[2017-06-29 10:54:17,454: INFO/job_scheduler] Job 8: locked_item 140667359923408 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-29 10:54:17,457: INFO/job_scheduler] Job 8 completing (errored=False, cancelled=False)
[2017-06-29 10:54:17,528: INFO/job_scheduler] run_next: 1 runnable jobs of (6 pending, 3 tasked)
[2017-06-29 10:54:17,541: INFO/job_scheduler] Job 10: RunJobThread.run
[2017-06-29 10:54:17,542: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com load_lnet {}
[2017-06-29 10:54:17,567: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 17a62573-1025-4fd0-a71d-a07d7acd89ce
[2017-06-29 10:54:17,567: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 17a62573-1025-4fd0-a71d-a07d7acd89ce
[2017-06-29 10:54:17,773: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:54:17,773: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:54:17,774: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:54:17,774: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:54:17,774: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 17a62573-1025-4fd0-a71d-a07d7acd89ce
[2017-06-29 10:54:17,774: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 17a62573-1025-4fd0-a71d-a07d7acd89ce
[2017-06-29 10:54:17,789: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 17a62573-1025-4fd0-a71d-a07d7acd89ce
[2017-06-29 10:54:17,789: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 17a62573-1025-4fd0-a71d-a07d7acd89ce
[2017-06-29 10:54:17,790: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-29 10:54:17,823: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 31a1503b-a057-40b3-acd7-75911559618f
[2017-06-29 10:54:17,823: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 31a1503b-a057-40b3-acd7-75911559618f
[2017-06-29 10:54:18,894: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:54:18,894: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:54:18,894: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:54:18,894: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:54:18,894: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 31a1503b-a057-40b3-acd7-75911559618f
[2017-06-29 10:54:18,894: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 31a1503b-a057-40b3-acd7-75911559618f
[2017-06-29 10:54:18,941: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 31a1503b-a057-40b3-acd7-75911559618f
[2017-06-29 10:54:18,941: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 31a1503b-a057-40b3-acd7-75911559618f
[2017-06-29 10:54:19,436: INFO/job_scheduler] Job 10 finished 2 steps successfully
[2017-06-29 10:54:19,481: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 140667359923408
[2017-06-29 10:54:19,487: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-06-29 10:54:19,495: INFO/job] Job 10: StateChangeJob complete, setting state lnet_down on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-29 10:54:19,495: INFO/job] Job 10: not_deleted=True
[2017-06-29 10:54:19,495: INFO/job_scheduler] Job 10: complete_job: Updating cache
[2017-06-29 10:54:19,495: INFO/job_scheduler] Job 10: locked_item 140667359923408 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-29 10:54:19,512: INFO/job_scheduler] Job 10 completing (errored=False, cancelled=False)
[2017-06-29 10:54:19,541: INFO/job_scheduler] run_next: 1 runnable jobs of (5 pending, 3 tasked)
[2017-06-29 10:54:19,550: INFO/job_scheduler] Job 13: RunJobThread.run
[2017-06-29 10:54:19,551: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_lnet {}
[2017-06-29 10:54:19,570: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 69844dd1-6b6b-47a2-baf8-e01f77ebe87e
[2017-06-29 10:54:19,570: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 69844dd1-6b6b-47a2-baf8-e01f77ebe87e
[2017-06-29 10:54:19,641: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:54:19,641: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:54:19,642: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:54:19,642: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:54:19,642: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 69844dd1-6b6b-47a2-baf8-e01f77ebe87e
[2017-06-29 10:54:19,642: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 69844dd1-6b6b-47a2-baf8-e01f77ebe87e
[2017-06-29 10:54:19,656: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 69844dd1-6b6b-47a2-baf8-e01f77ebe87e
[2017-06-29 10:54:19,656: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 69844dd1-6b6b-47a2-baf8-e01f77ebe87e
[2017-06-29 10:54:19,657: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-29 10:54:19,671: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b9a22a28-4b5a-4700-97b8-fcdf5baf4981
[2017-06-29 10:54:19,671: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b9a22a28-4b5a-4700-97b8-fcdf5baf4981
[2017-06-29 10:54:19,765: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:54:19,765: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:54:19,765: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:54:19,765: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:54:19,765: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b9a22a28-4b5a-4700-97b8-fcdf5baf4981
[2017-06-29 10:54:19,765: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b9a22a28-4b5a-4700-97b8-fcdf5baf4981
[2017-06-29 10:54:19,789: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b9a22a28-4b5a-4700-97b8-fcdf5baf4981
[2017-06-29 10:54:19,789: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b9a22a28-4b5a-4700-97b8-fcdf5baf4981
[2017-06-29 10:54:19,920: INFO/job_scheduler] Job 13 finished 2 steps successfully
[2017-06-29 10:54:19,949: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 140667359923408
[2017-06-29 10:54:19,969: INFO/job] Job 13: StateChangeJob complete, setting state lnet_up on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-29 10:54:19,969: INFO/job] Job 13: not_deleted=True
[2017-06-29 10:54:19,969: INFO/job_scheduler] Job 13: complete_job: Updating cache
[2017-06-29 10:54:19,969: INFO/job_scheduler] Job 13: locked_item 140667359923408 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-29 10:54:19,978: INFO/job_scheduler] Job 13 completing (errored=False, cancelled=False)
[2017-06-29 10:54:19,996: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 3 tasked)
[2017-06-29 10:54:20,416: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 10:54:20,417: INFO/job_scheduler]   Job 6 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:54:20,417: INFO/job_scheduler]   Job 7 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:54:20,417: INFO/job_scheduler]   Job 9 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:54:20,417: INFO/job_scheduler]   Job 12 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:54:20,417: INFO/job_scheduler]   Job 15 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-29 10:54:20,417: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 3 tasked)
[2017-06-29 10:54:20,471: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 10:54:20,471: INFO/job_scheduler]   Job 6 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:54:20,471: INFO/job_scheduler]   Job 7 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:54:20,472: INFO/job_scheduler]   Job 9 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:54:20,472: INFO/job_scheduler]   Job 12 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:54:20,472: INFO/job_scheduler]   Job 15 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-29 10:54:20,472: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 3 tasked)
[2017-06-29 10:54:23,775: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:54:23,775: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:54:23,776: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:54:23,776: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:54:23,776: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5e0cf750-64b6-44d9-a27a-5a147ed2cfba
[2017-06-29 10:54:23,776: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5e0cf750-64b6-44d9-a27a-5a147ed2cfba
[2017-06-29 10:54:23,812: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5e0cf750-64b6-44d9-a27a-5a147ed2cfba
[2017-06-29 10:54:23,812: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5e0cf750-64b6-44d9-a27a-5a147ed2cfba
[2017-06-29 10:54:23,813: INFO/job_scheduler] Job 7 finished 1 steps successfully
[2017-06-29 10:54:23,865: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 140667359941584
[2017-06-29 10:54:23,868: INFO/job] Job 7: StateChangeJob complete, setting state configured on lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration
[2017-06-29 10:54:23,868: INFO/job] Job 7: not_deleted=True
[2017-06-29 10:54:23,868: INFO/job_scheduler] Job 7: complete_job: Updating cache
[2017-06-29 10:54:23,869: INFO/job_scheduler] Job 7: locked_item 140667359941584 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-06-29 10:54:23,871: INFO/job_scheduler] Job 7 completing (errored=False, cancelled=False)
[2017-06-29 10:54:23,882: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-29 10:54:24,769: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:54:24,769: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:54:24,770: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:54:24,770: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:54:24,770: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5249e0f8-25ca-4e85-bcd6-edad8490e53b
[2017-06-29 10:54:24,770: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5249e0f8-25ca-4e85-bcd6-edad8490e53b
[2017-06-29 10:54:24,800: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5249e0f8-25ca-4e85-bcd6-edad8490e53b
[2017-06-29 10:54:24,800: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5249e0f8-25ca-4e85-bcd6-edad8490e53b
[2017-06-29 10:54:24,801: INFO/job_scheduler] Job 9 finished 1 steps successfully
[2017-06-29 10:54:24,855: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 140667359941264
[2017-06-29 10:54:24,859: INFO/job] Job 9: StateChangeJob complete, setting state configured on lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-06-29 10:54:24,859: INFO/job] Job 9: not_deleted=True
[2017-06-29 10:54:24,859: INFO/job_scheduler] Job 9: complete_job: Updating cache
[2017-06-29 10:54:24,859: INFO/job_scheduler] Job 9: locked_item 140667359941264 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-06-29 10:54:24,862: INFO/job_scheduler] Job 9 completing (errored=False, cancelled=False)
[2017-06-29 10:54:24,873: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 1 tasked)
[2017-06-29 10:55:01,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:55:01,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:55:01,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:55:01,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:55:01,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2b9bc7d9-df6c-4ec1-a346-7c236a176bc0
[2017-06-29 10:55:01,043: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2b9bc7d9-df6c-4ec1-a346-7c236a176bc0
[2017-06-29 10:55:01,054: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2b9bc7d9-df6c-4ec1-a346-7c236a176bc0
[2017-06-29 10:55:01,054: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2b9bc7d9-df6c-4ec1-a346-7c236a176bc0
[2017-06-29 10:55:01,055: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.224', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-06-29 10:55:01,068: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a4f6628e-2b4a-48e4-bd10-b44c79ee4477
[2017-06-29 10:55:01,068: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a4f6628e-2b4a-48e4-bd10-b44c79ee4477
[2017-06-29 10:55:01,161: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:55:01,161: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:55:01,162: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:55:01,162: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:55:01,162: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a4f6628e-2b4a-48e4-bd10-b44c79ee4477
[2017-06-29 10:55:01,162: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a4f6628e-2b4a-48e4-bd10-b44c79ee4477
[2017-06-29 10:55:01,182: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a4f6628e-2b4a-48e4-bd10-b44c79ee4477
[2017-06-29 10:55:01,182: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a4f6628e-2b4a-48e4-bd10-b44c79ee4477
[2017-06-29 10:55:01,191: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 52477, 'pcs_password': 'Ir04VEKuz195MvUbfz2q'}
[2017-06-29 10:55:01,203: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 239902f2-38b9-4e17-b769-95759febe53d
[2017-06-29 10:55:01,203: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 239902f2-38b9-4e17-b769-95759febe53d
[2017-06-29 10:55:14,363: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:55:14,363: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:55:14,363: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:55:14,363: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:55:14,363: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 239902f2-38b9-4e17-b769-95759febe53d
[2017-06-29 10:55:14,363: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 239902f2-38b9-4e17-b769-95759febe53d
[2017-06-29 10:55:14,375: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 239902f2-38b9-4e17-b769-95759febe53d
[2017-06-29 10:55:14,375: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 239902f2-38b9-4e17-b769-95759febe53d
[2017-06-29 10:55:14,375: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': True, 'mcast_port': 52477, 'new_node_fqdn': u'lotus-32vm5.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'Ir04VEKuz195MvUbfz2q'}
[2017-06-29 10:55:14,394: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9277cc28-fcdc-4253-9689-ad80e3fadbcc
[2017-06-29 10:55:14,394: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9277cc28-fcdc-4253-9689-ad80e3fadbcc
[2017-06-29 10:55:20,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:55:20,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:55:20,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:55:20,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:55:20,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9277cc28-fcdc-4253-9689-ad80e3fadbcc
[2017-06-29 10:55:20,514: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9277cc28-fcdc-4253-9689-ad80e3fadbcc
[2017-06-29 10:55:20,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9277cc28-fcdc-4253-9689-ad80e3fadbcc
[2017-06-29 10:55:20,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9277cc28-fcdc-4253-9689-ad80e3fadbcc
[2017-06-29 10:55:20,563: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-06-29 10:55:20.563449+00:00:
[2017-06-29 10:55:20,563: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-06-29 10:55:20.563449+00:00:
[2017-06-29 10:55:20,563: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->52477
[2017-06-29 10:55:20,563: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->52477
[2017-06-29 10:55:20,566: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-06-29 10:55:20,566: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-06-29 10:55:20,591: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-29 10:55:20,591: INFO/job_scheduler]   Job 6 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-06-29 10:55:20,591: INFO/job_scheduler]   Job 11 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-06-29 10:55:20,591: INFO/job_scheduler]   Job 12 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 10:55:20,591: INFO/job_scheduler]   Job 14 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 10:55:20,591: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 1 tasked)
[2017-06-29 10:55:20,593: INFO/job_scheduler] Job 6 finished 1 steps successfully
[2017-06-29 10:55:20,601: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 140667359953296
[2017-06-29 10:55:20,609: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-06-29 10:55:20,619: INFO/job] Job 6: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 10:55:20,621: INFO/job] Job 6: not_deleted=True
[2017-06-29 10:55:20,622: INFO/job_scheduler] Job 6: complete_job: Updating cache
[2017-06-29 10:55:20,623: INFO/job_scheduler] Job 6: locked_item 140667359953296 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-29 10:55:20,627: INFO/job_scheduler] Job 6 completing (errored=False, cancelled=False)
[2017-06-29 10:55:20,646: INFO/job_scheduler] run_next: 1 runnable jobs of (4 pending, 0 tasked)
[2017-06-29 10:55:20,652: INFO/job_scheduler] Job 11: RunJobThread.run
[2017-06-29 10:55:20,653: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-06-29 10:55:20,674: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 35155643-0b00-46c9-b957-a18588c7be6a
[2017-06-29 10:55:20,674: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 35155643-0b00-46c9-b957-a18588c7be6a
[2017-06-29 10:55:28,823: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:55:28,823: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:55:28,823: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:55:28,823: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:55:28,824: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 35155643-0b00-46c9-b957-a18588c7be6a
[2017-06-29 10:55:28,824: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 35155643-0b00-46c9-b957-a18588c7be6a
[2017-06-29 10:55:28,842: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 35155643-0b00-46c9-b957-a18588c7be6a
[2017-06-29 10:55:28,842: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 35155643-0b00-46c9-b957-a18588c7be6a
[2017-06-29 10:55:28,842: INFO/job_scheduler] Job 11 finished 1 steps successfully
[2017-06-29 10:55:28,890: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 140667359953296
[2017-06-29 10:55:28,910: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 30
[2017-06-29 10:55:28,918: INFO/job] Job 11: StateChangeJob complete, setting state started on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 10:55:28,918: INFO/job] Job 11: not_deleted=True
[2017-06-29 10:55:28,919: INFO/job_scheduler] Job 11: complete_job: Updating cache
[2017-06-29 10:55:28,919: INFO/job_scheduler] Job 11: locked_item 140667359953296 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-29 10:55:28,924: INFO/job_scheduler] Job 11 completing (errored=False, cancelled=False)
[2017-06-29 10:55:28,944: INFO/job_scheduler] run_next: 1 runnable jobs of (3 pending, 0 tasked)
[2017-06-29 10:55:28,980: INFO/job_scheduler] Job 12: RunJobThread.run
[2017-06-29 10:55:28,981: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-06-29 10:55:28,995: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9aee6063-5c39-48a5-a499-95c10132e427
[2017-06-29 10:55:28,995: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9aee6063-5c39-48a5-a499-95c10132e427
[2017-06-29 10:55:30,676: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-29 10:55:30,692: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-29 10:55:36,070: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:55:36,070: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:55:36,070: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:55:36,070: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:55:36,070: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9aee6063-5c39-48a5-a499-95c10132e427
[2017-06-29 10:55:36,070: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9aee6063-5c39-48a5-a499-95c10132e427
[2017-06-29 10:55:36,110: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9aee6063-5c39-48a5-a499-95c10132e427
[2017-06-29 10:55:36,110: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9aee6063-5c39-48a5-a499-95c10132e427
[2017-06-29 10:55:36,112: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-06-29 10:55:36,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8fb491cc-bb85-427c-a21e-2d39e49a3627
[2017-06-29 10:55:36,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8fb491cc-bb85-427c-a21e-2d39e49a3627
[2017-06-29 10:55:40,667: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-29 10:55:40,694: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-29 10:55:59,712: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-29 10:55:59,729: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-29 10:56:30,839: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-29 10:56:30,840: INFO/job_scheduler]   Job 12 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 10:56:30,840: INFO/job_scheduler]   Job 14 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 10:56:30,840: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-29 10:56:40,821: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-29 10:56:40,822: INFO/job_scheduler]   Job 12 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 10:56:40,822: INFO/job_scheduler]   Job 14 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 10:56:40,822: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-29 10:56:50,880: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-29 10:56:50,880: INFO/job_scheduler]   Job 12 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 10:56:50,880: INFO/job_scheduler]   Job 14 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 10:56:50,880: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-29 10:56:56,488: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:56:56,488: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:56:56,489: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:56:56,489: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:56:56,489: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8fb491cc-bb85-427c-a21e-2d39e49a3627
[2017-06-29 10:56:56,489: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8fb491cc-bb85-427c-a21e-2d39e49a3627
[2017-06-29 10:56:56,522: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8fb491cc-bb85-427c-a21e-2d39e49a3627
[2017-06-29 10:56:56,522: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8fb491cc-bb85-427c-a21e-2d39e49a3627
[2017-06-29 10:56:56,525: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-06-29 10:56:56,537: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2909d6d1-3ac8-450e-b928-921d521ccfe1
[2017-06-29 10:56:56,537: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2909d6d1-3ac8-450e-b928-921d521ccfe1
[2017-06-29 10:57:00,884: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-29 10:57:00,899: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-29 10:57:03,647: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:57:03,647: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:57:03,648: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:57:03,648: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:57:03,648: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2909d6d1-3ac8-450e-b928-921d521ccfe1
[2017-06-29 10:57:03,648: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2909d6d1-3ac8-450e-b928-921d521ccfe1
[2017-06-29 10:57:03,655: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2909d6d1-3ac8-450e-b928-921d521ccfe1
[2017-06-29 10:57:03,655: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2909d6d1-3ac8-450e-b928-921d521ccfe1
[2017-06-29 10:57:03,656: INFO/job_scheduler] Job 12 finished 3 steps successfully
[2017-06-29 10:57:03,722: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 140667359943056
[2017-06-29 10:57:03,740: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-06-29 10:57:03,759: INFO/job] Job 12: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-29 10:57:03,759: INFO/job] Job 12: not_deleted=True
[2017-06-29 10:57:03,759: INFO/job_scheduler] Job 12: complete_job: Updating cache
[2017-06-29 10:57:03,759: INFO/job_scheduler] Job 12: locked_item 140667359943056 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-29 10:57:03,765: INFO/job_scheduler] Job 12 completing (errored=False, cancelled=False)
[2017-06-29 10:57:03,892: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 0 tasked)
[2017-06-29 10:57:03,940: INFO/job_scheduler] Job 14: RunJobThread.run
[2017-06-29 10:57:03,941: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-06-29 10:57:04,008: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3f4621cf-9f6d-468f-903b-eefcd86e40c1
[2017-06-29 10:57:04,008: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3f4621cf-9f6d-468f-903b-eefcd86e40c1
[2017-06-29 10:57:10,920: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-06-29 10:57:10,936: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-06-29 10:57:11,072: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:57:11,072: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:57:11,072: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:57:11,072: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:57:11,073: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3f4621cf-9f6d-468f-903b-eefcd86e40c1
[2017-06-29 10:57:11,073: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3f4621cf-9f6d-468f-903b-eefcd86e40c1
[2017-06-29 10:57:11,075: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3f4621cf-9f6d-468f-903b-eefcd86e40c1
[2017-06-29 10:57:11,075: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3f4621cf-9f6d-468f-903b-eefcd86e40c1
[2017-06-29 10:57:11,075: INFO/job_scheduler] Job 14 finished 1 steps successfully
[2017-06-29 10:57:11,119: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 140667359943056
[2017-06-29 10:57:11,146: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 30
[2017-06-29 10:57:11,153: INFO/job] Job 14: StateChangeJob complete, setting state started on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-29 10:57:11,155: INFO/job] Job 14: not_deleted=True
[2017-06-29 10:57:11,155: INFO/job_scheduler] Job 14: complete_job: Updating cache
[2017-06-29 10:57:11,156: INFO/job_scheduler] Job 14: locked_item 140667359943056 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-29 10:57:11,159: INFO/job_scheduler] Job 14 completing (errored=False, cancelled=False)
[2017-06-29 10:57:11,170: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-1
[2017-06-29 10:57:11,171: INFO/job_scheduler] _notify: Updating .mcast_port of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 52477
[2017-06-29 10:57:11,175: INFO/job_scheduler] _notify: Set mcast_port=52477 on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-06-29 10:57:11,185: INFO/job_scheduler] _notify: Updating .network_interfaces of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-06-29 10:57:11,203: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-06-29 10:57:11,217: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-06-29 10:57:11,221: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-06-29 10:57:11,236: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-29 10:57:11,263: INFO/job_scheduler] Job 15: RunJobThread.run
[2017-06-29 10:57:11,265: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-06-29 10:57:11,283: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d59ee275-eed7-464c-a715-0df49f8e53a9
[2017-06-29 10:57:11,283: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d59ee275-eed7-464c-a715-0df49f8e53a9
[2017-06-29 10:57:20,956: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from True to False
[2017-06-29 10:57:20,962: INFO/job_scheduler] _notify: Set corosync_reported_up=False on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-06-29 10:57:20,969: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-29 10:57:30,965: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-06-29 10:57:30,971: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-1) and saved
[2017-06-29 10:57:30,978: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-29 10:57:37,622: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:57:37,622: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:57:37,622: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:57:37,622: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:57:37,622: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d59ee275-eed7-464c-a715-0df49f8e53a9
[2017-06-29 10:57:37,622: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d59ee275-eed7-464c-a715-0df49f8e53a9
[2017-06-29 10:57:37,660: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d59ee275-eed7-464c-a715-0df49f8e53a9
[2017-06-29 10:57:37,660: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d59ee275-eed7-464c-a715-0df49f8e53a9
[2017-06-29 10:57:37,662: INFO/job_scheduler] Job 15 finished 1 steps successfully
[2017-06-29 10:57:38,139: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 140667359942928
[2017-06-29 10:57:38,144: INFO/job] Job 15: StateChangeJob complete, setting state managed on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 10:57:38,144: INFO/job] Job 15: not_deleted=True
[2017-06-29 10:57:38,144: INFO/job_scheduler] Job 15: complete_job: Updating cache
[2017-06-29 10:57:38,144: INFO/job_scheduler] Job 15: locked_item 140667359942928 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-29 10:57:38,147: INFO/job_scheduler] Job 15 completing (errored=False, cancelled=False)
[2017-06-29 10:57:38,207: INFO/job_scheduler] Replaying 5 buffered notifications for ManagedHost-1
[2017-06-29 10:57:38,224: INFO/job_scheduler] _notify: Updating .immutable_state of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com) from True to False
[2017-06-29 10:57:38,229: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-1) and saved
[2017-06-29 10:57:38,240: INFO/job_scheduler] _notify: Updating .boot_time of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com) from 2017-06-23 08:35:25+00:00 to 2017-06-29 03:53:12+00:00
[2017-06-29 10:57:38,247: INFO/job_scheduler] _notify: Set boot_time=2017-06-29 03:53:12+00:00 on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-1) and saved
[2017-06-29 10:57:38,269: INFO/job_scheduler] _notify: Updating .properties of item 1 (lotus-32vm5.lotus.hpdd.lab.intel.com) from {"python_patchlevel": 5, "kernel_version": "3.10.0-514.21.1.el7_lustre.x86_64", "zfs_installed": false, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.21.1.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-06-29 10:57:38,292: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.21.1.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-1) and saved
[2017-06-29 10:57:38,338: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 10:59:51,649: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 10:59:51,651: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-1 transitioning from managed to removed
[2017-06-29 10:59:51,751: INFO/command_plan] Created command 6 (Remove host lotus-32vm5.lotus.hpdd.lab.intel.com from configuration) with 9 jobs
[2017-06-29 10:59:51,811: INFO/job_scheduler] run_next: 4 runnable jobs of (9 pending, 0 tasked)
[2017-06-29 10:59:51,856: INFO/job_scheduler] Job 16: RunJobThread.run
[2017-06-29 10:59:51,857: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-06-29 10:59:51,886: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2b6b114d-35e6-47ca-a401-d3254a98c6b8
[2017-06-29 10:59:51,886: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2b6b114d-35e6-47ca-a401-d3254a98c6b8
[2017-06-29 10:59:51,914: INFO/job_scheduler] Job 17: RunJobThread.run
[2017-06-29 10:59:51,915: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unconfigure_rsyslog {}
[2017-06-29 10:59:51,916: INFO/job_scheduler] Job 18: RunJobThread.run
[2017-06-29 10:59:51,916: INFO/job_scheduler] Job 19: RunJobThread.run
[2017-06-29 10:59:51,916: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_lnet {}
[2017-06-29 10:59:51,917: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unconfigure_ntp {}
[2017-06-29 10:59:51,933: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b4a813f2-abff-405b-badc-3925015167dc
[2017-06-29 10:59:51,933: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b4a813f2-abff-405b-badc-3925015167dc
[2017-06-29 10:59:51,949: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9f5571d1-24e8-47ed-90d2-7c60b354c736
[2017-06-29 10:59:51,949: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9f5571d1-24e8-47ed-90d2-7c60b354c736
[2017-06-29 10:59:51,985: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3701c664-1414-464d-8f31-b84a62b0504f
[2017-06-29 10:59:51,985: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3701c664-1414-464d-8f31-b84a62b0504f
[2017-06-29 10:59:58,069: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:59:58,069: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:59:58,069: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:59:58,069: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:59:58,069: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3701c664-1414-464d-8f31-b84a62b0504f
[2017-06-29 10:59:58,069: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3701c664-1414-464d-8f31-b84a62b0504f
[2017-06-29 10:59:58,101: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3701c664-1414-464d-8f31-b84a62b0504f
[2017-06-29 10:59:58,101: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3701c664-1414-464d-8f31-b84a62b0504f
[2017-06-29 10:59:58,101: INFO/job_scheduler] Job 19 finished 1 steps successfully
[2017-06-29 10:59:58,157: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration configured->unconfigured (intentional=True) 140667829873296
[2017-06-29 10:59:58,161: INFO/job] Job 19: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration
[2017-06-29 10:59:58,161: INFO/job] Job 19: not_deleted=True
[2017-06-29 10:59:58,161: INFO/job_scheduler] Job 19: complete_job: Updating cache
[2017-06-29 10:59:58,161: INFO/job_scheduler] Job 19: locked_item 140667829873296 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-06-29 10:59:58,164: INFO/job_scheduler] Job 19 completing (errored=False, cancelled=False)
[2017-06-29 10:59:58,207: INFO/job_scheduler] run_next: 0 runnable jobs of (5 pending, 3 tasked)
[2017-06-29 10:59:58,963: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:59:58,963: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:59:58,964: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:59:58,964: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:59:58,964: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2b6b114d-35e6-47ca-a401-d3254a98c6b8
[2017-06-29 10:59:58,964: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2b6b114d-35e6-47ca-a401-d3254a98c6b8
[2017-06-29 10:59:59,000: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:59:59,000: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:59:59,000: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:59:59,000: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 10:59:59,000: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b4a813f2-abff-405b-badc-3925015167dc
[2017-06-29 10:59:59,000: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b4a813f2-abff-405b-badc-3925015167dc
[2017-06-29 10:59:59,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2b6b114d-35e6-47ca-a401-d3254a98c6b8
[2017-06-29 10:59:59,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2b6b114d-35e6-47ca-a401-d3254a98c6b8
[2017-06-29 10:59:59,003: INFO/job_scheduler] Job 16 finished 1 steps successfully
[2017-06-29 10:59:59,049: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b4a813f2-abff-405b-badc-3925015167dc
[2017-06-29 10:59:59,049: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b4a813f2-abff-405b-badc-3925015167dc
[2017-06-29 10:59:59,050: INFO/job_scheduler] Job 17 finished 1 steps successfully
[2017-06-29 10:59:59,055: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration started->stopped (intentional=True) 140667830058000
[2017-06-29 10:59:59,062: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-06-29 10:59:59,066: INFO/job] Job 16: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-29 10:59:59,067: INFO/job] Job 16: not_deleted=True
[2017-06-29 10:59:59,069: INFO/job_scheduler] Job 16: complete_job: Updating cache
[2017-06-29 10:59:59,069: INFO/job_scheduler] Job 16: locked_item 140667830058000 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-29 10:59:59,074: INFO/job_scheduler] Job 16 completing (errored=False, cancelled=False)
[2017-06-29 10:59:59,088: INFO/job_scheduler] run_next: 1 runnable jobs of (5 pending, 2 tasked)
[2017-06-29 10:59:59,100: INFO/job_scheduler] Job 21: RunJobThread.run
[2017-06-29 10:59:59,101: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_corosync2 {}
[2017-06-29 10:59:59,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7fdd33cd-7b82-4307-b891-40a1b1eda83b
[2017-06-29 10:59:59,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7fdd33cd-7b82-4307-b891-40a1b1eda83b
[2017-06-29 10:59:59,196: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration configured->unconfigured (intentional=True) 140668098572176
[2017-06-29 10:59:59,200: INFO/job] Job 17: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-06-29 10:59:59,200: INFO/job] Job 17: not_deleted=True
[2017-06-29 10:59:59,200: INFO/job_scheduler] Job 17: complete_job: Updating cache
[2017-06-29 10:59:59,200: INFO/job_scheduler] Job 17: locked_item 140668098572176 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-06-29 10:59:59,203: INFO/job_scheduler] Job 17 completing (errored=False, cancelled=False)
[2017-06-29 10:59:59,233: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-29 11:00:00,342: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-29 11:00:00,355: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-29 11:00:03,106: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:03,106: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:03,106: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:03,106: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:03,106: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9f5571d1-24e8-47ed-90d2-7c60b354c736
[2017-06-29 11:00:03,106: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9f5571d1-24e8-47ed-90d2-7c60b354c736
[2017-06-29 11:00:03,117: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9f5571d1-24e8-47ed-90d2-7c60b354c736
[2017-06-29 11:00:03,117: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9f5571d1-24e8-47ed-90d2-7c60b354c736
[2017-06-29 11:00:03,119: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-29 11:00:03,137: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f8590e6c-2f95-464d-ae5f-51a4421e4125
[2017-06-29 11:00:03,137: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f8590e6c-2f95-464d-ae5f-51a4421e4125
[2017-06-29 11:00:04,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:04,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:04,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:04,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:04,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f8590e6c-2f95-464d-ae5f-51a4421e4125
[2017-06-29 11:00:04,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f8590e6c-2f95-464d-ae5f-51a4421e4125
[2017-06-29 11:00:04,251: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f8590e6c-2f95-464d-ae5f-51a4421e4125
[2017-06-29 11:00:04,251: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f8590e6c-2f95-464d-ae5f-51a4421e4125
[2017-06-29 11:00:04,376: INFO/job_scheduler] Job 18 finished 2 steps successfully
[2017-06-29 11:00:04,396: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_up->lnet_down (intentional=True) 140667830057552
[2017-06-29 11:00:04,402: INFO/job] Job 18: StateChangeJob complete, setting state lnet_down on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-29 11:00:04,402: INFO/job] Job 18: not_deleted=True
[2017-06-29 11:00:04,402: INFO/job_scheduler] Job 18: complete_job: Updating cache
[2017-06-29 11:00:04,402: INFO/job_scheduler] Job 18: locked_item 140667830057552 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-29 11:00:04,406: INFO/job_scheduler] Job 18 completing (errored=False, cancelled=False)
[2017-06-29 11:00:04,431: INFO/job_scheduler] run_next: 1 runnable jobs of (4 pending, 1 tasked)
[2017-06-29 11:00:04,457: INFO/job_scheduler] Job 20: RunJobThread.run
[2017-06-29 11:00:04,457: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unload_lnet {}
[2017-06-29 11:00:04,519: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c925e75f-5aa2-4ff7-9825-5a73f5c8265d
[2017-06-29 11:00:04,519: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c925e75f-5aa2-4ff7-9825-5a73f5c8265d
[2017-06-29 11:00:05,614: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:05,614: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:05,615: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:05,615: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:05,615: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c925e75f-5aa2-4ff7-9825-5a73f5c8265d
[2017-06-29 11:00:05,615: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c925e75f-5aa2-4ff7-9825-5a73f5c8265d
[2017-06-29 11:00:05,635: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c925e75f-5aa2-4ff7-9825-5a73f5c8265d
[2017-06-29 11:00:05,635: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c925e75f-5aa2-4ff7-9825-5a73f5c8265d
[2017-06-29 11:00:05,636: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-29 11:00:05,650: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 84f65980-a535-421e-8aae-50e4fe6969dd
[2017-06-29 11:00:05,650: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 84f65980-a535-421e-8aae-50e4fe6969dd
[2017-06-29 11:00:05,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:05,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:05,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:05,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:05,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 84f65980-a535-421e-8aae-50e4fe6969dd
[2017-06-29 11:00:05,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 84f65980-a535-421e-8aae-50e4fe6969dd
[2017-06-29 11:00:05,766: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 84f65980-a535-421e-8aae-50e4fe6969dd
[2017-06-29 11:00:05,766: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 84f65980-a535-421e-8aae-50e4fe6969dd
[2017-06-29 11:00:05,907: INFO/job_scheduler] Job 20 finished 2 steps successfully
[2017-06-29 11:00:05,954: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_unloaded (intentional=True) 140667830057552
[2017-06-29 11:00:05,959: INFO/job] Job 20: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-29 11:00:05,959: INFO/job] Job 20: not_deleted=True
[2017-06-29 11:00:05,959: INFO/job_scheduler] Job 20: complete_job: Updating cache
[2017-06-29 11:00:05,959: INFO/job_scheduler] Job 20: locked_item 140667830057552 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-29 11:00:05,962: INFO/job_scheduler] Job 20 completing (errored=False, cancelled=False)
[2017-06-29 11:00:05,980: INFO/job_scheduler] run_next: 1 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 11:00:05,996: INFO/job_scheduler] Job 22: RunJobThread.run
[2017-06-29 11:00:05,997: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unconfigure_lnet {}
[2017-06-29 11:00:06,026: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 76bc1005-7064-49f7-97a6-c1e1c2acbdf0
[2017-06-29 11:00:06,026: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 76bc1005-7064-49f7-97a6-c1e1c2acbdf0
[2017-06-29 11:00:06,114: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:06,114: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:06,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:06,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:06,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 76bc1005-7064-49f7-97a6-c1e1c2acbdf0
[2017-06-29 11:00:06,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 76bc1005-7064-49f7-97a6-c1e1c2acbdf0
[2017-06-29 11:00:06,147: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 76bc1005-7064-49f7-97a6-c1e1c2acbdf0
[2017-06-29 11:00:06,147: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 76bc1005-7064-49f7-97a6-c1e1c2acbdf0
[2017-06-29 11:00:06,147: INFO/job] Invalid result None fixed up on called to unconfigure_lnet with args {}
[2017-06-29 11:00:06,147: INFO/job_scheduler] Job 22 finished 1 steps successfully
[2017-06-29 11:00:06,206: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->unconfigured (intentional=True) 140667830057552
[2017-06-29 11:00:06,214: INFO/job] Job 22: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-29 11:00:06,215: INFO/job] Job 22: not_deleted=True
[2017-06-29 11:00:06,215: INFO/job_scheduler] Job 22: complete_job: Updating cache
[2017-06-29 11:00:06,215: INFO/job_scheduler] Job 22: locked_item 140667830057552 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-29 11:00:06,218: INFO/job_scheduler] Job 22 completing (errored=False, cancelled=False)
[2017-06-29 11:00:06,230: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-29 11:00:07,221: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:07,221: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:07,222: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:07,222: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:07,222: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7fdd33cd-7b82-4307-b891-40a1b1eda83b
[2017-06-29 11:00:07,222: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7fdd33cd-7b82-4307-b891-40a1b1eda83b
[2017-06-29 11:00:07,238: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7fdd33cd-7b82-4307-b891-40a1b1eda83b
[2017-06-29 11:00:07,238: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7fdd33cd-7b82-4307-b891-40a1b1eda83b
[2017-06-29 11:00:07,239: INFO/job_scheduler] Job 21 finished 1 steps successfully
[2017-06-29 11:00:07,281: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration started->stopped (intentional=True) 140668098635728
[2017-06-29 11:00:07,290: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-06-29 11:00:07,299: INFO/job] Job 21: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:00:07,300: INFO/job] Job 21: not_deleted=True
[2017-06-29 11:00:07,300: INFO/job_scheduler] Job 21: complete_job: Updating cache
[2017-06-29 11:00:07,300: INFO/job_scheduler] Job 21: locked_item 140668098635728 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-29 11:00:07,304: INFO/job_scheduler] Job 21 completing (errored=False, cancelled=False)
[2017-06-29 11:00:07,421: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 0 tasked)
[2017-06-29 11:00:07,426: INFO/job_scheduler] Job 23: RunJobThread.run
[2017-06-29 11:00:07,427: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unconfigure_corosync2 {'mcast_port': 52477, 'host_fqdn': u'lotus-32vm5.lotus.hpdd.lab.intel.com'}
[2017-06-29 11:00:07,447: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc eaa447e4-29bf-43b8-b951-3e578a26ff3f
[2017-06-29 11:00:07,447: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc eaa447e4-29bf-43b8-b951-3e578a26ff3f
[2017-06-29 11:00:09,400: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-06-29 11:00:09,423: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-06-29 11:00:13,552: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:13,552: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:13,552: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:13,552: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:13,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc eaa447e4-29bf-43b8-b951-3e578a26ff3f
[2017-06-29 11:00:13,553: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc eaa447e4-29bf-43b8-b951-3e578a26ff3f
[2017-06-29 11:00:13,561: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc eaa447e4-29bf-43b8-b951-3e578a26ff3f
[2017-06-29 11:00:13,561: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc eaa447e4-29bf-43b8-b951-3e578a26ff3f
[2017-06-29 11:00:13,562: INFO/job_scheduler] Job 23 finished 1 steps successfully
[2017-06-29 11:00:13,745: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->unconfigured (intentional=True) 140668098635728
[2017-06-29 11:00:13,751: INFO/job] Job 23: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:00:13,751: INFO/job] Job 23: not_deleted=True
[2017-06-29 11:00:13,751: INFO/job_scheduler] Job 23: complete_job: Updating cache
[2017-06-29 11:00:13,751: INFO/job_scheduler] Job 23: locked_item 140668098635728 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-29 11:00:13,755: INFO/job_scheduler] Job 23 completing (errored=False, cancelled=False)
[2017-06-29 11:00:13,770: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-29 11:00:13,791: INFO/job_scheduler] Job 24: RunJobThread.run
[2017-06-29 11:00:13,793: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com deregister_server {}
[2017-06-29 11:00:13,814: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d2ce22d7-fbdf-4b56-ad0a-e3af39296ee0
[2017-06-29 11:00:13,814: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d2ce22d7-fbdf-4b56-ad0a-e3af39296ee0
[2017-06-29 11:00:13,871: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:13,871: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:13,872: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:13,872: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:00:13,872: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d2ce22d7-fbdf-4b56-ad0a-e3af39296ee0
[2017-06-29 11:00:13,872: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d2ce22d7-fbdf-4b56-ad0a-e3af39296ee0
[2017-06-29 11:00:13,883: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d2ce22d7-fbdf-4b56-ad0a-e3af39296ee0
[2017-06-29 11:00:13,883: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d2ce22d7-fbdf-4b56-ad0a-e3af39296ee0
[2017-06-29 11:00:13,913: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:00:13,913: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration unconfigured->removed (intentional=False) 140668098733200
[2017-06-29 11:00:13,931: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.ntp.NTPConfiguration'> 1
[2017-06-29 11:00:13,946: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:00:13,946: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->removed (intentional=False) 140668098732112
[2017-06-29 11:00:13,955: INFO/job] Lowered 2 alerts while deleting <class 'chroma_core.models.corosync2.Corosync2Configuration'> 1
[2017-06-29 11:00:13,971: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:00:13,971: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->removed (intentional=False) 140668098732368
[2017-06-29 11:00:13,977: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> 1
[2017-06-29 11:00:13,998: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:00:13,999: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration unconfigured->removed (intentional=False) 140668098734352
[2017-06-29 11:00:14,009: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> 1
[2017-06-29 11:00:14,028: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:00:14,028: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->removed (intentional=False) 140668098734288
[2017-06-29 11:00:14,036: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> 1
[2017-06-29 11:00:14,761: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->removed (intentional=False) 140668098635728
[2017-06-29 11:00:14,785: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-06-29 11:00:14,798: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.corosync2.Corosync2Configuration'> 1
[2017-06-29 11:00:14,852: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration unconfigured->removed (intentional=False) 140667829873296
[2017-06-29 11:00:14,861: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.ntp.NTPConfiguration'> 1
[2017-06-29 11:00:14,893: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->removed (intentional=False) 140668098572176
[2017-06-29 11:00:14,898: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> 1
[2017-06-29 11:00:14,924: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.host.ManagedHost'> 1
[2017-06-29 11:00:14,927: INFO/job_scheduler] Job 24 finished 2 steps successfully
[2017-06-29 11:00:14,935: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com managed->removed (intentional=True) 140668098614544
[2017-06-29 11:00:14,939: INFO/job] Job 24: StateChangeJob complete, setting state removed on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:00:14,940: INFO/job] Job 24: not_deleted=None
[2017-06-29 11:00:14,940: INFO/job_scheduler] Job 24: complete_job: Updating cache
[2017-06-29 11:00:14,940: INFO/job_scheduler] Job 24: locked_item 140668098614544 <class 'chroma_core.models.host.ManagedHost'> True None
[2017-06-29 11:00:14,940: INFO/job_scheduler] Job 24 completing (errored=False, cancelled=False)
[2017-06-29 11:00:14,977: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 11:24:03,805: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 11:24:03,806: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-29 11:24:03,820: INFO/command_plan] add_jobs: created Job 25 (Test for host connectivity)
[2017-06-29 11:24:03,859: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 11:24:03,860: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-29 11:24:03,877: INFO/command_plan] add_jobs: created Job 26 (Test for host connectivity)
[2017-06-29 11:24:03,890: INFO/job_scheduler] run_next: 2 runnable jobs of (2 pending, 0 tasked)
[2017-06-29 11:24:03,911: INFO/job_scheduler] Job 25: RunJobThread.run
[2017-06-29 11:24:03,917: INFO/job_scheduler] Job 26: RunJobThread.run
[2017-06-29 11:24:03,976: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 11:24:03,994: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-29 11:24:04,043: INFO/command_plan] add_jobs: created Job 27 (Test for host connectivity)
[2017-06-29 11:24:04,054: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-29 11:24:04,059: INFO/job_scheduler] Job 27: RunJobThread.run
[2017-06-29 11:24:04,244: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 11:24:04,248: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-29 11:24:04,261: INFO/command_plan] add_jobs: created Job 28 (Test for host connectivity)
[2017-06-29 11:24:04,295: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-29 11:24:04,302: INFO/job_scheduler] Job 28: RunJobThread.run
[2017-06-29 11:24:04,387: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-29 11:24:18,398: INFO/job_scheduler] Job 27 finished 1 steps successfully
[2017-06-29 11:24:18,402: INFO/job_scheduler] Job 26 finished 1 steps successfully
[2017-06-29 11:24:18,425: INFO/job_scheduler] Job 27: complete_job: Updating cache
[2017-06-29 11:24:18,425: INFO/job_scheduler] Job 27 completing (errored=False, cancelled=False)
[2017-06-29 11:24:18,475: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-29 11:24:18,508: INFO/job_scheduler] Job 26: complete_job: Updating cache
[2017-06-29 11:24:18,508: INFO/job_scheduler] Job 26 completing (errored=False, cancelled=False)
[2017-06-29 11:24:18,525: INFO/job_scheduler] Job 25 finished 1 steps successfully
[2017-06-29 11:24:18,578: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-29 11:24:18,595: INFO/job_scheduler] Job 25: complete_job: Updating cache
[2017-06-29 11:24:18,595: INFO/job_scheduler] Job 25 completing (errored=False, cancelled=False)
[2017-06-29 11:24:18,676: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-29 11:24:18,990: INFO/job_scheduler] Job 28 finished 1 steps successfully
[2017-06-29 11:24:19,050: INFO/job_scheduler] Job 28: complete_job: Updating cache
[2017-06-29 11:24:19,050: INFO/job_scheduler] Job 28 completing (errored=False, cancelled=False)
[2017-06-29 11:24:19,083: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 11:24:19,290: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 11:24:19,291: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-29 11:24:19,333: INFO/command_plan] add_jobs: created Job 29 (Test for host connectivity)
[2017-06-29 11:24:19,352: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-29 11:24:19,356: INFO/job_scheduler] Job 29: RunJobThread.run
[2017-06-29 11:24:19,384: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 11:24:19,384: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-29 11:24:19,400: INFO/command_plan] add_jobs: created Job 30 (Test for host connectivity)
[2017-06-29 11:24:19,411: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 1 tasked)
[2017-06-29 11:24:19,417: INFO/job_scheduler] Job 30: RunJobThread.run
[2017-06-29 11:24:19,568: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 11:24:19,568: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-29 11:24:19,603: INFO/command_plan] add_jobs: created Job 31 (Test for host connectivity)
[2017-06-29 11:24:19,635: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 2 tasked)
[2017-06-29 11:24:19,657: INFO/job_scheduler] Job 31: RunJobThread.run
[2017-06-29 11:24:19,688: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 11:24:19,688: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-29 11:24:19,701: INFO/command_plan] add_jobs: created Job 32 (Test for host connectivity)
[2017-06-29 11:24:19,718: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 3 tasked)
[2017-06-29 11:24:19,741: INFO/job_scheduler] Job 32: RunJobThread.run
[2017-06-29 11:24:24,858: INFO/job_scheduler] Getting FQDN for 'lotus-32vm5.lotus.hpdd.lab.intel.com': lotus-32vm5
lotus-32vm5.lotus.hpdd.lab.intel.com

[2017-06-29 11:24:24,889: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 11:24:24,891: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-2 transitioning from undeployed to unconfigured
[2017-06-29 11:24:24,916: INFO/command_plan] Created command 15 (Deploying agent to host lotus-32vm5.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-06-29 11:24:24,936: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 4 tasked)
[2017-06-29 11:24:24,986: INFO/job_scheduler] Job 33: RunJobThread.run
[2017-06-29 11:24:25,609: INFO/job_scheduler] Getting FQDN for 'lotus-32vm6.lotus.hpdd.lab.intel.com': lotus-32vm6
lotus-32vm6.lotus.hpdd.lab.intel.com

[2017-06-29 11:24:25,631: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 11:24:25,636: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-3 transitioning from undeployed to unconfigured
[2017-06-29 11:24:25,653: INFO/command_plan] Created command 16 (Deploying agent to host lotus-32vm6.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-06-29 11:24:25,748: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 5 tasked)
[2017-06-29 11:24:25,812: INFO/job_scheduler] Job 34: RunJobThread.run
[2017-06-29 11:24:25,833: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 6 tasked)
[2017-06-29 11:24:26,384: INFO/job_scheduler] Getting FQDN for 'lotus-32vm7.lotus.hpdd.lab.intel.com': lotus-32vm7
lotus-32vm7.lotus.hpdd.lab.intel.com

[2017-06-29 11:24:26,431: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 11:24:26,433: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-4 transitioning from undeployed to unconfigured
[2017-06-29 11:24:26,454: INFO/command_plan] Created command 17 (Deploying agent to host lotus-32vm7.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-06-29 11:24:26,611: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 6 tasked)
[2017-06-29 11:24:26,660: INFO/job_scheduler] Job 35: RunJobThread.run
[2017-06-29 11:24:26,713: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 7 tasked)
[2017-06-29 11:24:27,183: INFO/job_scheduler] Getting FQDN for 'lotus-32vm8.lotus.hpdd.lab.intel.com': lotus-32vm8
lotus-32vm8.lotus.hpdd.lab.intel.com

[2017-06-29 11:24:27,205: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 11:24:27,210: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-5 transitioning from undeployed to unconfigured
[2017-06-29 11:24:27,228: INFO/command_plan] Created command 18 (Deploying agent to host lotus-32vm8.lotus.hpdd.lab.intel.com) with 1 jobs
[2017-06-29 11:24:27,294: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 7 tasked)
[2017-06-29 11:24:27,373: INFO/job_scheduler] Job 36: RunJobThread.run
[2017-06-29 11:24:27,584: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 8 tasked)
[2017-06-29 11:24:28,072: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 8 tasked)
[2017-06-29 11:24:30,519: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 11:24:30,519: INFO/job_scheduler]   Job 33 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-06-29 11:24:30,519: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 8 tasked)
[2017-06-29 11:24:30,533: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:24:30,533: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/dc7a8657-414f-4279-9918-3060dae4a4d7
[2017-06-29 11:24:30,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/1d3091b5-ca2c-4dcb-ba9a-138a177b462e
[2017-06-29 11:24:30,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/1d3091b5-ca2c-4dcb-ba9a-138a177b462e
[2017-06-29 11:24:30,559: INFO/job_scheduler] Job 33 finished 1 steps successfully
[2017-06-29 11:24:30,614: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 140667427197648
[2017-06-29 11:24:30,621: INFO/job] Job 33: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:24:30,621: INFO/job] Job 33: not_deleted=True
[2017-06-29 11:24:30,621: INFO/job_scheduler] Job 33: complete_job: Updating cache
[2017-06-29 11:24:30,621: INFO/job_scheduler] Job 33: locked_item 140667427197648 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-29 11:24:30,624: INFO/job_scheduler] Job 33 completing (errored=False, cancelled=False)
[2017-06-29 11:24:30,675: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-2
[2017-06-29 11:24:30,675: INFO/job_scheduler] _notify: Updating .boot_time of item 2 (lotus-32vm5.lotus.hpdd.lab.intel.com) from None to 2017-06-29 03:53:12+00:00
[2017-06-29 11:24:30,681: INFO/job_scheduler] _notify: Set boot_time=2017-06-29 03:53:12+00:00 on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-2) and saved
[2017-06-29 11:24:30,689: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 7 tasked)
[2017-06-29 11:24:31,906: INFO/job_scheduler] Job 29 finished 1 steps successfully
[2017-06-29 11:24:32,073: INFO/job_scheduler] Job 29: complete_job: Updating cache
[2017-06-29 11:24:32,073: INFO/job_scheduler] Job 29 completing (errored=False, cancelled=False)
[2017-06-29 11:24:32,137: INFO/job_scheduler] Job 31 finished 1 steps successfully
[2017-06-29 11:24:32,166: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 6 tasked)
[2017-06-29 11:24:32,222: INFO/job_scheduler] Job 31: complete_job: Updating cache
[2017-06-29 11:24:32,222: INFO/job_scheduler] Job 31 completing (errored=False, cancelled=False)
[2017-06-29 11:24:32,254: INFO/job_scheduler] Job 32 finished 1 steps successfully
[2017-06-29 11:24:32,340: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 5 tasked)
[2017-06-29 11:24:32,365: INFO/job_scheduler] Job 32: complete_job: Updating cache
[2017-06-29 11:24:32,365: INFO/job_scheduler] Job 32 completing (errored=False, cancelled=False)
[2017-06-29 11:24:32,489: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 4 tasked)
[2017-06-29 11:24:34,773: INFO/job_scheduler] Job 30 finished 1 steps successfully
[2017-06-29 11:24:34,830: INFO/job_scheduler] Job 30: complete_job: Updating cache
[2017-06-29 11:24:34,830: INFO/job_scheduler] Job 30 completing (errored=False, cancelled=False)
[2017-06-29 11:24:34,963: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-29 11:24:35,153: INFO/job_scheduler] _notify: Updating .properties of item 2 (lotus-32vm5.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.21.1.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-06-29 11:24:35,160: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.21.1.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-2) and saved
[2017-06-29 11:24:35,164: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-29 11:24:35,242: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-29 11:24:41,419: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/a993a546-dcf3-4bdf-bb1c-6c24620e676d
[2017-06-29 11:24:41,419: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/a993a546-dcf3-4bdf-bb1c-6c24620e676d
[2017-06-29 11:24:41,518: INFO/job_scheduler] Job 36 finished 1 steps successfully
[2017-06-29 11:24:41,632: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 11:24:41,632: INFO/job_scheduler]   Job 36 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-06-29 11:24:41,633: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 3 tasked)
[2017-06-29 11:24:42,529: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 140667427216336
[2017-06-29 11:24:42,535: INFO/job] Job 36: StateChangeJob complete, setting state unconfigured on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:24:42,536: INFO/job] Job 36: not_deleted=True
[2017-06-29 11:24:42,536: INFO/job_scheduler] Job 36: complete_job: Updating cache
[2017-06-29 11:24:42,536: INFO/job_scheduler] Job 36: locked_item 140667427216336 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-29 11:24:42,538: INFO/job_scheduler] Job 36 completing (errored=False, cancelled=False)
[2017-06-29 11:24:42,607: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-5
[2017-06-29 11:24:42,607: INFO/job_scheduler] _notify: Updating .boot_time of item 5 (lotus-32vm8.lotus.hpdd.lab.intel.com) from None to 2017-06-15 09:59:41+00:00
[2017-06-29 11:24:42,613: INFO/job_scheduler] _notify: Set boot_time=2017-06-15 09:59:41+00:00 on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-5) and saved
[2017-06-29 11:24:42,628: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-29 11:24:44,790: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/66a504ec-39ab-4976-a987-aa1c3ae3580d
[2017-06-29 11:24:44,790: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/66a504ec-39ab-4976-a987-aa1c3ae3580d
[2017-06-29 11:24:44,883: INFO/job_scheduler] Job 35 finished 1 steps successfully
[2017-06-29 11:24:44,967: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 11:24:44,967: INFO/job_scheduler]   Job 35 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-06-29 11:24:44,967: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 2 tasked)
[2017-06-29 11:24:44,978: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 140667427251856
[2017-06-29 11:24:44,986: INFO/job] Job 35: StateChangeJob complete, setting state unconfigured on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:24:44,986: INFO/job] Job 35: not_deleted=True
[2017-06-29 11:24:44,986: INFO/job_scheduler] Job 35: complete_job: Updating cache
[2017-06-29 11:24:44,986: INFO/job_scheduler] Job 35: locked_item 140667427251856 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-29 11:24:44,990: INFO/job_scheduler] Job 35 completing (errored=False, cancelled=False)
[2017-06-29 11:24:45,064: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-4
[2017-06-29 11:24:45,064: INFO/job_scheduler] _notify: Updating .boot_time of item 4 (lotus-32vm7.lotus.hpdd.lab.intel.com) from None to 2017-04-25 07:09:31+00:00
[2017-06-29 11:24:45,069: INFO/job_scheduler] _notify: Set boot_time=2017-04-25 07:09:31+00:00 on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-4) and saved
[2017-06-29 11:24:45,078: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-29 11:24:48,741: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/75745f62-f5cb-4d76-9425-fa61503a1b39
[2017-06-29 11:24:48,741: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/75745f62-f5cb-4d76-9425-fa61503a1b39
[2017-06-29 11:24:48,744: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 11:24:48,744: INFO/job_scheduler]   Job 34 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com undeployed->unconfigured
[2017-06-29 11:24:48,744: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 1 tasked)
[2017-06-29 11:24:48,786: INFO/job_scheduler] Job 34 finished 1 steps successfully
[2017-06-29 11:24:48,827: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com undeployed->unconfigured (intentional=True) 140667427186000
[2017-06-29 11:24:48,832: INFO/job] Job 34: StateChangeJob complete, setting state unconfigured on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-29 11:24:48,833: INFO/job] Job 34: not_deleted=True
[2017-06-29 11:24:48,833: INFO/job_scheduler] Job 34: complete_job: Updating cache
[2017-06-29 11:24:48,833: INFO/job_scheduler] Job 34: locked_item 140667427186000 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-29 11:24:48,836: INFO/job_scheduler] Job 34 completing (errored=False, cancelled=False)
[2017-06-29 11:24:48,883: INFO/job_scheduler] Replaying 1 buffered notifications for ManagedHost-3
[2017-06-29 11:24:48,883: INFO/job_scheduler] _notify: Updating .boot_time of item 3 (lotus-32vm6.lotus.hpdd.lab.intel.com) from None to 2017-04-25 07:09:19+00:00
[2017-06-29 11:24:48,896: INFO/job_scheduler] _notify: Set boot_time=2017-04-25 07:09:19+00:00 on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-3) and saved
[2017-06-29 11:24:48,901: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 11:24:49,713: INFO/job_scheduler] _notify: Updating .properties of item 5 (lotus-32vm8.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.21.1.el7_lustre.x86_64", "zfs_installed": false, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-06-29 11:24:49,719: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.21.1.el7_lustre.x86_64", "zfs_installed": false, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-5) and saved
[2017-06-29 11:24:49,725: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 11:24:49,849: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 11:24:51,725: INFO/job_scheduler] _notify: Updating .properties of item 4 (lotus-32vm7.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": false, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-06-29 11:24:51,731: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": false, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-4) and saved
[2017-06-29 11:24:51,737: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 11:24:51,782: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 11:24:53,347: INFO/job_scheduler] _notify: Updating .properties of item 3 (lotus-32vm6.lotus.hpdd.lab.intel.com) from {} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": false, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-06-29 11:24:53,354: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": false, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-3) and saved
[2017-06-29 11:24:53,362: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 11:24:53,410: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 11:24:59,576: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 11:24:59,582: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-2 transitioning from unconfigured to managed
[2017-06-29 11:24:59,743: INFO/command_plan] Created command 19 (Setup managed host lotus-32vm5.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-06-29 11:24:59,892: INFO/job_scheduler] run_next: 1 runnable jobs of (11 pending, 0 tasked)
[2017-06-29 11:24:59,965: INFO/job_scheduler] Job 37: RunJobThread.run
[2017-06-29 11:24:59,965: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-06-29 11:24:59,986: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8c24f608-5622-4e74-af3c-7ca80f668f8b
[2017-06-29 11:24:59,986: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8c24f608-5622-4e74-af3c-7ca80f668f8b
[2017-06-29 11:25:00,263: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 11:25:00,265: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-3 transitioning from unconfigured to managed
[2017-06-29 11:25:00,410: INFO/command_plan] Created command 20 (Setup managed host lotus-32vm6.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-06-29 11:25:00,474: INFO/job_scheduler] run_next: 1 runnable jobs of (21 pending, 1 tasked)
[2017-06-29 11:25:00,496: INFO/job_scheduler] Job 48: RunJobThread.run
[2017-06-29 11:25:00,497: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-06-29 11:25:00,519: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1515a4bf-5a8b-4f4b-986a-5cc18d7b751d
[2017-06-29 11:25:00,519: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1515a4bf-5a8b-4f4b-986a-5cc18d7b751d
[2017-06-29 11:25:00,700: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 11:25:00,702: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-4 transitioning from unconfigured to managed
[2017-06-29 11:25:00,839: INFO/command_plan] Created command 21 (Setup managed host lotus-32vm7.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-06-29 11:25:00,927: INFO/job_scheduler] run_next: 1 runnable jobs of (31 pending, 2 tasked)
[2017-06-29 11:25:00,994: INFO/job_scheduler] Job 59: RunJobThread.run
[2017-06-29 11:25:00,994: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-06-29 11:25:01,030: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5b0020cd-fef6-4fa0-a016-b4116b41fc58
[2017-06-29 11:25:01,030: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5b0020cd-fef6-4fa0-a016-b4116b41fc58
[2017-06-29 11:25:01,132: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 11:25:01,140: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-5 transitioning from unconfigured to managed
[2017-06-29 11:25:01,303: INFO/command_plan] Created command 22 (Setup managed host lotus-32vm8.lotus.hpdd.lab.intel.com) with 11 jobs
[2017-06-29 11:25:01,404: INFO/job_scheduler] run_next: 1 runnable jobs of (41 pending, 3 tasked)
[2017-06-29 11:25:01,467: INFO/job_scheduler] Job 70: RunJobThread.run
[2017-06-29 11:25:01,468: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com update_profile {'profile': {'initial_state': u'managed', 'managed': True, 'name': u'base_managed_rh7', 'default': False, 'ntp': True, 'worker': False, 'user_selectable': True, 'rsyslog': True, 'ui_description': u'A storage server suitable for creating new HA-enabled filesystem targets', 'corosync2': True, 'pacemaker': True, 'ui_name': u'Managed Storage Server', 'corosync': False}}
[2017-06-29 11:25:01,518: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 08b36c70-ee0e-4dcf-92c7-d565f6292b44
[2017-06-29 11:25:01,518: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 08b36c70-ee0e-4dcf-92c7-d565f6292b44
[2017-06-29 11:25:10,068: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/1d3091b5-ca2c-4dcb-ba9a-138a177b462e
[2017-06-29 11:25:10,068: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/1d3091b5-ca2c-4dcb-ba9a-138a177b462e
[2017-06-29 11:25:10,068: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/1d3091b5-ca2c-4dcb-ba9a-138a177b462e
[2017-06-29 11:25:10,068: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/1d3091b5-ca2c-4dcb-ba9a-138a177b462e
[2017-06-29 11:25:10,069: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8c24f608-5622-4e74-af3c-7ca80f668f8b
[2017-06-29 11:25:10,069: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8c24f608-5622-4e74-af3c-7ca80f668f8b
[2017-06-29 11:25:10,113: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8c24f608-5622-4e74-af3c-7ca80f668f8b
[2017-06-29 11:25:10,113: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8c24f608-5622-4e74-af3c-7ca80f668f8b
[2017-06-29 11:25:10,114: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-06-29 11:25:10.113994+00:00:
[2017-06-29 11:25:10,114: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-06-29 11:25:10.113994+00:00:
[2017-06-29 11:25:10,114: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-06-29 11:25:10,114: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-06-29 11:25:10,128: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 11:25:10,129: INFO/job_scheduler]   Job 37 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-29 11:25:10,129: INFO/job_scheduler]   Job 38 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:25:10,129: INFO/job_scheduler]   Job 39 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:25:10,129: INFO/job_scheduler]   Job 40 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:25:10,129: INFO/job_scheduler]   Job 41 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:25:10,129: INFO/job_scheduler]   Job 44 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:25:10,129: INFO/job_scheduler]   Job 47 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-29 11:25:10,130: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-29 11:25:10,131: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-06-29 11:25:10.131711+00:00:
[2017-06-29 11:25:10,131: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com at 2017-06-29 11:25:10.131711+00:00:
[2017-06-29 11:25:10,131: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-06-29 11:25:10,131: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-06-29 11:25:10,143: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 11:25:10,144: INFO/job_scheduler]   Job 37 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-29 11:25:10,144: INFO/job_scheduler]   Job 38 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:25:10,144: INFO/job_scheduler]   Job 39 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:25:10,144: INFO/job_scheduler]   Job 40 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:25:10,144: INFO/job_scheduler]   Job 41 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:25:10,144: INFO/job_scheduler]   Job 44 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:25:10,144: INFO/job_scheduler]   Job 47 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-29 11:25:10,145: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-29 11:25:10,146: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com restart_agent {}
[2017-06-29 11:25:10,164: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 930a4ccd-b46a-4ccd-b2b3-d73bd160e576
[2017-06-29 11:25:10,164: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 930a4ccd-b46a-4ccd-b2b3-d73bd160e576
[2017-06-29 11:25:10,251: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/1d3091b5-ca2c-4dcb-ba9a-138a177b462e
[2017-06-29 11:25:10,251: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/1d3091b5-ca2c-4dcb-ba9a-138a177b462e
[2017-06-29 11:25:10,251: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/1d3091b5-ca2c-4dcb-ba9a-138a177b462e
[2017-06-29 11:25:10,251: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/1d3091b5-ca2c-4dcb-ba9a-138a177b462e
[2017-06-29 11:25:10,252: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 930a4ccd-b46a-4ccd-b2b3-d73bd160e576
[2017-06-29 11:25:10,252: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 930a4ccd-b46a-4ccd-b2b3-d73bd160e576
[2017-06-29 11:25:10,284: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 930a4ccd-b46a-4ccd-b2b3-d73bd160e576
[2017-06-29 11:25:10,284: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 930a4ccd-b46a-4ccd-b2b3-d73bd160e576
[2017-06-29 11:25:10,284: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old 1d3091b5-ca2c-4dcb-ba9a-138a177b462e)
[2017-06-29 11:25:10,284: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old 1d3091b5-ca2c-4dcb-ba9a-138a177b462e)
[2017-06-29 11:25:39,202: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/1d3091b5-ca2c-4dcb-ba9a-138a177b462e
[2017-06-29 11:25:39,202: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/1d3091b5-ca2c-4dcb-ba9a-138a177b462e
[2017-06-29 11:25:39,202: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 1d3091b5-ca2c-4dcb-ba9a-138a177b462e because session terminated
[2017-06-29 11:25:39,202: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 1d3091b5-ca2c-4dcb-ba9a-138a177b462e because session terminated
[2017-06-29 11:25:39,205: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/837e9998-84f5-4866-9097-3ca80252bd9f
[2017-06-29 11:25:39,205: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/837e9998-84f5-4866-9097-3ca80252bd9f
[2017-06-29 11:25:39,327: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new 837e9998-84f5-4866-9097-3ca80252bd9f
[2017-06-29 11:25:39,327: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new 837e9998-84f5-4866-9097-3ca80252bd9f
[2017-06-29 11:25:39,327: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-29 11:25:39,340: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 142ac66d-b621-494d-a0af-c7c5bdb1204b
[2017-06-29 11:25:39,340: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 142ac66d-b621-494d-a0af-c7c5bdb1204b
[2017-06-29 11:25:39,435: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/837e9998-84f5-4866-9097-3ca80252bd9f
[2017-06-29 11:25:39,435: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/837e9998-84f5-4866-9097-3ca80252bd9f
[2017-06-29 11:25:39,436: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/837e9998-84f5-4866-9097-3ca80252bd9f
[2017-06-29 11:25:39,436: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/837e9998-84f5-4866-9097-3ca80252bd9f
[2017-06-29 11:25:39,436: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 142ac66d-b621-494d-a0af-c7c5bdb1204b
[2017-06-29 11:25:39,436: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 142ac66d-b621-494d-a0af-c7c5bdb1204b
[2017-06-29 11:25:39,454: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 142ac66d-b621-494d-a0af-c7c5bdb1204b
[2017-06-29 11:25:39,454: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 142ac66d-b621-494d-a0af-c7c5bdb1204b
[2017-06-29 11:25:39,455: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-06-29 11:25:39,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cb4adf5f-7ba2-4322-b3e6-6e1538cf10a3
[2017-06-29 11:25:39,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cb4adf5f-7ba2-4322-b3e6-6e1538cf10a3
[2017-06-29 11:25:45,615: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 11:25:45,615: INFO/job_scheduler]   Job 37 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-29 11:25:45,615: INFO/job_scheduler]   Job 38 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:25:45,615: INFO/job_scheduler]   Job 39 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:25:45,615: INFO/job_scheduler]   Job 40 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:25:45,615: INFO/job_scheduler]   Job 41 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:25:45,615: INFO/job_scheduler]   Job 44 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:25:45,615: INFO/job_scheduler]   Job 47 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-29 11:25:45,616: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-29 11:26:23,714: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/a993a546-dcf3-4bdf-bb1c-6c24620e676d
[2017-06-29 11:26:23,714: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/a993a546-dcf3-4bdf-bb1c-6c24620e676d
[2017-06-29 11:26:23,714: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/a993a546-dcf3-4bdf-bb1c-6c24620e676d
[2017-06-29 11:26:23,714: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/a993a546-dcf3-4bdf-bb1c-6c24620e676d
[2017-06-29 11:26:23,714: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 08b36c70-ee0e-4dcf-92c7-d565f6292b44
[2017-06-29 11:26:23,714: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 08b36c70-ee0e-4dcf-92c7-d565f6292b44
[2017-06-29 11:26:23,744: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 08b36c70-ee0e-4dcf-92c7-d565f6292b44
[2017-06-29 11:26:23,744: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 08b36c70-ee0e-4dcf-92c7-d565f6292b44
[2017-06-29 11:26:23,744: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-06-29 11:26:23.744774+00:00:
[2017-06-29 11:26:23,744: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-06-29 11:26:23.744774+00:00:
[2017-06-29 11:26:23,745: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-06-29 11:26:23,745: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-06-29 11:26:23,765: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 11:26:23,765: INFO/job_scheduler]   Job 70 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-29 11:26:23,766: INFO/job_scheduler]   Job 71 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:23,766: INFO/job_scheduler]   Job 72 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:23,766: INFO/job_scheduler]   Job 73 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:23,766: INFO/job_scheduler]   Job 74 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:23,766: INFO/job_scheduler]   Job 78 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:23,766: INFO/job_scheduler]   Job 80 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-29 11:26:23,767: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-29 11:26:23,768: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-06-29 11:26:23.768400+00:00:
[2017-06-29 11:26:23,768: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com at 2017-06-29 11:26:23.768400+00:00:
[2017-06-29 11:26:23,768: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-06-29 11:26:23,768: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-06-29 11:26:23,786: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 11:26:23,786: INFO/job_scheduler]   Job 70 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-29 11:26:23,786: INFO/job_scheduler]   Job 71 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:23,786: INFO/job_scheduler]   Job 72 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:23,786: INFO/job_scheduler]   Job 73 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:23,786: INFO/job_scheduler]   Job 74 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:23,786: INFO/job_scheduler]   Job 78 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:23,787: INFO/job_scheduler]   Job 80 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-29 11:26:23,787: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-29 11:26:23,789: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com restart_agent {}
[2017-06-29 11:26:23,803: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6857e2ee-25f4-4950-8e61-bba509650fc3
[2017-06-29 11:26:23,803: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6857e2ee-25f4-4950-8e61-bba509650fc3
[2017-06-29 11:26:23,848: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/a993a546-dcf3-4bdf-bb1c-6c24620e676d
[2017-06-29 11:26:23,848: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/a993a546-dcf3-4bdf-bb1c-6c24620e676d
[2017-06-29 11:26:23,849: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/a993a546-dcf3-4bdf-bb1c-6c24620e676d
[2017-06-29 11:26:23,849: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/a993a546-dcf3-4bdf-bb1c-6c24620e676d
[2017-06-29 11:26:23,849: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6857e2ee-25f4-4950-8e61-bba509650fc3
[2017-06-29 11:26:23,849: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6857e2ee-25f4-4950-8e61-bba509650fc3
[2017-06-29 11:26:23,870: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6857e2ee-25f4-4950-8e61-bba509650fc3
[2017-06-29 11:26:23,870: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6857e2ee-25f4-4950-8e61-bba509650fc3
[2017-06-29 11:26:23,870: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm8.lotus.hpdd.lab.intel.com (old a993a546-dcf3-4bdf-bb1c-6c24620e676d)
[2017-06-29 11:26:23,870: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm8.lotus.hpdd.lab.intel.com (old a993a546-dcf3-4bdf-bb1c-6c24620e676d)
[2017-06-29 11:26:25,900: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/66a504ec-39ab-4976-a987-aa1c3ae3580d
[2017-06-29 11:26:25,900: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/66a504ec-39ab-4976-a987-aa1c3ae3580d
[2017-06-29 11:26:25,900: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/66a504ec-39ab-4976-a987-aa1c3ae3580d
[2017-06-29 11:26:25,900: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/66a504ec-39ab-4976-a987-aa1c3ae3580d
[2017-06-29 11:26:25,901: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5b0020cd-fef6-4fa0-a016-b4116b41fc58
[2017-06-29 11:26:25,901: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5b0020cd-fef6-4fa0-a016-b4116b41fc58
[2017-06-29 11:26:25,913: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5b0020cd-fef6-4fa0-a016-b4116b41fc58
[2017-06-29 11:26:25,913: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5b0020cd-fef6-4fa0-a016-b4116b41fc58
[2017-06-29 11:26:25,913: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-06-29 11:26:25.913491+00:00:
[2017-06-29 11:26:25,913: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-06-29 11:26:25.913491+00:00:
[2017-06-29 11:26:25,913: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-06-29 11:26:25,913: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-06-29 11:26:25,933: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 11:26:25,934: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-06-29 11:26:25.933965+00:00:
[2017-06-29 11:26:25,934: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com at 2017-06-29 11:26:25.933965+00:00:
[2017-06-29 11:26:25,934: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-06-29 11:26:25,934: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-06-29 11:26:25,933: INFO/job_scheduler]   Job 59 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-29 11:26:25,942: INFO/job_scheduler]   Job 60 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:25,942: INFO/job_scheduler]   Job 61 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:25,942: INFO/job_scheduler]   Job 62 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:25,942: INFO/job_scheduler]   Job 63 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:25,942: INFO/job_scheduler]   Job 67 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:25,943: INFO/job_scheduler]   Job 69 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-29 11:26:25,943: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-29 11:26:25,957: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com restart_agent {}
[2017-06-29 11:26:25,966: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 11:26:25,967: INFO/job_scheduler]   Job 59 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-29 11:26:25,967: INFO/job_scheduler]   Job 60 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:25,967: INFO/job_scheduler]   Job 61 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:25,967: INFO/job_scheduler]   Job 62 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:25,967: INFO/job_scheduler]   Job 63 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:25,967: INFO/job_scheduler]   Job 67 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:25,967: INFO/job_scheduler]   Job 69 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-29 11:26:25,968: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-29 11:26:25,980: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 16efdf6e-dc79-4945-9b06-83c2f89e5574
[2017-06-29 11:26:25,980: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 16efdf6e-dc79-4945-9b06-83c2f89e5574
[2017-06-29 11:26:26,069: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/66a504ec-39ab-4976-a987-aa1c3ae3580d
[2017-06-29 11:26:26,069: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/66a504ec-39ab-4976-a987-aa1c3ae3580d
[2017-06-29 11:26:26,069: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/66a504ec-39ab-4976-a987-aa1c3ae3580d
[2017-06-29 11:26:26,069: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/66a504ec-39ab-4976-a987-aa1c3ae3580d
[2017-06-29 11:26:26,069: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 16efdf6e-dc79-4945-9b06-83c2f89e5574
[2017-06-29 11:26:26,069: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 16efdf6e-dc79-4945-9b06-83c2f89e5574
[2017-06-29 11:26:26,100: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 16efdf6e-dc79-4945-9b06-83c2f89e5574
[2017-06-29 11:26:26,100: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 16efdf6e-dc79-4945-9b06-83c2f89e5574
[2017-06-29 11:26:26,100: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm7.lotus.hpdd.lab.intel.com (old 66a504ec-39ab-4976-a987-aa1c3ae3580d)
[2017-06-29 11:26:26,100: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm7.lotus.hpdd.lab.intel.com (old 66a504ec-39ab-4976-a987-aa1c3ae3580d)
[2017-06-29 11:26:26,354: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/75745f62-f5cb-4d76-9425-fa61503a1b39
[2017-06-29 11:26:26,354: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/75745f62-f5cb-4d76-9425-fa61503a1b39
[2017-06-29 11:26:26,355: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/75745f62-f5cb-4d76-9425-fa61503a1b39
[2017-06-29 11:26:26,355: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/75745f62-f5cb-4d76-9425-fa61503a1b39
[2017-06-29 11:26:26,355: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1515a4bf-5a8b-4f4b-986a-5cc18d7b751d
[2017-06-29 11:26:26,355: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1515a4bf-5a8b-4f4b-986a-5cc18d7b751d
[2017-06-29 11:26:26,401: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1515a4bf-5a8b-4f4b-986a-5cc18d7b751d
[2017-06-29 11:26:26,401: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1515a4bf-5a8b-4f4b-986a-5cc18d7b751d
[2017-06-29 11:26:26,403: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-06-29 11:26:26.403387+00:00:
[2017-06-29 11:26:26,403: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-06-29 11:26:26.403387+00:00:
[2017-06-29 11:26:26,403: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-06-29 11:26:26,403: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .server_profile_id base_managed_rh7->base_managed_rh7
[2017-06-29 11:26:26,416: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 11:26:26,416: INFO/job_scheduler]   Job 48 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-29 11:26:26,416: INFO/job_scheduler]   Job 49 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:26,416: INFO/job_scheduler]   Job 50 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:26,417: INFO/job_scheduler]   Job 51 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:26,417: INFO/job_scheduler]   Job 52 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:26,417: INFO/job_scheduler]   Job 55 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:26,417: INFO/job_scheduler]   Job 58 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-29 11:26:26,417: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-29 11:26:26,419: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-06-29 11:26:26.419131+00:00:
[2017-06-29 11:26:26,419: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm6.lotus.hpdd.lab.intel.com at 2017-06-29 11:26:26.419131+00:00:
[2017-06-29 11:26:26,419: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-06-29 11:26:26,419: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .immutable_state True->False
[2017-06-29 11:26:26,431: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 11:26:26,431: INFO/job_scheduler]   Job 48 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-29 11:26:26,431: INFO/job_scheduler]   Job 49 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:26,431: INFO/job_scheduler]   Job 50 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:26,432: INFO/job_scheduler]   Job 51 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:26,432: INFO/job_scheduler]   Job 52 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:26,432: INFO/job_scheduler]   Job 55 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:26,432: INFO/job_scheduler]   Job 58 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-29 11:26:26,432: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-29 11:26:26,433: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com restart_agent {}
[2017-06-29 11:26:26,450: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 13cf333f-9d16-496e-9d2e-45266f2aded8
[2017-06-29 11:26:26,450: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 13cf333f-9d16-496e-9d2e-45266f2aded8
[2017-06-29 11:26:26,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/75745f62-f5cb-4d76-9425-fa61503a1b39
[2017-06-29 11:26:26,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/75745f62-f5cb-4d76-9425-fa61503a1b39
[2017-06-29 11:26:26,527: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/75745f62-f5cb-4d76-9425-fa61503a1b39
[2017-06-29 11:26:26,527: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/75745f62-f5cb-4d76-9425-fa61503a1b39
[2017-06-29 11:26:26,527: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 13cf333f-9d16-496e-9d2e-45266f2aded8
[2017-06-29 11:26:26,527: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 13cf333f-9d16-496e-9d2e-45266f2aded8
[2017-06-29 11:26:26,572: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 13cf333f-9d16-496e-9d2e-45266f2aded8
[2017-06-29 11:26:26,572: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 13cf333f-9d16-496e-9d2e-45266f2aded8
[2017-06-29 11:26:26,572: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm6.lotus.hpdd.lab.intel.com (old 75745f62-f5cb-4d76-9425-fa61503a1b39)
[2017-06-29 11:26:26,572: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm6.lotus.hpdd.lab.intel.com (old 75745f62-f5cb-4d76-9425-fa61503a1b39)
[2017-06-29 11:26:38,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/837e9998-84f5-4866-9097-3ca80252bd9f
[2017-06-29 11:26:38,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/837e9998-84f5-4866-9097-3ca80252bd9f
[2017-06-29 11:26:38,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/837e9998-84f5-4866-9097-3ca80252bd9f
[2017-06-29 11:26:38,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/837e9998-84f5-4866-9097-3ca80252bd9f
[2017-06-29 11:26:38,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cb4adf5f-7ba2-4322-b3e6-6e1538cf10a3
[2017-06-29 11:26:38,212: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cb4adf5f-7ba2-4322-b3e6-6e1538cf10a3
[2017-06-29 11:26:38,237: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cb4adf5f-7ba2-4322-b3e6-6e1538cf10a3
[2017-06-29 11:26:38,237: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cb4adf5f-7ba2-4322-b3e6-6e1538cf10a3
[2017-06-29 11:26:38,731: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com install_packages {'repos': [u'iml-agent'], 'packages': [u'zfs', u'zfs-dkms', u'kernel-devel-[0-9]*_lustre', u'kernel-[0-9]*_lustre', u'lustre-osd-zfs-mount', u'kmod-lustre-osd-ldiskfs', u'lustre-dkms', u'lustre', u'chroma-agent-management']}
[2017-06-29 11:26:38,745: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 128b201c-f296-4b2e-ae68-b14672c93472
[2017-06-29 11:26:38,745: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 128b201c-f296-4b2e-ae68-b14672c93472
[2017-06-29 11:26:43,546: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/a993a546-dcf3-4bdf-bb1c-6c24620e676d
[2017-06-29 11:26:43,546: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/a993a546-dcf3-4bdf-bb1c-6c24620e676d
[2017-06-29 11:26:43,546: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session a993a546-dcf3-4bdf-bb1c-6c24620e676d because session terminated
[2017-06-29 11:26:43,546: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session a993a546-dcf3-4bdf-bb1c-6c24620e676d because session terminated
[2017-06-29 11:26:43,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:26:43,549: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:26:43,896: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm8.lotus.hpdd.lab.intel.com new 6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:26:43,896: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm8.lotus.hpdd.lab.intel.com new 6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:26:43,899: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-29 11:26:43,915: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5957aeb1-8bce-4066-8207-82492de930d5
[2017-06-29 11:26:43,915: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5957aeb1-8bce-4066-8207-82492de930d5
[2017-06-29 11:26:44,015: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:26:44,015: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:26:44,015: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:26:44,015: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:26:44,015: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5957aeb1-8bce-4066-8207-82492de930d5
[2017-06-29 11:26:44,015: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5957aeb1-8bce-4066-8207-82492de930d5
[2017-06-29 11:26:44,036: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5957aeb1-8bce-4066-8207-82492de930d5
[2017-06-29 11:26:44,036: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5957aeb1-8bce-4066-8207-82492de930d5
[2017-06-29 11:26:44,036: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-06-29 11:26:44,056: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3c9c0d05-12fd-4183-9097-67e0a6b52cbf
[2017-06-29 11:26:44,056: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3c9c0d05-12fd-4183-9097-67e0a6b52cbf
[2017-06-29 11:26:45,795: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/66a504ec-39ab-4976-a987-aa1c3ae3580d
[2017-06-29 11:26:45,795: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/66a504ec-39ab-4976-a987-aa1c3ae3580d
[2017-06-29 11:26:45,795: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 66a504ec-39ab-4976-a987-aa1c3ae3580d because session terminated
[2017-06-29 11:26:45,795: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 66a504ec-39ab-4976-a987-aa1c3ae3580d because session terminated
[2017-06-29 11:26:45,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:26:45,798: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:26:46,103: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/75745f62-f5cb-4d76-9425-fa61503a1b39
[2017-06-29 11:26:46,103: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/75745f62-f5cb-4d76-9425-fa61503a1b39
[2017-06-29 11:26:46,103: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 75745f62-f5cb-4d76-9425-fa61503a1b39 because session terminated
[2017-06-29 11:26:46,103: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 75745f62-f5cb-4d76-9425-fa61503a1b39 because session terminated
[2017-06-29 11:26:46,105: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 11:26:46,105: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 11:26:46,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm7.lotus.hpdd.lab.intel.com new ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:26:46,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm7.lotus.hpdd.lab.intel.com new ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:26:46,125: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-29 11:26:46,148: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc db207254-90ad-4a8a-bb13-69a0cc960bc5
[2017-06-29 11:26:46,148: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc db207254-90ad-4a8a-bb13-69a0cc960bc5
[2017-06-29 11:26:46,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm6.lotus.hpdd.lab.intel.com new e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 11:26:46,595: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm6.lotus.hpdd.lab.intel.com new e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 11:26:46,596: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-29 11:26:46,623: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9f0a6eef-381f-453c-8e05-452ba6009c9f
[2017-06-29 11:26:46,623: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9f0a6eef-381f-453c-8e05-452ba6009c9f
[2017-06-29 11:26:46,741: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 11:26:46,741: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 11:26:46,742: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 11:26:46,742: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 11:26:46,742: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9f0a6eef-381f-453c-8e05-452ba6009c9f
[2017-06-29 11:26:46,742: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9f0a6eef-381f-453c-8e05-452ba6009c9f
[2017-06-29 11:26:46,749: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9f0a6eef-381f-453c-8e05-452ba6009c9f
[2017-06-29 11:26:46,749: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9f0a6eef-381f-453c-8e05-452ba6009c9f
[2017-06-29 11:26:46,749: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-06-29 11:26:46,793: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6e0a7572-c92c-4c27-90e7-04a3f9fd0c8a
[2017-06-29 11:26:46,793: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6e0a7572-c92c-4c27-90e7-04a3f9fd0c8a
[2017-06-29 11:26:47,239: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:26:47,239: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:26:47,239: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:26:47,239: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:26:47,239: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc db207254-90ad-4a8a-bb13-69a0cc960bc5
[2017-06-29 11:26:47,239: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc db207254-90ad-4a8a-bb13-69a0cc960bc5
[2017-06-29 11:26:47,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc db207254-90ad-4a8a-bb13-69a0cc960bc5
[2017-06-29 11:26:47,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc db207254-90ad-4a8a-bb13-69a0cc960bc5
[2017-06-29 11:26:47,270: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux'}
[2017-06-29 11:26:47,287: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 43f17b16-8cbb-469d-bc95-c27344dc27e7
[2017-06-29 11:26:47,287: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 43f17b16-8cbb-469d-bc95-c27344dc27e7
[2017-06-29 11:26:47,782: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 11:26:47,782: INFO/job_scheduler]   Job 70 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-29 11:26:47,782: INFO/job_scheduler]   Job 71 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:47,782: INFO/job_scheduler]   Job 72 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:47,782: INFO/job_scheduler]   Job 73 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:47,782: INFO/job_scheduler]   Job 74 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:47,782: INFO/job_scheduler]   Job 78 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:47,783: INFO/job_scheduler]   Job 80 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-29 11:26:47,783: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-29 11:26:50,449: INFO/job_scheduler] _notify: Buffering update to lotus-32vm6.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 11:26:50,450: INFO/job_scheduler]   Job 48 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-29 11:26:50,450: INFO/job_scheduler]   Job 49 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:50,450: INFO/job_scheduler]   Job 50 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:50,450: INFO/job_scheduler]   Job 51 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:50,450: INFO/job_scheduler]   Job 52 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:50,450: INFO/job_scheduler]   Job 55 readlock on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:50,450: INFO/job_scheduler]   Job 58 writelock on lotus-32vm6.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-29 11:26:50,451: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-29 11:26:51,071: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 11:26:51,071: INFO/job_scheduler]   Job 59 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-29 11:26:51,072: INFO/job_scheduler]   Job 60 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:51,072: INFO/job_scheduler]   Job 61 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:51,072: INFO/job_scheduler]   Job 62 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:51,072: INFO/job_scheduler]   Job 63 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:51,072: INFO/job_scheduler]   Job 67 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:26:51,072: INFO/job_scheduler]   Job 69 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-29 11:26:51,073: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-29 11:27:17,425: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:27:17,425: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:27:17,425: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:27:17,425: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:27:17,426: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3c9c0d05-12fd-4183-9097-67e0a6b52cbf
[2017-06-29 11:27:17,426: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3c9c0d05-12fd-4183-9097-67e0a6b52cbf
[2017-06-29 11:27:17,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3c9c0d05-12fd-4183-9097-67e0a6b52cbf
[2017-06-29 11:27:17,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3c9c0d05-12fd-4183-9097-67e0a6b52cbf
[2017-06-29 11:27:18,148: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com install_packages {'repos': [u'iml-agent'], 'packages': [u'zfs', u'zfs-dkms', u'kernel-devel-[0-9]*_lustre', u'kernel-[0-9]*_lustre', u'lustre-osd-zfs-mount', u'kmod-lustre-osd-ldiskfs', u'lustre-dkms', u'lustre', u'chroma-agent-management']}
[2017-06-29 11:27:18,177: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bcf1bcc8-a214-4b6a-a8f1-38fd97ec5084
[2017-06-29 11:27:18,177: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bcf1bcc8-a214-4b6a-a8f1-38fd97ec5084
[2017-06-29 11:27:18,214: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 11:27:18,214: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 11:27:18,215: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 11:27:18,215: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 11:27:18,215: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6e0a7572-c92c-4c27-90e7-04a3f9fd0c8a
[2017-06-29 11:27:18,215: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6e0a7572-c92c-4c27-90e7-04a3f9fd0c8a
[2017-06-29 11:27:18,238: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6e0a7572-c92c-4c27-90e7-04a3f9fd0c8a
[2017-06-29 11:27:18,238: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6e0a7572-c92c-4c27-90e7-04a3f9fd0c8a
[2017-06-29 11:27:18,630: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com install_packages {'repos': [u'iml-agent'], 'packages': [u'zfs', u'zfs-dkms', u'kernel-devel-[0-9]*_lustre', u'kernel-[0-9]*_lustre', u'lustre-osd-zfs-mount', u'kmod-lustre-osd-ldiskfs', u'lustre-dkms', u'lustre', u'chroma-agent-management']}
[2017-06-29 11:27:18,644: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a6d6ec84-7171-46b9-a596-486039ae6a61
[2017-06-29 11:27:18,644: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a6d6ec84-7171-46b9-a596-486039ae6a61
[2017-06-29 11:27:18,686: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:27:18,686: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:27:18,687: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:27:18,687: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:27:18,687: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 43f17b16-8cbb-469d-bc95-c27344dc27e7
[2017-06-29 11:27:18,687: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 43f17b16-8cbb-469d-bc95-c27344dc27e7
[2017-06-29 11:27:18,690: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 43f17b16-8cbb-469d-bc95-c27344dc27e7
[2017-06-29 11:27:18,690: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 43f17b16-8cbb-469d-bc95-c27344dc27e7
[2017-06-29 11:27:18,952: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com install_packages {'repos': [u'iml-agent'], 'packages': [u'zfs', u'zfs-dkms', u'kernel-devel-[0-9]*_lustre', u'kernel-[0-9]*_lustre', u'lustre-osd-zfs-mount', u'kmod-lustre-osd-ldiskfs', u'lustre-dkms', u'lustre', u'chroma-agent-management']}
[2017-06-29 11:27:18,970: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7dcd4dae-db7c-4baa-bcc7-f01e3b5146c3
[2017-06-29 11:27:18,970: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7dcd4dae-db7c-4baa-bcc7-f01e3b5146c3
[2017-06-29 11:27:22,866: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 11:27:22,867: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-29 11:27:22,918: INFO/command_plan] add_jobs: created Job 81 (Trigger plugin poll for linux plugins)
[2017-06-29 11:27:22,931: INFO/job_scheduler] run_next: 1 runnable jobs of (41 pending, 4 tasked)
[2017-06-29 11:27:22,953: INFO/job_scheduler] Job 81: RunJobThread.run
[2017-06-29 11:27:22,988: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 11:27:23,041: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2e3f7c08-e9c6-4f3f-b65b-f2842fd375fc
[2017-06-29 11:27:23,041: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2e3f7c08-e9c6-4f3f-b65b-f2842fd375fc
[2017-06-29 11:27:23,044: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 11:27:23,112: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6cd00eb7-e1a7-43f7-8b49-77099c1f2cf9
[2017-06-29 11:27:23,112: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6cd00eb7-e1a7-43f7-8b49-77099c1f2cf9
[2017-06-29 11:27:23,114: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 11:27:23,114: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 11:27:23,114: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 11:27:23,114: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 11:27:23,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2e3f7c08-e9c6-4f3f-b65b-f2842fd375fc
[2017-06-29 11:27:23,115: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2e3f7c08-e9c6-4f3f-b65b-f2842fd375fc
[2017-06-29 11:27:23,114: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 11:27:23,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2e3f7c08-e9c6-4f3f-b65b-f2842fd375fc
[2017-06-29 11:27:23,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2e3f7c08-e9c6-4f3f-b65b-f2842fd375fc
[2017-06-29 11:27:23,123: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 11:27:23,160: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4f557f32-0f8f-4a40-a32b-87f5e9df879a
[2017-06-29 11:27:23,160: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4f557f32-0f8f-4a40-a32b-87f5e9df879a
[2017-06-29 11:27:23,230: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:27:23,230: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:27:23,230: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:27:23,230: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:27:23,231: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6cd00eb7-e1a7-43f7-8b49-77099c1f2cf9
[2017-06-29 11:27:23,231: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6cd00eb7-e1a7-43f7-8b49-77099c1f2cf9
[2017-06-29 11:27:23,240: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6cd00eb7-e1a7-43f7-8b49-77099c1f2cf9
[2017-06-29 11:27:23,240: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6cd00eb7-e1a7-43f7-8b49-77099c1f2cf9
[2017-06-29 11:27:23,240: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 11:27:23,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:27:23,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:27:23,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:27:23,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:27:23,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4f557f32-0f8f-4a40-a32b-87f5e9df879a
[2017-06-29 11:27:23,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4f557f32-0f8f-4a40-a32b-87f5e9df879a
[2017-06-29 11:27:23,281: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4f557f32-0f8f-4a40-a32b-87f5e9df879a
[2017-06-29 11:27:23,281: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4f557f32-0f8f-4a40-a32b-87f5e9df879a
[2017-06-29 11:27:23,282: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 11:27:23,285: INFO/job_scheduler] Job 81 finished 1 steps successfully
[2017-06-29 11:27:23,313: INFO/job_scheduler] Job 81: complete_job: Updating cache
[2017-06-29 11:27:23,313: INFO/job_scheduler] Job 81 completing (errored=False, cancelled=False)
[2017-06-29 11:27:23,436: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-29 11:27:38,938: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/837e9998-84f5-4866-9097-3ca80252bd9f
[2017-06-29 11:27:38,938: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/837e9998-84f5-4866-9097-3ca80252bd9f
[2017-06-29 11:27:38,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/837e9998-84f5-4866-9097-3ca80252bd9f
[2017-06-29 11:27:38,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/837e9998-84f5-4866-9097-3ca80252bd9f
[2017-06-29 11:27:38,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 128b201c-f296-4b2e-ae68-b14672c93472
[2017-06-29 11:27:38,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 128b201c-f296-4b2e-ae68-b14672c93472
[2017-06-29 11:27:38,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 128b201c-f296-4b2e-ae68-b14672c93472
[2017-06-29 11:27:38,976: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 128b201c-f296-4b2e-ae68-b14672c93472
[2017-06-29 11:27:39,077: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com kernel_status {}
[2017-06-29 11:27:39,116: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0746ac76-d35a-48d2-b289-21c05776c74f
[2017-06-29 11:27:39,116: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0746ac76-d35a-48d2-b289-21c05776c74f
[2017-06-29 11:27:41,216: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/837e9998-84f5-4866-9097-3ca80252bd9f
[2017-06-29 11:27:41,216: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/837e9998-84f5-4866-9097-3ca80252bd9f
[2017-06-29 11:27:41,216: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/837e9998-84f5-4866-9097-3ca80252bd9f
[2017-06-29 11:27:41,216: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/837e9998-84f5-4866-9097-3ca80252bd9f
[2017-06-29 11:27:41,216: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0746ac76-d35a-48d2-b289-21c05776c74f
[2017-06-29 11:27:41,216: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0746ac76-d35a-48d2-b289-21c05776c74f
[2017-06-29 11:27:41,237: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0746ac76-d35a-48d2-b289-21c05776c74f
[2017-06-29 11:27:41,237: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0746ac76-d35a-48d2-b289-21c05776c74f
[2017-06-29 11:27:41,238: INFO/job] Job 37 RebootIfNeededStep: Reboot of lotus-32vm5.lotus.hpdd.lab.intel.com required to switch from running kernel kernel-3.10.0-514.21.1.el7_lustre.x86_64 to required kernel-3.10.0-514.10.2.el7_lustre.x86_64
[2017-06-29 11:27:41,238: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com reboot_server {}
[2017-06-29 11:27:41,258: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 931a054b-8ce1-48b3-a422-ccd6fb98ccab
[2017-06-29 11:27:41,258: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 931a054b-8ce1-48b3-a422-ccd6fb98ccab
[2017-06-29 11:27:41,406: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/837e9998-84f5-4866-9097-3ca80252bd9f
[2017-06-29 11:27:41,406: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/837e9998-84f5-4866-9097-3ca80252bd9f
[2017-06-29 11:27:41,407: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/837e9998-84f5-4866-9097-3ca80252bd9f
[2017-06-29 11:27:41,407: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/837e9998-84f5-4866-9097-3ca80252bd9f
[2017-06-29 11:27:41,407: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 931a054b-8ce1-48b3-a422-ccd6fb98ccab
[2017-06-29 11:27:41,407: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 931a054b-8ce1-48b3-a422-ccd6fb98ccab
[2017-06-29 11:27:41,431: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 931a054b-8ce1-48b3-a422-ccd6fb98ccab
[2017-06-29 11:27:41,431: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 931a054b-8ce1-48b3-a422-ccd6fb98ccab
[2017-06-29 11:27:41,431: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old 837e9998-84f5-4866-9097-3ca80252bd9f)
[2017-06-29 11:27:41,431: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm5.lotus.hpdd.lab.intel.com (old 837e9998-84f5-4866-9097-3ca80252bd9f)
[2017-06-29 11:28:42,309: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 11:28:42,309: INFO/job_scheduler]   Job 37 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-29 11:28:42,309: INFO/job_scheduler]   Job 38 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:28:42,310: INFO/job_scheduler]   Job 39 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:28:42,310: INFO/job_scheduler]   Job 40 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:28:42,310: INFO/job_scheduler]   Job 41 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:28:42,310: INFO/job_scheduler]   Job 44 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:28:42,310: INFO/job_scheduler]   Job 47 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-29 11:28:42,311: INFO/job_scheduler] run_next: 0 runnable jobs of (40 pending, 4 tasked)
[2017-06-29 11:28:42,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/837e9998-84f5-4866-9097-3ca80252bd9f
[2017-06-29 11:28:42,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/837e9998-84f5-4866-9097-3ca80252bd9f
[2017-06-29 11:28:42,412: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 837e9998-84f5-4866-9097-3ca80252bd9f because session terminated
[2017-06-29 11:28:42,412: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 837e9998-84f5-4866-9097-3ca80252bd9f because session terminated
[2017-06-29 11:28:42,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:28:42,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:28:42,510: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new 4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:28:42,510: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm5.lotus.hpdd.lab.intel.com new 4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:28:42,510: INFO/job_scheduler] Job 37 finished 4 steps successfully
[2017-06-29 11:28:42,538: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 140667427258064
[2017-06-29 11:28:42,543: INFO/job] Job 37: StateChangeJob complete, setting state packages_installed on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:28:42,543: INFO/job] Job 37: not_deleted=True
[2017-06-29 11:28:42,543: INFO/job_scheduler] Job 37: complete_job: Updating cache
[2017-06-29 11:28:42,543: INFO/job_scheduler] Job 37: locked_item 140668098672336 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-29 11:28:42,552: INFO/job_scheduler] Job 37 completing (errored=False, cancelled=False)
[2017-06-29 11:28:42,572: INFO/job_scheduler] run_next: 4 runnable jobs of (40 pending, 3 tasked)
[2017-06-29 11:28:42,608: INFO/job_scheduler] Job 38: RunJobThread.run
[2017-06-29 11:28:42,609: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-06-29 11:28:42,623: INFO/job_scheduler] Job 39: RunJobThread.run
[2017-06-29 11:28:42,627: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 068a1337-6334-4523-9ab1-87cf13002a46
[2017-06-29 11:28:42,627: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 068a1337-6334-4523-9ab1-87cf13002a46
[2017-06-29 11:28:42,628: INFO/job_scheduler] Job 40: RunJobThread.run
[2017-06-29 11:28:42,629: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-06-29 11:28:42,636: INFO/job_scheduler] Job 41: open DB connection during completion
[2017-06-29 11:28:42,637: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-06-29 11:28:42,642: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 585d2811-ba0d-4578-967b-1336f6c22a4b
[2017-06-29 11:28:42,642: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 585d2811-ba0d-4578-967b-1336f6c22a4b
[2017-06-29 11:28:42,675: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 06454145-2fad-42e5-a028-675f55103166
[2017-06-29 11:28:42,675: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 06454145-2fad-42e5-a028-675f55103166
[2017-06-29 11:28:42,740: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 140668098672720
[2017-06-29 11:28:42,748: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-06-29 11:28:42,759: INFO/job] Job 41: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-29 11:28:42,759: INFO/job] Job 41: not_deleted=True
[2017-06-29 11:28:42,759: INFO/job_scheduler] Job 41: complete_job: Updating cache
[2017-06-29 11:28:42,760: INFO/job_scheduler] Job 41: locked_item 140668098672720 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-29 11:28:42,769: INFO/job_scheduler] Job 41 completing (errored=False, cancelled=False)
[2017-06-29 11:28:42,796: INFO/job_scheduler] run_next: 1 runnable jobs of (36 pending, 6 tasked)
[2017-06-29 11:28:42,816: INFO/job_scheduler] Job 42: RunJobThread.run
[2017-06-29 11:28:42,816: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com load_lnet {}
[2017-06-29 11:28:42,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c34c3b63-a004-4c49-a7fe-1d8c82fb488a
[2017-06-29 11:28:42,859: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c34c3b63-a004-4c49-a7fe-1d8c82fb488a
[2017-06-29 11:28:43,927: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:28:43,927: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:28:43,928: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:28:43,928: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:28:43,928: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c34c3b63-a004-4c49-a7fe-1d8c82fb488a
[2017-06-29 11:28:43,928: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c34c3b63-a004-4c49-a7fe-1d8c82fb488a
[2017-06-29 11:28:43,930: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c34c3b63-a004-4c49-a7fe-1d8c82fb488a
[2017-06-29 11:28:43,930: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c34c3b63-a004-4c49-a7fe-1d8c82fb488a
[2017-06-29 11:28:43,931: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-29 11:28:43,974: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e52d1b14-63a1-4a93-ae97-92a36a12d29e
[2017-06-29 11:28:43,974: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e52d1b14-63a1-4a93-ae97-92a36a12d29e
[2017-06-29 11:28:44,057: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:28:44,057: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:28:44,058: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:28:44,058: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:28:44,058: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e52d1b14-63a1-4a93-ae97-92a36a12d29e
[2017-06-29 11:28:44,058: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e52d1b14-63a1-4a93-ae97-92a36a12d29e
[2017-06-29 11:28:44,098: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e52d1b14-63a1-4a93-ae97-92a36a12d29e
[2017-06-29 11:28:44,098: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e52d1b14-63a1-4a93-ae97-92a36a12d29e
[2017-06-29 11:28:44,202: INFO/job_scheduler] Job 42 finished 2 steps successfully
[2017-06-29 11:28:44,238: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 140668098672720
[2017-06-29 11:28:44,250: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-06-29 11:28:44,264: INFO/job] Job 42: StateChangeJob complete, setting state lnet_down on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-29 11:28:44,264: INFO/job] Job 42: not_deleted=True
[2017-06-29 11:28:44,264: INFO/job_scheduler] Job 42: complete_job: Updating cache
[2017-06-29 11:28:44,264: INFO/job_scheduler] Job 42: locked_item 140668098672720 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-29 11:28:44,271: INFO/job_scheduler] Job 42 completing (errored=False, cancelled=False)
[2017-06-29 11:28:44,310: INFO/job_scheduler] run_next: 1 runnable jobs of (35 pending, 6 tasked)
[2017-06-29 11:28:44,344: INFO/job_scheduler] Job 45: RunJobThread.run
[2017-06-29 11:28:44,345: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_lnet {}
[2017-06-29 11:28:44,382: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 49d399a4-5e48-4314-b221-10a68a529de6
[2017-06-29 11:28:44,382: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 49d399a4-5e48-4314-b221-10a68a529de6
[2017-06-29 11:28:44,490: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:28:44,490: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:28:44,490: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:28:44,490: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:28:44,490: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 49d399a4-5e48-4314-b221-10a68a529de6
[2017-06-29 11:28:44,490: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 49d399a4-5e48-4314-b221-10a68a529de6
[2017-06-29 11:28:44,507: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 49d399a4-5e48-4314-b221-10a68a529de6
[2017-06-29 11:28:44,507: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 49d399a4-5e48-4314-b221-10a68a529de6
[2017-06-29 11:28:44,508: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-29 11:28:44,530: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 17b8a6c6-e0c7-4ac6-9bf5-13a640da82bf
[2017-06-29 11:28:44,530: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 17b8a6c6-e0c7-4ac6-9bf5-13a640da82bf
[2017-06-29 11:28:45,648: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:28:45,648: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:28:45,648: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:28:45,648: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:28:45,650: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 17b8a6c6-e0c7-4ac6-9bf5-13a640da82bf
[2017-06-29 11:28:45,650: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 17b8a6c6-e0c7-4ac6-9bf5-13a640da82bf
[2017-06-29 11:28:45,698: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 17b8a6c6-e0c7-4ac6-9bf5-13a640da82bf
[2017-06-29 11:28:45,698: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 17b8a6c6-e0c7-4ac6-9bf5-13a640da82bf
[2017-06-29 11:28:45,812: INFO/job_scheduler] Job 45 finished 2 steps successfully
[2017-06-29 11:28:45,871: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 140668098672720
[2017-06-29 11:28:45,905: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 30
[2017-06-29 11:28:45,920: INFO/job] Job 45: StateChangeJob complete, setting state lnet_up on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-29 11:28:45,920: INFO/job] Job 45: not_deleted=True
[2017-06-29 11:28:45,920: INFO/job_scheduler] Job 45: complete_job: Updating cache
[2017-06-29 11:28:45,921: INFO/job_scheduler] Job 45: locked_item 140668098672720 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-29 11:28:45,927: INFO/job_scheduler] Job 45 completing (errored=False, cancelled=False)
[2017-06-29 11:28:45,952: INFO/job_scheduler] run_next: 0 runnable jobs of (34 pending, 6 tasked)
[2017-06-29 11:28:46,691: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 11:28:46,691: INFO/job_scheduler]   Job 38 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:28:46,691: INFO/job_scheduler]   Job 39 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:28:46,691: INFO/job_scheduler]   Job 40 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:28:46,692: INFO/job_scheduler]   Job 44 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:28:46,692: INFO/job_scheduler]   Job 47 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-29 11:28:46,692: INFO/job_scheduler] run_next: 0 runnable jobs of (34 pending, 6 tasked)
[2017-06-29 11:28:48,782: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:28:48,782: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:28:48,783: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:28:48,783: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:28:48,783: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 06454145-2fad-42e5-a028-675f55103166
[2017-06-29 11:28:48,783: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 06454145-2fad-42e5-a028-675f55103166
[2017-06-29 11:28:48,800: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 06454145-2fad-42e5-a028-675f55103166
[2017-06-29 11:28:48,800: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 06454145-2fad-42e5-a028-675f55103166
[2017-06-29 11:28:48,800: INFO/job_scheduler] Job 39 finished 1 steps successfully
[2017-06-29 11:28:48,886: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 140668098646544
[2017-06-29 11:28:48,889: INFO/job] Job 39: StateChangeJob complete, setting state configured on lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration
[2017-06-29 11:28:48,889: INFO/job] Job 39: not_deleted=True
[2017-06-29 11:28:48,889: INFO/job_scheduler] Job 39: complete_job: Updating cache
[2017-06-29 11:28:48,889: INFO/job_scheduler] Job 39: locked_item 140668098646544 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-06-29 11:28:48,892: INFO/job_scheduler] Job 39 completing (errored=False, cancelled=False)
[2017-06-29 11:28:48,910: INFO/job_scheduler] run_next: 0 runnable jobs of (34 pending, 5 tasked)
[2017-06-29 11:28:49,857: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:28:49,857: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:28:49,857: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:28:49,857: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:28:49,857: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 068a1337-6334-4523-9ab1-87cf13002a46
[2017-06-29 11:28:49,857: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 068a1337-6334-4523-9ab1-87cf13002a46
[2017-06-29 11:28:49,905: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 068a1337-6334-4523-9ab1-87cf13002a46
[2017-06-29 11:28:49,905: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 068a1337-6334-4523-9ab1-87cf13002a46
[2017-06-29 11:28:49,905: INFO/job_scheduler] Job 38 finished 1 steps successfully
[2017-06-29 11:28:49,962: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 140668098649616
[2017-06-29 11:28:49,965: INFO/job] Job 38: StateChangeJob complete, setting state configured on lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-06-29 11:28:49,965: INFO/job] Job 38: not_deleted=True
[2017-06-29 11:28:49,965: INFO/job_scheduler] Job 38: complete_job: Updating cache
[2017-06-29 11:28:49,965: INFO/job_scheduler] Job 38: locked_item 140668098649616 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-06-29 11:28:49,968: INFO/job_scheduler] Job 38 completing (errored=False, cancelled=False)
[2017-06-29 11:28:49,991: INFO/job_scheduler] run_next: 0 runnable jobs of (34 pending, 4 tasked)
[2017-06-29 11:29:26,493: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:29:26,493: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:29:26,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:29:26,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:29:26,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 585d2811-ba0d-4578-967b-1336f6c22a4b
[2017-06-29 11:29:26,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 585d2811-ba0d-4578-967b-1336f6c22a4b
[2017-06-29 11:29:26,507: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 585d2811-ba0d-4578-967b-1336f6c22a4b
[2017-06-29 11:29:26,507: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 585d2811-ba0d-4578-967b-1336f6c22a4b
[2017-06-29 11:29:26,508: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.224', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-06-29 11:29:26,527: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ef20ac8f-3351-4aa6-b269-bf822f390dd2
[2017-06-29 11:29:26,527: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ef20ac8f-3351-4aa6-b269-bf822f390dd2
[2017-06-29 11:29:26,618: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:29:26,618: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:29:26,618: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:29:26,618: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:29:26,618: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ef20ac8f-3351-4aa6-b269-bf822f390dd2
[2017-06-29 11:29:26,618: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ef20ac8f-3351-4aa6-b269-bf822f390dd2
[2017-06-29 11:29:26,649: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ef20ac8f-3351-4aa6-b269-bf822f390dd2
[2017-06-29 11:29:26,649: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ef20ac8f-3351-4aa6-b269-bf822f390dd2
[2017-06-29 11:29:26,690: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 40759, 'pcs_password': 'netjRpT0yjzpeBc9aYvm'}
[2017-06-29 11:29:26,715: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1522c2d1-5434-4881-a0e5-439b4e873d87
[2017-06-29 11:29:26,715: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1522c2d1-5434-4881-a0e5-439b4e873d87
[2017-06-29 11:29:38,811: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:29:38,811: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:29:38,811: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:29:38,811: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:29:38,811: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1522c2d1-5434-4881-a0e5-439b4e873d87
[2017-06-29 11:29:38,811: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1522c2d1-5434-4881-a0e5-439b4e873d87
[2017-06-29 11:29:38,852: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1522c2d1-5434-4881-a0e5-439b4e873d87
[2017-06-29 11:29:38,852: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1522c2d1-5434-4881-a0e5-439b4e873d87
[2017-06-29 11:29:38,853: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': True, 'mcast_port': 40759, 'new_node_fqdn': u'lotus-32vm5.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'netjRpT0yjzpeBc9aYvm'}
[2017-06-29 11:29:38,926: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bc212c82-0b0f-460b-8556-75eda63cf7f1
[2017-06-29 11:29:38,926: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bc212c82-0b0f-460b-8556-75eda63cf7f1
[2017-06-29 11:29:45,007: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:29:45,007: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:29:45,008: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:29:45,008: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:29:45,008: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bc212c82-0b0f-460b-8556-75eda63cf7f1
[2017-06-29 11:29:45,008: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bc212c82-0b0f-460b-8556-75eda63cf7f1
[2017-06-29 11:29:45,008: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bc212c82-0b0f-460b-8556-75eda63cf7f1
[2017-06-29 11:29:45,008: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bc212c82-0b0f-460b-8556-75eda63cf7f1
[2017-06-29 11:29:45,009: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-06-29 11:29:45.009076+00:00:
[2017-06-29 11:29:45,009: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-06-29 11:29:45.009076+00:00:
[2017-06-29 11:29:45,009: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->40759
[2017-06-29 11:29:45,009: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->40759
[2017-06-29 11:29:45,012: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-06-29 11:29:45,012: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-06-29 11:29:45,035: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-29 11:29:45,035: INFO/job_scheduler]   Job 40 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-06-29 11:29:45,035: INFO/job_scheduler]   Job 43 writelock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-06-29 11:29:45,035: INFO/job_scheduler]   Job 44 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:29:45,035: INFO/job_scheduler]   Job 46 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:29:45,036: INFO/job_scheduler] run_next: 0 runnable jobs of (34 pending, 4 tasked)
[2017-06-29 11:29:45,037: INFO/job_scheduler] Job 40 finished 1 steps successfully
[2017-06-29 11:29:45,114: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 140667427275664
[2017-06-29 11:29:45,123: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-06-29 11:29:45,137: INFO/job] Job 40: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:29:45,137: INFO/job] Job 40: not_deleted=True
[2017-06-29 11:29:45,137: INFO/job_scheduler] Job 40: complete_job: Updating cache
[2017-06-29 11:29:45,138: INFO/job_scheduler] Job 40: locked_item 140667427275664 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-29 11:29:45,149: INFO/job_scheduler] Job 40 completing (errored=False, cancelled=False)
[2017-06-29 11:29:45,272: INFO/job_scheduler] run_next: 1 runnable jobs of (34 pending, 3 tasked)
[2017-06-29 11:29:45,323: INFO/job_scheduler] Job 43: RunJobThread.run
[2017-06-29 11:29:45,323: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-06-29 11:29:45,373: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e6fb95e5-7ee0-49c7-a508-ccbfb4e02e8b
[2017-06-29 11:29:45,373: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e6fb95e5-7ee0-49c7-a508-ccbfb4e02e8b
[2017-06-29 11:29:53,500: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:29:53,500: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:29:53,500: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:29:53,500: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:29:53,500: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e6fb95e5-7ee0-49c7-a508-ccbfb4e02e8b
[2017-06-29 11:29:53,500: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e6fb95e5-7ee0-49c7-a508-ccbfb4e02e8b
[2017-06-29 11:29:53,548: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e6fb95e5-7ee0-49c7-a508-ccbfb4e02e8b
[2017-06-29 11:29:53,548: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e6fb95e5-7ee0-49c7-a508-ccbfb4e02e8b
[2017-06-29 11:29:53,549: INFO/job_scheduler] Job 43 finished 1 steps successfully
[2017-06-29 11:29:53,887: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 140667427275664
[2017-06-29 11:29:53,921: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 30
[2017-06-29 11:29:53,952: INFO/job] Job 43: StateChangeJob complete, setting state started on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:29:53,952: INFO/job] Job 43: not_deleted=True
[2017-06-29 11:29:53,954: INFO/job_scheduler] Job 43: complete_job: Updating cache
[2017-06-29 11:29:53,954: INFO/job_scheduler] Job 43: locked_item 140667427275664 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-29 11:29:53,971: INFO/job_scheduler] Job 43 completing (errored=False, cancelled=False)
[2017-06-29 11:29:54,010: INFO/job_scheduler] run_next: 1 runnable jobs of (33 pending, 3 tasked)
[2017-06-29 11:29:54,047: INFO/job_scheduler] Job 44: RunJobThread.run
[2017-06-29 11:29:54,048: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-06-29 11:29:54,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5a66767e-8ffb-44f3-b1a1-900fe055d5b4
[2017-06-29 11:29:54,087: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5a66767e-8ffb-44f3-b1a1-900fe055d5b4
[2017-06-29 11:29:57,141: INFO/job_scheduler] run_next: 0 runnable jobs of (32 pending, 4 tasked)
[2017-06-29 11:29:57,155: INFO/job_scheduler] run_next: 0 runnable jobs of (32 pending, 4 tasked)
[2017-06-29 11:30:01,228: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:30:01,228: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:30:01,228: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:30:01,228: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:30:01,228: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5a66767e-8ffb-44f3-b1a1-900fe055d5b4
[2017-06-29 11:30:01,228: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5a66767e-8ffb-44f3-b1a1-900fe055d5b4
[2017-06-29 11:30:01,263: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5a66767e-8ffb-44f3-b1a1-900fe055d5b4
[2017-06-29 11:30:01,263: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5a66767e-8ffb-44f3-b1a1-900fe055d5b4
[2017-06-29 11:30:01,264: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-06-29 11:30:01,277: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e2e10deb-acf7-4b9a-806c-dc8a4865d977
[2017-06-29 11:30:01,277: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e2e10deb-acf7-4b9a-806c-dc8a4865d977
[2017-06-29 11:30:07,176: INFO/job_scheduler] run_next: 0 runnable jobs of (32 pending, 4 tasked)
[2017-06-29 11:30:07,191: INFO/job_scheduler] run_next: 0 runnable jobs of (32 pending, 4 tasked)
[2017-06-29 11:30:27,255: INFO/job_scheduler] run_next: 0 runnable jobs of (32 pending, 4 tasked)
[2017-06-29 11:30:27,275: INFO/job_scheduler] run_next: 0 runnable jobs of (32 pending, 4 tasked)
[2017-06-29 11:30:57,394: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-29 11:30:57,395: INFO/job_scheduler]   Job 44 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:30:57,395: INFO/job_scheduler]   Job 46 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:30:57,395: INFO/job_scheduler] run_next: 0 runnable jobs of (32 pending, 4 tasked)
[2017-06-29 11:31:07,400: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-29 11:31:07,401: INFO/job_scheduler]   Job 44 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:31:07,401: INFO/job_scheduler]   Job 46 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:31:07,401: INFO/job_scheduler] run_next: 0 runnable jobs of (32 pending, 4 tasked)
[2017-06-29 11:31:17,363: INFO/job_scheduler] _notify: Buffering update to lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-29 11:31:17,364: INFO/job_scheduler]   Job 44 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:31:17,364: INFO/job_scheduler]   Job 46 readlock on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:31:17,364: INFO/job_scheduler] run_next: 0 runnable jobs of (32 pending, 4 tasked)
[2017-06-29 11:31:22,656: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:31:22,656: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:31:22,656: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:31:22,656: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:31:22,657: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e2e10deb-acf7-4b9a-806c-dc8a4865d977
[2017-06-29 11:31:22,657: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e2e10deb-acf7-4b9a-806c-dc8a4865d977
[2017-06-29 11:31:22,669: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e2e10deb-acf7-4b9a-806c-dc8a4865d977
[2017-06-29 11:31:22,669: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e2e10deb-acf7-4b9a-806c-dc8a4865d977
[2017-06-29 11:31:22,671: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-06-29 11:31:22,694: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bd5f3042-7be8-4412-9027-228adc519afa
[2017-06-29 11:31:22,694: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bd5f3042-7be8-4412-9027-228adc519afa
[2017-06-29 11:31:27,400: INFO/job_scheduler] run_next: 0 runnable jobs of (32 pending, 4 tasked)
[2017-06-29 11:31:27,433: INFO/job_scheduler] run_next: 0 runnable jobs of (32 pending, 4 tasked)
[2017-06-29 11:31:29,813: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:31:29,813: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:31:29,814: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:31:29,814: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:31:29,814: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bd5f3042-7be8-4412-9027-228adc519afa
[2017-06-29 11:31:29,814: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bd5f3042-7be8-4412-9027-228adc519afa
[2017-06-29 11:31:29,862: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bd5f3042-7be8-4412-9027-228adc519afa
[2017-06-29 11:31:29,862: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bd5f3042-7be8-4412-9027-228adc519afa
[2017-06-29 11:31:29,863: INFO/job_scheduler] Job 44 finished 3 steps successfully
[2017-06-29 11:31:29,918: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 140668098646416
[2017-06-29 11:31:29,925: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-06-29 11:31:29,939: INFO/job] Job 44: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-29 11:31:29,939: INFO/job] Job 44: not_deleted=True
[2017-06-29 11:31:29,939: INFO/job_scheduler] Job 44: complete_job: Updating cache
[2017-06-29 11:31:29,939: INFO/job_scheduler] Job 44: locked_item 140668098646416 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-29 11:31:29,942: INFO/job_scheduler] Job 44 completing (errored=False, cancelled=False)
[2017-06-29 11:31:29,988: INFO/job_scheduler] run_next: 1 runnable jobs of (32 pending, 3 tasked)
[2017-06-29 11:31:30,020: INFO/job_scheduler] Job 46: RunJobThread.run
[2017-06-29 11:31:30,021: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-06-29 11:31:30,061: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3855d549-c041-4460-9482-94f2b1f6548d
[2017-06-29 11:31:30,061: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3855d549-c041-4460-9482-94f2b1f6548d
[2017-06-29 11:31:37,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:31:37,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:31:37,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:31:37,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:31:37,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3855d549-c041-4460-9482-94f2b1f6548d
[2017-06-29 11:31:37,198: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3855d549-c041-4460-9482-94f2b1f6548d
[2017-06-29 11:31:37,240: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3855d549-c041-4460-9482-94f2b1f6548d
[2017-06-29 11:31:37,240: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3855d549-c041-4460-9482-94f2b1f6548d
[2017-06-29 11:31:37,241: INFO/job_scheduler] Job 46 finished 1 steps successfully
[2017-06-29 11:31:37,325: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 140668098646416
[2017-06-29 11:31:37,354: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm5.lotus.hpdd.lab.intel.com at severity 30
[2017-06-29 11:31:37,361: INFO/job] Job 46: StateChangeJob complete, setting state started on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-29 11:31:37,361: INFO/job] Job 46: not_deleted=True
[2017-06-29 11:31:37,361: INFO/job_scheduler] Job 46: complete_job: Updating cache
[2017-06-29 11:31:37,361: INFO/job_scheduler] Job 46: locked_item 140668098646416 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-29 11:31:37,364: INFO/job_scheduler] Job 46 completing (errored=False, cancelled=False)
[2017-06-29 11:31:37,398: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-2
[2017-06-29 11:31:37,399: INFO/job_scheduler] _notify: Updating .mcast_port of item 2 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 40759
[2017-06-29 11:31:37,434: INFO/job_scheduler] _notify: Set mcast_port=40759 on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-2) and saved
[2017-06-29 11:31:37,436: INFO/job_scheduler] _notify: Updating .network_interfaces of item 2 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-06-29 11:31:37,456: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-2) and saved
[2017-06-29 11:31:37,511: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 2 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-06-29 11:31:37,528: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-2) and saved
[2017-06-29 11:31:37,547: INFO/job_scheduler] run_next: 1 runnable jobs of (31 pending, 3 tasked)
[2017-06-29 11:31:37,639: INFO/job_scheduler] Job 47: RunJobThread.run
[2017-06-29 11:31:37,641: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-06-29 11:31:37,709: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 742cca69-56e1-44a4-b644-65e07959a1ff
[2017-06-29 11:31:37,709: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 742cca69-56e1-44a4-b644-65e07959a1ff
[2017-06-29 11:31:37,725: INFO/job_scheduler] run_next: 0 runnable jobs of (30 pending, 4 tasked)
[2017-06-29 11:31:37,744: INFO/job_scheduler] run_next: 0 runnable jobs of (30 pending, 4 tasked)
[2017-06-29 11:31:47,450: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 2 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from True to False
[2017-06-29 11:31:47,455: INFO/job_scheduler] _notify: Set corosync_reported_up=False on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-2) and saved
[2017-06-29 11:31:47,466: INFO/job_scheduler] run_next: 0 runnable jobs of (30 pending, 4 tasked)
[2017-06-29 11:31:57,510: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 2 (lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-06-29 11:31:57,516: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-2) and saved
[2017-06-29 11:31:57,523: INFO/job_scheduler] run_next: 0 runnable jobs of (30 pending, 4 tasked)
[2017-06-29 11:32:05,114: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:32:05,114: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:32:05,114: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:32:05,114: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:32:05,114: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 742cca69-56e1-44a4-b644-65e07959a1ff
[2017-06-29 11:32:05,114: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 742cca69-56e1-44a4-b644-65e07959a1ff
[2017-06-29 11:32:05,158: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 742cca69-56e1-44a4-b644-65e07959a1ff
[2017-06-29 11:32:05,158: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 742cca69-56e1-44a4-b644-65e07959a1ff
[2017-06-29 11:32:05,162: INFO/job_scheduler] Job 47 finished 1 steps successfully
[2017-06-29 11:32:05,850: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 140668098672336
[2017-06-29 11:32:05,855: INFO/job] Job 47: StateChangeJob complete, setting state managed on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 11:32:05,855: INFO/job] Job 47: not_deleted=True
[2017-06-29 11:32:05,855: INFO/job_scheduler] Job 47: complete_job: Updating cache
[2017-06-29 11:32:05,855: INFO/job_scheduler] Job 47: locked_item 140668098672336 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-29 11:32:05,858: INFO/job_scheduler] Job 47 completing (errored=False, cancelled=False)
[2017-06-29 11:32:05,967: INFO/job_scheduler] Replaying 4 buffered notifications for ManagedHost-2
[2017-06-29 11:32:06,014: INFO/job_scheduler] _notify: Updating .immutable_state of item 2 (lotus-32vm5.lotus.hpdd.lab.intel.com) from True to False
[2017-06-29 11:32:06,034: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-2) and saved
[2017-06-29 11:32:06,051: INFO/job_scheduler] _notify: Updating .boot_time of item 2 (lotus-32vm5.lotus.hpdd.lab.intel.com) from 2017-06-29 03:53:12+00:00 to 2017-06-29 04:28:01+00:00
[2017-06-29 11:32:06,058: INFO/job_scheduler] _notify: Set boot_time=2017-06-29 04:28:01+00:00 on lotus-32vm5.lotus.hpdd.lab.intel.com (ManagedHost-2) and saved
[2017-06-29 11:32:06,103: INFO/job_scheduler] run_next: 0 runnable jobs of (30 pending, 3 tasked)
[2017-06-29 11:33:33,168: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 11:33:33,168: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 11:33:33,169: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 11:33:33,169: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 11:33:33,169: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a6d6ec84-7171-46b9-a596-486039ae6a61
[2017-06-29 11:33:33,169: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a6d6ec84-7171-46b9-a596-486039ae6a61
[2017-06-29 11:33:33,183: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a6d6ec84-7171-46b9-a596-486039ae6a61
[2017-06-29 11:33:33,183: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a6d6ec84-7171-46b9-a596-486039ae6a61
[2017-06-29 11:33:33,184: ERROR/job_scheduler] Job 48 step 2 encountered an agent error: Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/chroma_agent/device_plugins/action_runner.py", line 164, in run

  File "/usr/lib/python2.7/site-packages/chroma_agent/plugin_manager.py", line 305, in run

  File "/usr/lib/python2.7/site-packages/chroma_agent/action_plugins/agent_updates.py", line 113, in install_packages

  File "/usr/lib/python2.7/site-packages/chroma_agent/lib/yum_utils.py", line 71, in yum_util

CommandExecutionError: Error (1) running 'yum install -y --enablerepo=iml-agent zfs zfs-dkms kernel-devel-[0-9]*_lustre kernel-[0-9]*_lustre lustre-osd-zfs-mount kmod-lustre-osd-ldiskfs lustre-dkms lustre chroma-agent-management chroma-agent-3.1.1.0-4946.134.gd32299f.jenkins_manager_for_lustre_444.el7 kmod-lustre-2.10.50 lustre-osd-ldiskfs-mount-2.10.50 kmod-lustre-2.10.50 libnvpair1-0.6.5.9 libuutil1-0.6.5.9 libzfs2-0.6.5.9 libzpool2-0.6.5.9 spl-0.6.5.9 zfs-kmod-0.6.5.9 spl-dkms-0.6.5.9': 'Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * epel: archive.linux.duke.edu
Package kernel-devel-3.10.0-514.21.1.el7_lustre.x86_64 already installed and latest version
Package kernel-3.10.0-514.21.1.el7_lustre.x86_64 already installed and latest version
Package chroma-agent-management-3.1.1.0-4946.134.gd32299f.jenkins_manager_for_lustre_444.el7.noarch already installed and latest version
Package chroma-agent-3.1.1.0-4946.134.gd32299f.jenkins_manager_for_lustre_444.el7.noarch already installed and latest version
Package libnvpair1-0.6.5.9-1.el7.x86_64 already installed and latest version
Package libuutil1-0.6.5.9-1.el7.x86_64 already installed and latest version
Package libzfs2-0.6.5.9-1.el7.x86_64 already installed and latest version
Package libzpool2-0.6.5.9-1.el7.x86_64 already installed and latest version
Package spl-0.6.5.9-1.el7.x86_64 already installed and latest version
No package zfs-kmod-0.6.5.9 available.
Package spl-dkms-0.6.5.9-1.el7.noarch already installed and latest version
Resolving Dependencies
--> Running transaction check
---> Package kmod-lustre.x86_64 0:2.9.58_86_g2383a62-1.el7 will be updated
---> Package kmod-lustre.x86_64 0:2.10.50-1.el7 will be an update
---> Package kmod-lustre-osd-ldiskfs.x86_64 0:2.10.50-1.el7 will be installed
---> Package lustre.x86_64 0:2.10.50-1.el7 will be installed
---> Package lustre-dkms.noarch 0:2.10.50-1.el7 will be installed
---> Package lustre-osd-ldiskfs-mount.x86_64 0:2.10.50-1.el7 will be installed
---> Package lustre-osd-zfs-mount.x86_64 0:2.10.50-1.el7 will be installed
---> Package zfs.x86_64 0:0.6.5.9-1.el7 will be installed
---> Package zfs-dkms.noarch 0:0.6.5.9-1.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package                       Arch        Version            Repository   Size
================================================================================
Installing:
 kmod-lustre-osd-ldiskfs       x86_64      2.10.50-1.el7      lustre      454 k
 lustre                        x86_64      2.10.50-1.el7      lustre      738 k
 lustre-dkms                   noarch      2.10.50-1.el7      lustre       13 M
 lustre-osd-ldiskfs-mount      x86_64      2.10.50-1.el7      lustre       15 k
 lustre-osd-zfs-mount          x86_64      2.10.50-1.el7      lustre       10 k
 zfs                           x86_64      0.6.5.9-1.el7      lustre      331 k
 zfs-dkms                      noarch      0.6.5.9-1.el7      lustre      1.9 M
Updating:
 kmod-lustre                   x86_64      2.10.50-1.el7      lustre      3.7 M

Transaction Summary
================================================================================
Install  7 Packages
Upgrade  1 Package

Total size: 20 M
Total download size: 13 M
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
' 'Bad id for repo: build.whamcloud.com_job_lustre-master_lastSuccessfulBuild_arch=x86_64,build_type=,distro=el7,ib_stack=inkernel_artifact_artifacts_, byte = = 62
https://build.whamcloud.com/job/lustre-master/lastSuccessfulBuild/arch=x86_64%2Cbuild_type=server%2Cdistro=el7%2Cib_stack=inkernel/artifact/artifacts/RPMS/noarch/lustre-dkms-2.10.50-1.el7.noarch.rpm: [Errno 14] curl#18 - "transfer closed with outstanding read data remaining"
Trying other mirror.


Error downloading packages:
  lustre-dkms-2.10.50-1.el7.noarch: [Errno 256] No more mirrors to try.

'

[2017-06-29 11:33:33,434: INFO/job_scheduler] Job 48: complete_job: Updating cache
[2017-06-29 11:33:33,434: INFO/job_scheduler] Job 48: locked_item 140667427276176 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-29 11:33:33,438: INFO/job_scheduler] Job 48 completing (errored=True, cancelled=False)
[2017-06-29 11:33:33,456: INFO/job_scheduler] run_next: 4 runnable jobs of (30 pending, 2 tasked)
[2017-06-29 11:33:33,458: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com in state unconfigured, not one of ['packages_installed'] (preferred packages_installed)
[2017-06-29 11:33:33,459: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com in state unconfigured, not one of ['packages_installed'] (preferred packages_installed)
[2017-06-29 11:33:33,460: WARNING/job_scheduler] Job 49: cancelling because of failed dependency
[2017-06-29 11:33:33,470: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com in state unconfigured, not one of ['packages_installed'] (preferred packages_installed)
[2017-06-29 11:33:33,471: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com in state unconfigured, not one of ['packages_installed'] (preferred packages_installed)
[2017-06-29 11:33:33,472: WARNING/job_scheduler] Job 50: cancelling because of failed dependency
[2017-06-29 11:33:33,473: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com in state unconfigured, not one of ['packages_installed'] (preferred packages_installed)
[2017-06-29 11:33:33,474: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com in state unconfigured, not one of ['packages_installed'] (preferred packages_installed)
[2017-06-29 11:33:33,476: WARNING/job_scheduler] Job 51: cancelling because of failed dependency
[2017-06-29 11:33:33,477: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com in state unconfigured, not one of ['packages_installed'] (preferred packages_installed)
[2017-06-29 11:33:33,478: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com in state unconfigured, not one of ['packages_installed'] (preferred packages_installed)
[2017-06-29 11:33:33,479: WARNING/job_scheduler] Job 52: cancelling because of failed dependency
[2017-06-29 11:33:33,480: INFO/job_scheduler] Job 49 completing (errored=False, cancelled=True)
[2017-06-29 11:33:33,485: INFO/job_scheduler] Job 50 completing (errored=False, cancelled=True)
[2017-06-29 11:33:33,491: INFO/job_scheduler] Job 51 completing (errored=False, cancelled=True)
[2017-06-29 11:33:33,497: INFO/job_scheduler] Job 52 completing (errored=False, cancelled=True)
[2017-06-29 11:33:33,503: INFO/job_scheduler] run_next: 2 runnable jobs of (26 pending, 2 tasked)
[2017-06-29 11:33:33,506: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration in state unconfigured, not one of ['lnet_unloaded'] (preferred lnet_unloaded)
[2017-06-29 11:33:33,510: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration in state unconfigured, not one of ['lnet_unloaded'] (preferred lnet_unloaded)
[2017-06-29 11:33:33,510: WARNING/job_scheduler] Job 53: cancelling because of failed dependency
[2017-06-29 11:33:33,518: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration in state unconfigured, not one of ['stopped'] (preferred stopped)
[2017-06-29 11:33:33,525: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration in state unconfigured, not one of ['stopped'] (preferred stopped)
[2017-06-29 11:33:33,526: WARNING/job_scheduler] Job 54: cancelling because of failed dependency
[2017-06-29 11:33:33,526: INFO/job_scheduler] Job 53 completing (errored=False, cancelled=True)
[2017-06-29 11:33:33,545: INFO/job_scheduler] Job 54 completing (errored=False, cancelled=True)
[2017-06-29 11:33:33,554: INFO/job_scheduler] run_next: 2 runnable jobs of (24 pending, 2 tasked)
[2017-06-29 11:33:33,558: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com in state unconfigured, not one of ['packages_installed'] (preferred packages_installed)
[2017-06-29 11:33:33,569: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com in state unconfigured, not one of ['packages_installed'] (preferred packages_installed)
[2017-06-29 11:33:33,574: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration in state unconfigured, not one of ['started'] (preferred started)
[2017-06-29 11:33:33,577: WARNING/job_scheduler] Job 55: cancelling because of failed dependency
[2017-06-29 11:33:33,583: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration in state unconfigured, not one of ['lnet_down'] (preferred lnet_down)
[2017-06-29 11:33:33,588: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration in state unconfigured, not one of ['lnet_down'] (preferred lnet_down)
[2017-06-29 11:33:33,588: WARNING/job_scheduler] Job 56: cancelling because of failed dependency
[2017-06-29 11:33:33,588: INFO/job_scheduler] Job 55 completing (errored=False, cancelled=True)
[2017-06-29 11:33:33,603: INFO/job_scheduler] Job 56 completing (errored=False, cancelled=True)
[2017-06-29 11:33:33,614: INFO/job_scheduler] run_next: 1 runnable jobs of (22 pending, 2 tasked)
[2017-06-29 11:33:33,620: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration in state unconfigured, not one of ['started'] (preferred started)
[2017-06-29 11:33:33,626: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration in state unconfigured, not one of ['started'] (preferred started)
[2017-06-29 11:33:33,628: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration in state unconfigured, not one of ['stopped'] (preferred stopped)
[2017-06-29 11:33:33,628: WARNING/job_scheduler] Job 57: cancelling because of failed dependency
[2017-06-29 11:33:33,628: INFO/job_scheduler] Job 57 completing (errored=False, cancelled=True)
[2017-06-29 11:33:33,640: INFO/job_scheduler] run_next: 1 runnable jobs of (21 pending, 2 tasked)
[2017-06-29 11:33:33,647: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration in state unconfigured, not one of ['lnet_up'] (preferred lnet_up)
[2017-06-29 11:33:33,654: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration in state unconfigured, not one of ['lnet_up'] (preferred lnet_up)
[2017-06-29 11:33:33,668: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration in state unconfigured, not one of ['started'] (preferred started)
[2017-06-29 11:33:33,674: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration in state unconfigured, not one of ['configured'] (preferred configured)
[2017-06-29 11:33:33,676: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration in state unconfigured, not one of ['configured'] (preferred configured)
[2017-06-29 11:33:33,680: WARNING/job] DependOn not satisfied: lotus-32vm6.lotus.hpdd.lab.intel.com in state unconfigured, not one of ['packages_installed'] (preferred packages_installed)
[2017-06-29 11:33:33,685: WARNING/job_scheduler] Job 58: cancelling because of failed dependency
[2017-06-29 11:33:33,685: INFO/job_scheduler] Job 58 completing (errored=False, cancelled=True)
[2017-06-29 11:33:33,758: INFO/job_scheduler] run_next: 0 runnable jobs of (20 pending, 2 tasked)
[2017-06-29 11:38:48,654: INFO/job_scheduler] run_next: 0 runnable jobs of (20 pending, 2 tasked)
[2017-06-29 11:38:48,686: INFO/job_scheduler] run_next: 0 runnable jobs of (20 pending, 2 tasked)
[2017-06-29 11:39:29,645: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 11:39:29,645: INFO/job_scheduler]   Job 70 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-29 11:39:29,645: INFO/job_scheduler]   Job 71 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:39:29,646: INFO/job_scheduler]   Job 72 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:39:29,646: INFO/job_scheduler]   Job 73 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:39:29,646: INFO/job_scheduler]   Job 74 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:39:29,646: INFO/job_scheduler]   Job 78 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:39:29,646: INFO/job_scheduler]   Job 80 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-29 11:39:29,646: INFO/job_scheduler] run_next: 0 runnable jobs of (20 pending, 2 tasked)
[2017-06-29 11:40:13,317: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 11:40:13,317: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-29 11:40:13,335: INFO/command_plan] add_jobs: created Job 82 (Trigger plugin poll for linux plugins)
[2017-06-29 11:40:13,344: INFO/job_scheduler] run_next: 1 runnable jobs of (21 pending, 2 tasked)
[2017-06-29 11:40:13,403: INFO/job_scheduler] Job 82: RunJobThread.run
[2017-06-29 11:40:13,441: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 11:40:13,481: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 11:40:13,489: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 264d7800-5f4d-40d8-a534-84fe03d46a02
[2017-06-29 11:40:13,489: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 264d7800-5f4d-40d8-a534-84fe03d46a02
[2017-06-29 11:40:13,522: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 986e7db5-e1d5-4534-869c-cbd951c1fcae
[2017-06-29 11:40:13,522: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 986e7db5-e1d5-4534-869c-cbd951c1fcae
[2017-06-29 11:40:13,523: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 11:40:13,561: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8cac49fa-13f6-4b58-8bd2-25d6d28463b6
[2017-06-29 11:40:13,561: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8cac49fa-13f6-4b58-8bd2-25d6d28463b6
[2017-06-29 11:40:13,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:40:13,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:40:13,590: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:40:13,590: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:40:13,590: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 264d7800-5f4d-40d8-a534-84fe03d46a02
[2017-06-29 11:40:13,590: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 264d7800-5f4d-40d8-a534-84fe03d46a02
[2017-06-29 11:40:13,618: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 264d7800-5f4d-40d8-a534-84fe03d46a02
[2017-06-29 11:40:13,618: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 264d7800-5f4d-40d8-a534-84fe03d46a02
[2017-06-29 11:40:13,618: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 11:40:13,626: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 11:40:13,626: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 11:40:13,627: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 11:40:13,627: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 11:40:13,627: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 986e7db5-e1d5-4534-869c-cbd951c1fcae
[2017-06-29 11:40:13,627: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 986e7db5-e1d5-4534-869c-cbd951c1fcae
[2017-06-29 11:40:13,648: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:40:13,648: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:40:13,648: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:40:13,648: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:40:13,648: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8cac49fa-13f6-4b58-8bd2-25d6d28463b6
[2017-06-29 11:40:13,648: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8cac49fa-13f6-4b58-8bd2-25d6d28463b6
[2017-06-29 11:40:13,655: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 986e7db5-e1d5-4534-869c-cbd951c1fcae
[2017-06-29 11:40:13,655: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 986e7db5-e1d5-4534-869c-cbd951c1fcae
[2017-06-29 11:40:13,656: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 11:40:13,688: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8cac49fa-13f6-4b58-8bd2-25d6d28463b6
[2017-06-29 11:40:13,688: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8cac49fa-13f6-4b58-8bd2-25d6d28463b6
[2017-06-29 11:40:13,688: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 11:40:13,688: INFO/job_scheduler] Job 82 finished 1 steps successfully
[2017-06-29 11:40:13,717: INFO/job_scheduler] Job 82: complete_job: Updating cache
[2017-06-29 11:40:13,717: INFO/job_scheduler] Job 82 completing (errored=False, cancelled=False)
[2017-06-29 11:40:13,812: INFO/job_scheduler] Replaying 3 buffered notifications for ManagedHost-3
[2017-06-29 11:40:13,846: INFO/job_scheduler] _notify: Updating .immutable_state of item 3 (lotus-32vm6.lotus.hpdd.lab.intel.com) from True to False
[2017-06-29 11:40:13,866: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm6.lotus.hpdd.lab.intel.com (ManagedHost-3) and saved
[2017-06-29 11:40:13,931: INFO/job_scheduler] run_next: 0 runnable jobs of (20 pending, 2 tasked)
[2017-06-29 11:41:21,744: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 11:41:21,744: INFO/job_scheduler]   Job 59 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-29 11:41:21,744: INFO/job_scheduler]   Job 60 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:41:21,745: INFO/job_scheduler]   Job 61 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:41:21,745: INFO/job_scheduler]   Job 62 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:41:21,745: INFO/job_scheduler]   Job 63 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:41:21,745: INFO/job_scheduler]   Job 67 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:41:21,745: INFO/job_scheduler]   Job 69 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-29 11:41:21,745: INFO/job_scheduler] run_next: 0 runnable jobs of (20 pending, 2 tasked)
[2017-06-29 11:41:31,284: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 11:41:31,284: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-29 11:41:31,459: INFO/command_plan] add_jobs: created Job 83 (Trigger plugin poll for linux plugins)
[2017-06-29 11:41:31,660: INFO/job_scheduler] run_next: 1 runnable jobs of (21 pending, 2 tasked)
[2017-06-29 11:41:31,679: INFO/job_scheduler] Job 83: RunJobThread.run
[2017-06-29 11:41:31,694: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 11:41:31,722: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c6e0e064-d507-4ca1-9843-35ea70973942
[2017-06-29 11:41:31,722: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c6e0e064-d507-4ca1-9843-35ea70973942
[2017-06-29 11:41:31,723: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 11:41:31,745: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b08b3abe-a96b-491e-8ede-959732104220
[2017-06-29 11:41:31,745: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b08b3abe-a96b-491e-8ede-959732104220
[2017-06-29 11:41:31,751: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 11:41:31,795: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 06de5501-f3ff-400d-8cdb-923c50770f6d
[2017-06-29 11:41:31,795: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 06de5501-f3ff-400d-8cdb-923c50770f6d
[2017-06-29 11:41:31,827: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 11:41:31,827: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 11:41:31,827: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 11:41:31,827: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 11:41:31,827: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b08b3abe-a96b-491e-8ede-959732104220
[2017-06-29 11:41:31,827: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b08b3abe-a96b-491e-8ede-959732104220
[2017-06-29 11:41:31,830: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b08b3abe-a96b-491e-8ede-959732104220
[2017-06-29 11:41:31,830: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b08b3abe-a96b-491e-8ede-959732104220
[2017-06-29 11:41:31,831: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 11:41:31,833: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:41:31,833: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:41:31,833: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:41:31,833: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 11:41:31,833: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c6e0e064-d507-4ca1-9843-35ea70973942
[2017-06-29 11:41:31,833: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c6e0e064-d507-4ca1-9843-35ea70973942
[2017-06-29 11:41:31,850: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c6e0e064-d507-4ca1-9843-35ea70973942
[2017-06-29 11:41:31,850: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c6e0e064-d507-4ca1-9843-35ea70973942
[2017-06-29 11:41:31,850: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 11:41:31,903: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:41:31,903: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:41:31,904: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:41:31,904: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:41:31,904: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 06de5501-f3ff-400d-8cdb-923c50770f6d
[2017-06-29 11:41:31,904: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 06de5501-f3ff-400d-8cdb-923c50770f6d
[2017-06-29 11:41:31,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 06de5501-f3ff-400d-8cdb-923c50770f6d
[2017-06-29 11:41:31,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 06de5501-f3ff-400d-8cdb-923c50770f6d
[2017-06-29 11:41:31,923: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 11:41:31,924: INFO/job_scheduler] Job 83 finished 1 steps successfully
[2017-06-29 11:41:32,002: INFO/job_scheduler] Job 83: complete_job: Updating cache
[2017-06-29 11:41:32,002: INFO/job_scheduler] Job 83 completing (errored=False, cancelled=False)
[2017-06-29 11:41:32,115: INFO/job_scheduler] run_next: 0 runnable jobs of (20 pending, 2 tasked)
[2017-06-29 11:42:13,792: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:42:13,792: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:42:13,792: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:42:13,792: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:42:13,792: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bcf1bcc8-a214-4b6a-a8f1-38fd97ec5084
[2017-06-29 11:42:13,792: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bcf1bcc8-a214-4b6a-a8f1-38fd97ec5084
[2017-06-29 11:42:13,808: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bcf1bcc8-a214-4b6a-a8f1-38fd97ec5084
[2017-06-29 11:42:13,808: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bcf1bcc8-a214-4b6a-a8f1-38fd97ec5084
[2017-06-29 11:42:13,856: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com kernel_status {}
[2017-06-29 11:42:13,887: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 84acb5e3-d461-44f9-afbd-954d0d32a11f
[2017-06-29 11:42:13,887: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 84acb5e3-d461-44f9-afbd-954d0d32a11f
[2017-06-29 11:42:15,953: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:42:15,953: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:42:15,953: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:42:15,953: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:42:15,954: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 84acb5e3-d461-44f9-afbd-954d0d32a11f
[2017-06-29 11:42:15,954: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 84acb5e3-d461-44f9-afbd-954d0d32a11f
[2017-06-29 11:42:15,955: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 84acb5e3-d461-44f9-afbd-954d0d32a11f
[2017-06-29 11:42:15,955: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 84acb5e3-d461-44f9-afbd-954d0d32a11f
[2017-06-29 11:42:15,955: INFO/job] Job 70 RebootIfNeededStep: Reboot of lotus-32vm8.lotus.hpdd.lab.intel.com required to switch from running kernel kernel-3.10.0-514.21.1.el7_lustre.x86_64 to required kernel-3.10.0-514.2.2.el7_lustre.x86_64
[2017-06-29 11:42:15,955: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com reboot_server {}
[2017-06-29 11:42:15,979: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 155eb4b0-fe91-480e-89ab-6b5911cb385f
[2017-06-29 11:42:15,979: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 155eb4b0-fe91-480e-89ab-6b5911cb385f
[2017-06-29 11:42:16,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:42:16,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:42:16,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:42:16,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:42:16,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 155eb4b0-fe91-480e-89ab-6b5911cb385f
[2017-06-29 11:42:16,084: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 155eb4b0-fe91-480e-89ab-6b5911cb385f
[2017-06-29 11:42:16,096: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 155eb4b0-fe91-480e-89ab-6b5911cb385f
[2017-06-29 11:42:16,096: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 155eb4b0-fe91-480e-89ab-6b5911cb385f
[2017-06-29 11:42:16,097: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm8.lotus.hpdd.lab.intel.com (old 6a1f5417-eba9-4141-9fdf-bf5f161307ce)
[2017-06-29 11:42:16,097: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm8.lotus.hpdd.lab.intel.com (old 6a1f5417-eba9-4141-9fdf-bf5f161307ce)
[2017-06-29 11:43:16,319: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:43:16,319: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/6a1f5417-eba9-4141-9fdf-bf5f161307ce
[2017-06-29 11:43:16,319: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 6a1f5417-eba9-4141-9fdf-bf5f161307ce because session terminated
[2017-06-29 11:43:16,319: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 6a1f5417-eba9-4141-9fdf-bf5f161307ce because session terminated
[2017-06-29 11:43:37,969: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 11:43:37,969: INFO/job_scheduler]   Job 70 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-29 11:43:37,969: INFO/job_scheduler]   Job 71 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:43:37,970: INFO/job_scheduler]   Job 72 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:43:37,970: INFO/job_scheduler]   Job 73 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:43:37,970: INFO/job_scheduler]   Job 74 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:43:37,970: INFO/job_scheduler]   Job 78 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:43:37,970: INFO/job_scheduler]   Job 80 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-29 11:43:37,970: INFO/job_scheduler] run_next: 0 runnable jobs of (20 pending, 2 tasked)
[2017-06-29 11:43:38,029: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:43:38,029: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:43:38,217: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm8.lotus.hpdd.lab.intel.com new 5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:43:38,217: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm8.lotus.hpdd.lab.intel.com new 5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:43:38,218: INFO/job_scheduler] Job 70 finished 4 steps successfully
[2017-06-29 11:43:38,246: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 140668098699216
[2017-06-29 11:43:38,251: INFO/job] Job 70: StateChangeJob complete, setting state packages_installed on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:43:38,251: INFO/job] Job 70: not_deleted=True
[2017-06-29 11:43:38,251: INFO/job_scheduler] Job 70: complete_job: Updating cache
[2017-06-29 11:43:38,251: INFO/job_scheduler] Job 70: locked_item 140667427279504 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-29 11:43:38,254: INFO/job_scheduler] Job 70 completing (errored=False, cancelled=False)
[2017-06-29 11:43:38,275: INFO/job_scheduler] run_next: 4 runnable jobs of (20 pending, 1 tasked)
[2017-06-29 11:43:38,315: INFO/job_scheduler] Job 71: RunJobThread.run
[2017-06-29 11:43:38,315: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-06-29 11:43:38,323: INFO/job_scheduler] Job 72: open DB connection during completion
[2017-06-29 11:43:38,335: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dcfc678a-7dc9-4be8-9f67-fe652f43e930
[2017-06-29 11:43:38,335: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dcfc678a-7dc9-4be8-9f67-fe652f43e930
[2017-06-29 11:43:38,337: INFO/job_scheduler] Job 73: RunJobThread.run
[2017-06-29 11:43:38,347: INFO/job_scheduler] Job 74: RunJobThread.run
[2017-06-29 11:43:38,348: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-06-29 11:43:38,375: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2ca563a0-6610-4e1b-a9d5-71e39198accd
[2017-06-29 11:43:38,375: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2ca563a0-6610-4e1b-a9d5-71e39198accd
[2017-06-29 11:43:38,387: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-06-29 11:43:38,397: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 140668098710928
[2017-06-29 11:43:38,408: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-06-29 11:43:38,426: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8713a63a-72a7-47df-be87-281e0edd55a1
[2017-06-29 11:43:38,426: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8713a63a-72a7-47df-be87-281e0edd55a1
[2017-06-29 11:43:38,427: INFO/job] Job 72: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-29 11:43:38,427: INFO/job] Job 72: not_deleted=True
[2017-06-29 11:43:38,427: INFO/job_scheduler] Job 72: complete_job: Updating cache
[2017-06-29 11:43:38,427: INFO/job_scheduler] Job 72: locked_item 140668098710928 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-29 11:43:38,430: INFO/job_scheduler] Job 72 completing (errored=False, cancelled=False)
[2017-06-29 11:43:38,459: INFO/job_scheduler] run_next: 1 runnable jobs of (16 pending, 4 tasked)
[2017-06-29 11:43:38,477: INFO/job_scheduler] Job 76: RunJobThread.run
[2017-06-29 11:43:38,477: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com load_lnet {}
[2017-06-29 11:43:38,520: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1399eeb5-52fc-4146-9265-a9de4a61599e
[2017-06-29 11:43:38,520: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1399eeb5-52fc-4146-9265-a9de4a61599e
[2017-06-29 11:43:38,686: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:43:38,686: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:43:38,686: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:43:38,686: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:43:38,686: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1399eeb5-52fc-4146-9265-a9de4a61599e
[2017-06-29 11:43:38,686: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1399eeb5-52fc-4146-9265-a9de4a61599e
[2017-06-29 11:43:38,697: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1399eeb5-52fc-4146-9265-a9de4a61599e
[2017-06-29 11:43:38,697: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1399eeb5-52fc-4146-9265-a9de4a61599e
[2017-06-29 11:43:38,698: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-29 11:43:38,731: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 888831f6-89e8-4b3e-b0d6-813a786e4217
[2017-06-29 11:43:38,731: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 888831f6-89e8-4b3e-b0d6-813a786e4217
[2017-06-29 11:43:39,915: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:43:39,915: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:43:39,915: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:43:39,915: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:43:39,915: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 888831f6-89e8-4b3e-b0d6-813a786e4217
[2017-06-29 11:43:39,915: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 888831f6-89e8-4b3e-b0d6-813a786e4217
[2017-06-29 11:43:39,957: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 888831f6-89e8-4b3e-b0d6-813a786e4217
[2017-06-29 11:43:39,957: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 888831f6-89e8-4b3e-b0d6-813a786e4217
[2017-06-29 11:43:40,022: INFO/job_scheduler] Job 76 finished 2 steps successfully
[2017-06-29 11:43:40,079: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 140668098710928
[2017-06-29 11:43:40,087: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-06-29 11:43:40,092: INFO/job] Job 76: StateChangeJob complete, setting state lnet_down on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-29 11:43:40,093: INFO/job] Job 76: not_deleted=True
[2017-06-29 11:43:40,095: INFO/job_scheduler] Job 76: complete_job: Updating cache
[2017-06-29 11:43:40,096: INFO/job_scheduler] Job 76: locked_item 140668098710928 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-29 11:43:40,105: INFO/job_scheduler] Job 76 completing (errored=False, cancelled=False)
[2017-06-29 11:43:40,180: INFO/job_scheduler] run_next: 1 runnable jobs of (15 pending, 4 tasked)
[2017-06-29 11:43:40,200: INFO/job_scheduler] Job 77: RunJobThread.run
[2017-06-29 11:43:40,200: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_lnet {}
[2017-06-29 11:43:40,224: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cf76324e-cc10-473d-8233-cf30ae7a8670
[2017-06-29 11:43:40,224: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cf76324e-cc10-473d-8233-cf30ae7a8670
[2017-06-29 11:43:41,326: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:43:41,326: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:43:41,326: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:43:41,326: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:43:41,326: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cf76324e-cc10-473d-8233-cf30ae7a8670
[2017-06-29 11:43:41,326: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cf76324e-cc10-473d-8233-cf30ae7a8670
[2017-06-29 11:43:41,349: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cf76324e-cc10-473d-8233-cf30ae7a8670
[2017-06-29 11:43:41,349: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cf76324e-cc10-473d-8233-cf30ae7a8670
[2017-06-29 11:43:41,350: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-29 11:43:41,377: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 805d1f93-7a57-41ef-8cbf-ae7415fc397e
[2017-06-29 11:43:41,377: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 805d1f93-7a57-41ef-8cbf-ae7415fc397e
[2017-06-29 11:43:41,454: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:43:41,454: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:43:41,454: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:43:41,454: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:43:41,454: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 805d1f93-7a57-41ef-8cbf-ae7415fc397e
[2017-06-29 11:43:41,454: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 805d1f93-7a57-41ef-8cbf-ae7415fc397e
[2017-06-29 11:43:41,498: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 805d1f93-7a57-41ef-8cbf-ae7415fc397e
[2017-06-29 11:43:41,498: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 805d1f93-7a57-41ef-8cbf-ae7415fc397e
[2017-06-29 11:43:41,817: INFO/job_scheduler] Job 77 finished 2 steps successfully
[2017-06-29 11:43:41,881: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 140668098710928
[2017-06-29 11:43:41,887: INFO/job] Job 77: StateChangeJob complete, setting state lnet_up on lotus-32vm8.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-29 11:43:41,887: INFO/job] Job 77: not_deleted=True
[2017-06-29 11:43:41,888: INFO/job_scheduler] Job 77: complete_job: Updating cache
[2017-06-29 11:43:41,888: INFO/job_scheduler] Job 77: locked_item 140668098710928 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-29 11:43:41,892: INFO/job_scheduler] Job 77 completing (errored=False, cancelled=False)
[2017-06-29 11:43:42,034: INFO/job_scheduler] run_next: 0 runnable jobs of (14 pending, 4 tasked)
[2017-06-29 11:43:43,054: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 11:43:43,054: INFO/job_scheduler]   Job 71 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:43:43,054: INFO/job_scheduler]   Job 73 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:43:43,054: INFO/job_scheduler]   Job 74 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:43:43,054: INFO/job_scheduler]   Job 78 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:43:43,054: INFO/job_scheduler]   Job 80 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-29 11:43:43,055: INFO/job_scheduler] run_next: 0 runnable jobs of (14 pending, 4 tasked)
[2017-06-29 11:43:43,134: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 11:43:43,134: INFO/job_scheduler]   Job 71 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:43:43,134: INFO/job_scheduler]   Job 73 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:43:43,134: INFO/job_scheduler]   Job 74 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:43:43,134: INFO/job_scheduler]   Job 78 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:43:43,134: INFO/job_scheduler]   Job 80 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-29 11:43:43,135: INFO/job_scheduler] run_next: 0 runnable jobs of (14 pending, 4 tasked)
[2017-06-29 11:43:43,988: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:43:43,988: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:43:43,988: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:43:43,988: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:43:43,988: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8713a63a-72a7-47df-be87-281e0edd55a1
[2017-06-29 11:43:43,988: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8713a63a-72a7-47df-be87-281e0edd55a1
[2017-06-29 11:43:44,009: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8713a63a-72a7-47df-be87-281e0edd55a1
[2017-06-29 11:43:44,009: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8713a63a-72a7-47df-be87-281e0edd55a1
[2017-06-29 11:43:44,010: INFO/job_scheduler] Job 73 finished 1 steps successfully
[2017-06-29 11:43:44,071: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 140668098708304
[2017-06-29 11:43:44,074: INFO/job] Job 73: StateChangeJob complete, setting state configured on lotus-32vm8.lotus.hpdd.lab.intel.com NTP configuration
[2017-06-29 11:43:44,075: INFO/job] Job 73: not_deleted=True
[2017-06-29 11:43:44,075: INFO/job_scheduler] Job 73: complete_job: Updating cache
[2017-06-29 11:43:44,075: INFO/job_scheduler] Job 73: locked_item 140668098708304 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-06-29 11:43:44,078: INFO/job_scheduler] Job 73 completing (errored=False, cancelled=False)
[2017-06-29 11:43:44,109: INFO/job_scheduler] run_next: 0 runnable jobs of (14 pending, 3 tasked)
[2017-06-29 11:43:46,000: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:43:46,000: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:43:46,000: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:43:46,000: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:43:46,001: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dcfc678a-7dc9-4be8-9f67-fe652f43e930
[2017-06-29 11:43:46,001: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dcfc678a-7dc9-4be8-9f67-fe652f43e930
[2017-06-29 11:43:46,024: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dcfc678a-7dc9-4be8-9f67-fe652f43e930
[2017-06-29 11:43:46,024: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dcfc678a-7dc9-4be8-9f67-fe652f43e930
[2017-06-29 11:43:46,025: INFO/job_scheduler] Job 71 finished 1 steps successfully
[2017-06-29 11:43:46,214: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 140667427257936
[2017-06-29 11:43:46,217: INFO/job] Job 71: StateChangeJob complete, setting state configured on lotus-32vm8.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-06-29 11:43:46,217: INFO/job] Job 71: not_deleted=True
[2017-06-29 11:43:46,218: INFO/job_scheduler] Job 71: complete_job: Updating cache
[2017-06-29 11:43:46,218: INFO/job_scheduler] Job 71: locked_item 140667427257936 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-06-29 11:43:46,220: INFO/job_scheduler] Job 71 completing (errored=False, cancelled=False)
[2017-06-29 11:43:46,242: INFO/job_scheduler] run_next: 0 runnable jobs of (14 pending, 2 tasked)
[2017-06-29 11:43:50,457: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:43:50,457: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:43:50,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:43:50,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:43:50,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7dcd4dae-db7c-4baa-bcc7-f01e3b5146c3
[2017-06-29 11:43:50,458: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7dcd4dae-db7c-4baa-bcc7-f01e3b5146c3
[2017-06-29 11:43:50,493: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7dcd4dae-db7c-4baa-bcc7-f01e3b5146c3
[2017-06-29 11:43:50,493: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7dcd4dae-db7c-4baa-bcc7-f01e3b5146c3
[2017-06-29 11:43:50,582: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com kernel_status {}
[2017-06-29 11:43:50,615: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4c19d59d-2c0d-4ffd-8862-cba9f6bc0a92
[2017-06-29 11:43:50,615: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4c19d59d-2c0d-4ffd-8862-cba9f6bc0a92
[2017-06-29 11:43:53,714: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:43:53,714: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:43:53,715: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:43:53,715: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:43:53,715: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4c19d59d-2c0d-4ffd-8862-cba9f6bc0a92
[2017-06-29 11:43:53,715: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4c19d59d-2c0d-4ffd-8862-cba9f6bc0a92
[2017-06-29 11:43:53,736: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4c19d59d-2c0d-4ffd-8862-cba9f6bc0a92
[2017-06-29 11:43:53,736: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4c19d59d-2c0d-4ffd-8862-cba9f6bc0a92
[2017-06-29 11:43:53,737: INFO/job] Job 59 RebootIfNeededStep: Reboot of lotus-32vm7.lotus.hpdd.lab.intel.com required to switch from running kernel kernel-3.10.0-514.10.2.el7_lustre.x86_64 to required kernel-3.10.0-514.21.1.el7_lustre.x86_64
[2017-06-29 11:43:53,737: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com reboot_server {}
[2017-06-29 11:43:53,763: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f71f944e-dc55-4fc7-a6b9-51b7fcc4d53e
[2017-06-29 11:43:53,763: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f71f944e-dc55-4fc7-a6b9-51b7fcc4d53e
[2017-06-29 11:43:53,888: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:43:53,888: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:43:53,889: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:43:53,889: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:43:53,889: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f71f944e-dc55-4fc7-a6b9-51b7fcc4d53e
[2017-06-29 11:43:53,889: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f71f944e-dc55-4fc7-a6b9-51b7fcc4d53e
[2017-06-29 11:43:53,936: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f71f944e-dc55-4fc7-a6b9-51b7fcc4d53e
[2017-06-29 11:43:53,936: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f71f944e-dc55-4fc7-a6b9-51b7fcc4d53e
[2017-06-29 11:43:53,936: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm7.lotus.hpdd.lab.intel.com (old ef1b08c2-d8e1-45c1-ae99-20c909aa0c61)
[2017-06-29 11:43:53,936: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: awaiting lotus-32vm7.lotus.hpdd.lab.intel.com (old ef1b08c2-d8e1-45c1-ae99-20c909aa0c61)
[2017-06-29 11:44:22,411: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:44:22,411: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:44:22,411: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:44:22,411: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:44:22,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2ca563a0-6610-4e1b-a9d5-71e39198accd
[2017-06-29 11:44:22,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2ca563a0-6610-4e1b-a9d5-71e39198accd
[2017-06-29 11:44:22,413: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2ca563a0-6610-4e1b-a9d5-71e39198accd
[2017-06-29 11:44:22,413: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2ca563a0-6610-4e1b-a9d5-71e39198accd
[2017-06-29 11:44:22,413: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.227', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-06-29 11:44:22,427: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a69658b8-13a3-477d-96f6-a113cf33db95
[2017-06-29 11:44:22,427: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a69658b8-13a3-477d-96f6-a113cf33db95
[2017-06-29 11:44:22,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:44:22,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:44:22,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:44:22,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:44:22,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a69658b8-13a3-477d-96f6-a113cf33db95
[2017-06-29 11:44:22,479: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a69658b8-13a3-477d-96f6-a113cf33db95
[2017-06-29 11:44:22,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a69658b8-13a3-477d-96f6-a113cf33db95
[2017-06-29 11:44:22,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a69658b8-13a3-477d-96f6-a113cf33db95
[2017-06-29 11:44:22,508: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 42897, 'pcs_password': 'ihmVsEIhOymqEScpS8dA'}
[2017-06-29 11:44:22,541: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0f6bd09f-1e6d-4dc3-af2b-c2bd132d6c13
[2017-06-29 11:44:22,541: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0f6bd09f-1e6d-4dc3-af2b-c2bd132d6c13
[2017-06-29 11:44:35,656: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:44:35,656: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:44:35,656: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:44:35,656: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:44:35,656: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0f6bd09f-1e6d-4dc3-af2b-c2bd132d6c13
[2017-06-29 11:44:35,656: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0f6bd09f-1e6d-4dc3-af2b-c2bd132d6c13
[2017-06-29 11:44:35,674: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0f6bd09f-1e6d-4dc3-af2b-c2bd132d6c13
[2017-06-29 11:44:35,674: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0f6bd09f-1e6d-4dc3-af2b-c2bd132d6c13
[2017-06-29 11:44:35,675: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': True, 'mcast_port': 42897, 'new_node_fqdn': u'lotus-32vm8.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'ihmVsEIhOymqEScpS8dA'}
[2017-06-29 11:44:35,692: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f63eb992-602b-493a-bb34-45b8e8a39d9a
[2017-06-29 11:44:35,692: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f63eb992-602b-493a-bb34-45b8e8a39d9a
[2017-06-29 11:44:41,835: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:44:41,835: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:44:41,836: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:44:41,836: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:44:41,836: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f63eb992-602b-493a-bb34-45b8e8a39d9a
[2017-06-29 11:44:41,836: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f63eb992-602b-493a-bb34-45b8e8a39d9a
[2017-06-29 11:44:41,870: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f63eb992-602b-493a-bb34-45b8e8a39d9a
[2017-06-29 11:44:41,870: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f63eb992-602b-493a-bb34-45b8e8a39d9a
[2017-06-29 11:44:41,870: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-06-29 11:44:41.870827+00:00:
[2017-06-29 11:44:41,870: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-06-29 11:44:41.870827+00:00:
[2017-06-29 11:44:41,871: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->42897
[2017-06-29 11:44:41,871: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->42897
[2017-06-29 11:44:41,872: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-06-29 11:44:41,872: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-06-29 11:44:41,894: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-29 11:44:41,894: INFO/job_scheduler]   Job 74 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-06-29 11:44:41,894: INFO/job_scheduler]   Job 75 writelock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-06-29 11:44:41,894: INFO/job_scheduler]   Job 78 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:44:41,894: INFO/job_scheduler]   Job 79 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:44:41,895: INFO/job_scheduler] run_next: 0 runnable jobs of (14 pending, 2 tasked)
[2017-06-29 11:44:41,896: INFO/job_scheduler] Job 74 finished 1 steps successfully
[2017-06-29 11:44:42,004: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 140668098585168
[2017-06-29 11:44:42,040: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-06-29 11:44:42,067: INFO/job] Job 74: StateChangeJob complete, setting state stopped on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:44:42,068: INFO/job] Job 74: not_deleted=True
[2017-06-29 11:44:42,068: INFO/job_scheduler] Job 74: complete_job: Updating cache
[2017-06-29 11:44:42,068: INFO/job_scheduler] Job 74: locked_item 140668098585168 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-29 11:44:42,085: INFO/job_scheduler] Job 74 completing (errored=False, cancelled=False)
[2017-06-29 11:44:42,101: INFO/job_scheduler] run_next: 1 runnable jobs of (14 pending, 1 tasked)
[2017-06-29 11:44:42,111: INFO/job_scheduler] Job 75: RunJobThread.run
[2017-06-29 11:44:42,112: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-06-29 11:44:42,164: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5a2acbc3-c7ab-4317-a590-172afe839299
[2017-06-29 11:44:42,164: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5a2acbc3-c7ab-4317-a590-172afe839299
[2017-06-29 11:44:50,303: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:44:50,303: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:44:50,303: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:44:50,303: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:44:50,303: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5a2acbc3-c7ab-4317-a590-172afe839299
[2017-06-29 11:44:50,303: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5a2acbc3-c7ab-4317-a590-172afe839299
[2017-06-29 11:44:50,351: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5a2acbc3-c7ab-4317-a590-172afe839299
[2017-06-29 11:44:50,351: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5a2acbc3-c7ab-4317-a590-172afe839299
[2017-06-29 11:44:50,351: INFO/job_scheduler] Job 75 finished 1 steps successfully
[2017-06-29 11:44:50,427: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 140668098585168
[2017-06-29 11:44:50,463: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm8.lotus.hpdd.lab.intel.com at severity 30
[2017-06-29 11:44:50,478: INFO/job] Job 75: StateChangeJob complete, setting state started on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:44:50,478: INFO/job] Job 75: not_deleted=True
[2017-06-29 11:44:50,478: INFO/job_scheduler] Job 75: complete_job: Updating cache
[2017-06-29 11:44:50,478: INFO/job_scheduler] Job 75: locked_item 140668098585168 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-29 11:44:50,486: INFO/job_scheduler] Job 75 completing (errored=False, cancelled=False)
[2017-06-29 11:44:50,515: INFO/job_scheduler] run_next: 1 runnable jobs of (13 pending, 1 tasked)
[2017-06-29 11:44:50,564: INFO/job_scheduler] Job 78: RunJobThread.run
[2017-06-29 11:44:50,565: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-06-29 11:44:50,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d471a9cb-a710-4022-97fe-01e89783e1b7
[2017-06-29 11:44:50,589: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d471a9cb-a710-4022-97fe-01e89783e1b7
[2017-06-29 11:44:52,334: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 2 tasked)
[2017-06-29 11:44:52,350: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 2 tasked)
[2017-06-29 11:44:56,374: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:44:56,374: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/ef1b08c2-d8e1-45c1-ae99-20c909aa0c61
[2017-06-29 11:44:56,374: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session ef1b08c2-d8e1-45c1-ae99-20c909aa0c61 because session terminated
[2017-06-29 11:44:56,374: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session ef1b08c2-d8e1-45c1-ae99-20c909aa0c61 because session terminated
[2017-06-29 11:44:57,715: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:44:57,715: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:44:57,715: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:44:57,715: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:44:57,715: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d471a9cb-a710-4022-97fe-01e89783e1b7
[2017-06-29 11:44:57,715: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d471a9cb-a710-4022-97fe-01e89783e1b7
[2017-06-29 11:44:57,716: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d471a9cb-a710-4022-97fe-01e89783e1b7
[2017-06-29 11:44:57,716: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d471a9cb-a710-4022-97fe-01e89783e1b7
[2017-06-29 11:44:57,717: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-06-29 11:44:57,731: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f7584619-1af2-4e1f-8ce1-9f2694b49fd8
[2017-06-29 11:44:57,731: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f7584619-1af2-4e1f-8ce1-9f2694b49fd8
[2017-06-29 11:45:02,359: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 2 tasked)
[2017-06-29 11:45:02,373: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 2 tasked)
[2017-06-29 11:45:22,405: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 2 tasked)
[2017-06-29 11:45:22,447: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 2 tasked)
[2017-06-29 11:45:52,553: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-29 11:45:52,554: INFO/job_scheduler]   Job 78 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:45:52,554: INFO/job_scheduler]   Job 79 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:45:52,554: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 2 tasked)
[2017-06-29 11:46:02,532: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-29 11:46:02,532: INFO/job_scheduler]   Job 78 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:46:02,533: INFO/job_scheduler]   Job 79 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:46:02,533: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 2 tasked)
[2017-06-29 11:46:12,593: INFO/job_scheduler] _notify: Buffering update to lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-29 11:46:12,594: INFO/job_scheduler]   Job 78 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:46:12,594: INFO/job_scheduler]   Job 79 readlock on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:46:12,594: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 2 tasked)
[2017-06-29 11:46:18,155: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:46:18,155: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:46:18,155: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:46:18,155: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:46:18,155: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f7584619-1af2-4e1f-8ce1-9f2694b49fd8
[2017-06-29 11:46:18,155: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f7584619-1af2-4e1f-8ce1-9f2694b49fd8
[2017-06-29 11:46:18,190: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f7584619-1af2-4e1f-8ce1-9f2694b49fd8
[2017-06-29 11:46:18,190: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f7584619-1af2-4e1f-8ce1-9f2694b49fd8
[2017-06-29 11:46:18,192: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-06-29 11:46:18,220: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 38cb2c8e-0034-49b2-93aa-c448e5adfd7f
[2017-06-29 11:46:18,220: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 38cb2c8e-0034-49b2-93aa-c448e5adfd7f
[2017-06-29 11:46:22,610: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 2 tasked)
[2017-06-29 11:46:22,643: INFO/job_scheduler] run_next: 0 runnable jobs of (12 pending, 2 tasked)
[2017-06-29 11:46:25,341: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:46:25,341: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:46:25,342: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:46:25,342: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:46:25,342: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 38cb2c8e-0034-49b2-93aa-c448e5adfd7f
[2017-06-29 11:46:25,342: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 38cb2c8e-0034-49b2-93aa-c448e5adfd7f
[2017-06-29 11:46:25,345: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 38cb2c8e-0034-49b2-93aa-c448e5adfd7f
[2017-06-29 11:46:25,345: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 38cb2c8e-0034-49b2-93aa-c448e5adfd7f
[2017-06-29 11:46:25,345: INFO/job_scheduler] Job 78 finished 3 steps successfully
[2017-06-29 11:46:25,433: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 140668098710864
[2017-06-29 11:46:25,445: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-06-29 11:46:25,448: INFO/job] Job 78: StateChangeJob complete, setting state stopped on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-29 11:46:25,448: INFO/job] Job 78: not_deleted=True
[2017-06-29 11:46:25,448: INFO/job_scheduler] Job 78: complete_job: Updating cache
[2017-06-29 11:46:25,448: INFO/job_scheduler] Job 78: locked_item 140668098710864 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-29 11:46:25,454: INFO/job_scheduler] Job 78 completing (errored=False, cancelled=False)
[2017-06-29 11:46:25,498: INFO/job_scheduler] run_next: 1 runnable jobs of (12 pending, 1 tasked)
[2017-06-29 11:46:25,552: INFO/job_scheduler] Job 79: RunJobThread.run
[2017-06-29 11:46:25,553: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-06-29 11:46:25,585: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0870da8b-2270-4790-be12-8ae1b92273d4
[2017-06-29 11:46:25,585: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0870da8b-2270-4790-be12-8ae1b92273d4
[2017-06-29 11:46:32,616: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 2 tasked)
[2017-06-29 11:46:32,632: INFO/job_scheduler] run_next: 0 runnable jobs of (11 pending, 2 tasked)
[2017-06-29 11:46:32,676: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:46:32,676: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:46:32,676: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:46:32,676: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:46:32,677: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0870da8b-2270-4790-be12-8ae1b92273d4
[2017-06-29 11:46:32,677: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0870da8b-2270-4790-be12-8ae1b92273d4
[2017-06-29 11:46:32,712: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0870da8b-2270-4790-be12-8ae1b92273d4
[2017-06-29 11:46:32,712: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0870da8b-2270-4790-be12-8ae1b92273d4
[2017-06-29 11:46:32,713: INFO/job_scheduler] Job 79 finished 1 steps successfully
[2017-06-29 11:46:32,795: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 140668098710864
[2017-06-29 11:46:32,814: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm8.lotus.hpdd.lab.intel.com at severity 30
[2017-06-29 11:46:32,821: INFO/job] Job 79: StateChangeJob complete, setting state started on lotus-32vm8.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-29 11:46:32,821: INFO/job] Job 79: not_deleted=True
[2017-06-29 11:46:32,821: INFO/job_scheduler] Job 79: complete_job: Updating cache
[2017-06-29 11:46:32,821: INFO/job_scheduler] Job 79: locked_item 140668098710864 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-29 11:46:32,827: INFO/job_scheduler] Job 79 completing (errored=False, cancelled=False)
[2017-06-29 11:46:32,844: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-5
[2017-06-29 11:46:32,844: INFO/job_scheduler] _notify: Updating .mcast_port of item 5 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 42897
[2017-06-29 11:46:32,848: INFO/job_scheduler] _notify: Set mcast_port=42897 on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-5) and saved
[2017-06-29 11:46:32,850: INFO/job_scheduler] _notify: Updating .network_interfaces of item 5 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-06-29 11:46:32,866: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-5) and saved
[2017-06-29 11:46:32,873: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 5 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-06-29 11:46:32,891: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-5) and saved
[2017-06-29 11:46:32,914: INFO/job_scheduler] run_next: 1 runnable jobs of (11 pending, 1 tasked)
[2017-06-29 11:46:32,947: INFO/job_scheduler] Job 80: RunJobThread.run
[2017-06-29 11:46:32,949: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-06-29 11:46:32,999: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9093fd02-1e7b-41e9-b26a-fa35ef5a2d79
[2017-06-29 11:46:32,999: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9093fd02-1e7b-41e9-b26a-fa35ef5a2d79
[2017-06-29 11:46:42,629: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 5 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from True to False
[2017-06-29 11:46:42,634: INFO/job_scheduler] _notify: Set corosync_reported_up=False on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-5) and saved
[2017-06-29 11:46:42,644: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 2 tasked)
[2017-06-29 11:46:52,693: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 5 (lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-06-29 11:46:52,698: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm8.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-5) and saved
[2017-06-29 11:46:52,707: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 2 tasked)
[2017-06-29 11:46:58,295: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:46:58,295: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:46:58,296: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:46:58,296: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:46:58,296: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9093fd02-1e7b-41e9-b26a-fa35ef5a2d79
[2017-06-29 11:46:58,296: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9093fd02-1e7b-41e9-b26a-fa35ef5a2d79
[2017-06-29 11:46:58,339: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9093fd02-1e7b-41e9-b26a-fa35ef5a2d79
[2017-06-29 11:46:58,339: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9093fd02-1e7b-41e9-b26a-fa35ef5a2d79
[2017-06-29 11:46:58,341: INFO/job_scheduler] Job 80 finished 1 steps successfully
[2017-06-29 11:46:58,930: INFO/job] StatefulObject.set_state lotus-32vm8.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 140667427279504
[2017-06-29 11:46:58,940: INFO/job] Job 80: StateChangeJob complete, setting state managed on lotus-32vm8.lotus.hpdd.lab.intel.com
[2017-06-29 11:46:58,940: INFO/job] Job 80: not_deleted=True
[2017-06-29 11:46:58,940: INFO/job_scheduler] Job 80: complete_job: Updating cache
[2017-06-29 11:46:58,940: INFO/job_scheduler] Job 80: locked_item 140667427279504 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-29 11:46:58,943: INFO/job_scheduler] Job 80 completing (errored=False, cancelled=False)
[2017-06-29 11:46:58,982: INFO/job_scheduler] Replaying 5 buffered notifications for ManagedHost-5
[2017-06-29 11:46:58,999: INFO/job_scheduler] _notify: Updating .immutable_state of item 5 (lotus-32vm8.lotus.hpdd.lab.intel.com) from True to False
[2017-06-29 11:46:59,005: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-5) and saved
[2017-06-29 11:46:59,019: INFO/job_scheduler] _notify: Updating .boot_time of item 5 (lotus-32vm8.lotus.hpdd.lab.intel.com) from 2017-06-15 09:59:41+00:00 to 2017-06-29 04:42:35+00:00
[2017-06-29 11:46:59,026: INFO/job_scheduler] _notify: Set boot_time=2017-06-29 04:42:35+00:00 on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-5) and saved
[2017-06-29 11:46:59,044: INFO/job_scheduler] _notify: Updating .properties of item 5 (lotus-32vm8.lotus.hpdd.lab.intel.com) from {"python_patchlevel": 5, "kernel_version": "3.10.0-514.21.1.el7_lustre.x86_64", "zfs_installed": false, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.21.1.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-06-29 11:46:59,076: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.21.1.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-5) and saved
[2017-06-29 11:46:59,142: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 1 tasked)
[2017-06-29 11:48:08,820: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 11:48:08,821: INFO/job_scheduler]   Job 59 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed
[2017-06-29 11:48:08,821: INFO/job_scheduler]   Job 60 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:48:08,821: INFO/job_scheduler]   Job 61 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:48:08,821: INFO/job_scheduler]   Job 62 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:48:08,821: INFO/job_scheduler]   Job 63 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:48:08,821: INFO/job_scheduler]   Job 67 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:48:08,821: INFO/job_scheduler]   Job 69 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-29 11:48:08,822: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 1 tasked)
[2017-06-29 11:48:08,877: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:08,877: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:09,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm7.lotus.hpdd.lab.intel.com new 98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:09,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.await_restart: lotus-32vm7.lotus.hpdd.lab.intel.com new 98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:09,283: INFO/job_scheduler] Job 59 finished 4 steps successfully
[2017-06-29 11:48:09,332: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->packages_installed (intentional=True) 140668098571600
[2017-06-29 11:48:09,336: INFO/job] Job 59: StateChangeJob complete, setting state packages_installed on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:48:09,336: INFO/job] Job 59: not_deleted=True
[2017-06-29 11:48:09,336: INFO/job_scheduler] Job 59: complete_job: Updating cache
[2017-06-29 11:48:09,336: INFO/job_scheduler] Job 59: locked_item 140668098748304 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-29 11:48:09,339: INFO/job_scheduler] Job 59 completing (errored=False, cancelled=False)
[2017-06-29 11:48:09,375: INFO/job_scheduler] run_next: 4 runnable jobs of (10 pending, 0 tasked)
[2017-06-29 11:48:09,481: INFO/job_scheduler] Job 60: RunJobThread.run
[2017-06-29 11:48:09,482: INFO/job_scheduler] Job 61: open DB connection during completion
[2017-06-29 11:48:09,487: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_ntp {'ntp_server': 'lotus-32vm3.lotus.hpdd.lab.intel.com'}
[2017-06-29 11:48:09,514: INFO/job_scheduler] Job 62: RunJobThread.run
[2017-06-29 11:48:09,515: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com get_corosync_autoconfig {}
[2017-06-29 11:48:09,525: INFO/job_scheduler] Job 63: RunJobThread.run
[2017-06-29 11:48:09,526: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_rsyslog {}
[2017-06-29 11:48:09,531: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration unconfigured->lnet_unloaded (intentional=True) 140668098588432
[2017-06-29 11:48:09,532: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 93c4f611-fae2-4277-8ea8-dcb69f26686b
[2017-06-29 11:48:09,532: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 93c4f611-fae2-4277-8ea8-dcb69f26686b
[2017-06-29 11:48:09,548: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-06-29 11:48:09,557: INFO/job] Job 61: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-29 11:48:09,558: INFO/job] Job 61: not_deleted=True
[2017-06-29 11:48:09,558: INFO/job_scheduler] Job 61: complete_job: Updating cache
[2017-06-29 11:48:09,558: INFO/job_scheduler] Job 61: locked_item 140668098588432 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-29 11:48:09,572: INFO/job_scheduler] Job 61 completing (errored=False, cancelled=False)
[2017-06-29 11:48:09,605: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3f3b2395-2a15-4568-ab34-65c5ddb5a877
[2017-06-29 11:48:09,605: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3f3b2395-2a15-4568-ab34-65c5ddb5a877
[2017-06-29 11:48:09,616: INFO/job_scheduler] run_next: 1 runnable jobs of (6 pending, 3 tasked)
[2017-06-29 11:48:09,636: INFO/job_scheduler] Job 65: RunJobThread.run
[2017-06-29 11:48:09,637: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com load_lnet {}
[2017-06-29 11:48:09,646: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8903bef9-e38e-4e17-83d9-9a36775f04a8
[2017-06-29 11:48:09,646: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8903bef9-e38e-4e17-83d9-9a36775f04a8
[2017-06-29 11:48:09,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1d22469f-3c15-4f91-a564-02942d5a1cce
[2017-06-29 11:48:09,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1d22469f-3c15-4f91-a564-02942d5a1cce
[2017-06-29 11:48:09,905: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:09,905: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:09,906: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:09,906: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:09,906: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1d22469f-3c15-4f91-a564-02942d5a1cce
[2017-06-29 11:48:09,906: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1d22469f-3c15-4f91-a564-02942d5a1cce
[2017-06-29 11:48:09,925: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1d22469f-3c15-4f91-a564-02942d5a1cce
[2017-06-29 11:48:09,925: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1d22469f-3c15-4f91-a564-02942d5a1cce
[2017-06-29 11:48:09,925: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-29 11:48:09,943: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cc260f42-3280-42b3-b9c7-3f261bb6b917
[2017-06-29 11:48:09,943: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc cc260f42-3280-42b3-b9c7-3f261bb6b917
[2017-06-29 11:48:11,016: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:11,016: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:11,017: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:11,017: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:11,017: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cc260f42-3280-42b3-b9c7-3f261bb6b917
[2017-06-29 11:48:11,017: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc cc260f42-3280-42b3-b9c7-3f261bb6b917
[2017-06-29 11:48:11,068: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cc260f42-3280-42b3-b9c7-3f261bb6b917
[2017-06-29 11:48:11,068: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc cc260f42-3280-42b3-b9c7-3f261bb6b917
[2017-06-29 11:48:11,200: INFO/job_scheduler] Job 65 finished 2 steps successfully
[2017-06-29 11:48:11,289: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->lnet_down (intentional=True) 140668098588432
[2017-06-29 11:48:11,296: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-06-29 11:48:11,313: INFO/job] Job 65: StateChangeJob complete, setting state lnet_down on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-29 11:48:11,313: INFO/job] Job 65: not_deleted=True
[2017-06-29 11:48:11,313: INFO/job_scheduler] Job 65: complete_job: Updating cache
[2017-06-29 11:48:11,313: INFO/job_scheduler] Job 65: locked_item 140668098588432 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-29 11:48:11,322: INFO/job_scheduler] Job 65 completing (errored=False, cancelled=False)
[2017-06-29 11:48:11,350: INFO/job_scheduler] run_next: 1 runnable jobs of (5 pending, 3 tasked)
[2017-06-29 11:48:11,382: INFO/job_scheduler] Job 66: RunJobThread.run
[2017-06-29 11:48:11,382: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_lnet {}
[2017-06-29 11:48:11,416: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a3a58f3f-4287-4da2-8755-c15800901a18
[2017-06-29 11:48:11,416: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a3a58f3f-4287-4da2-8755-c15800901a18
[2017-06-29 11:48:11,508: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:11,508: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:11,508: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:11,508: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:11,508: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a3a58f3f-4287-4da2-8755-c15800901a18
[2017-06-29 11:48:11,508: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a3a58f3f-4287-4da2-8755-c15800901a18
[2017-06-29 11:48:11,541: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a3a58f3f-4287-4da2-8755-c15800901a18
[2017-06-29 11:48:11,541: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a3a58f3f-4287-4da2-8755-c15800901a18
[2017-06-29 11:48:11,542: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-29 11:48:11,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5414c91f-73bb-4f03-b7be-f036db8b5807
[2017-06-29 11:48:11,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5414c91f-73bb-4f03-b7be-f036db8b5807
[2017-06-29 11:48:11,699: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:11,699: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:11,699: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:11,699: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:11,699: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5414c91f-73bb-4f03-b7be-f036db8b5807
[2017-06-29 11:48:11,699: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5414c91f-73bb-4f03-b7be-f036db8b5807
[2017-06-29 11:48:11,725: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5414c91f-73bb-4f03-b7be-f036db8b5807
[2017-06-29 11:48:11,725: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5414c91f-73bb-4f03-b7be-f036db8b5807
[2017-06-29 11:48:12,024: INFO/job_scheduler] Job 66 finished 2 steps successfully
[2017-06-29 11:48:12,084: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_up (intentional=True) 140668098588432
[2017-06-29 11:48:12,089: INFO/job] Job 66: StateChangeJob complete, setting state lnet_up on lotus-32vm7.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-29 11:48:12,089: INFO/job] Job 66: not_deleted=True
[2017-06-29 11:48:12,090: INFO/job_scheduler] Job 66: complete_job: Updating cache
[2017-06-29 11:48:12,090: INFO/job_scheduler] Job 66: locked_item 140668098588432 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-29 11:48:12,094: INFO/job_scheduler] Job 66 completing (errored=False, cancelled=False)
[2017-06-29 11:48:12,119: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 3 tasked)
[2017-06-29 11:48:13,596: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 11:48:13,597: INFO/job_scheduler]   Job 60 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:48:13,597: INFO/job_scheduler]   Job 62 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:48:13,597: INFO/job_scheduler]   Job 63 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:48:13,597: INFO/job_scheduler]   Job 67 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:48:13,597: INFO/job_scheduler]   Job 69 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-29 11:48:13,597: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 3 tasked)
[2017-06-29 11:48:13,653: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 11:48:13,653: INFO/job_scheduler]   Job 60 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:48:13,654: INFO/job_scheduler]   Job 62 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:48:13,654: INFO/job_scheduler]   Job 63 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:48:13,654: INFO/job_scheduler]   Job 67 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:48:13,654: INFO/job_scheduler]   Job 69 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-29 11:48:13,654: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 3 tasked)
[2017-06-29 11:48:15,666: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:15,666: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:15,667: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:15,667: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:15,667: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 93c4f611-fae2-4277-8ea8-dcb69f26686b
[2017-06-29 11:48:15,667: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 93c4f611-fae2-4277-8ea8-dcb69f26686b
[2017-06-29 11:48:15,703: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 93c4f611-fae2-4277-8ea8-dcb69f26686b
[2017-06-29 11:48:15,703: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 93c4f611-fae2-4277-8ea8-dcb69f26686b
[2017-06-29 11:48:15,704: INFO/job_scheduler] Job 60 finished 1 steps successfully
[2017-06-29 11:48:15,800: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com NTP configuration unconfigured->configured (intentional=True) 140668098588304
[2017-06-29 11:48:15,804: INFO/job] Job 60: StateChangeJob complete, setting state configured on lotus-32vm7.lotus.hpdd.lab.intel.com NTP configuration
[2017-06-29 11:48:15,804: INFO/job] Job 60: not_deleted=True
[2017-06-29 11:48:15,804: INFO/job_scheduler] Job 60: complete_job: Updating cache
[2017-06-29 11:48:15,804: INFO/job_scheduler] Job 60: locked_item 140668098588304 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-06-29 11:48:15,807: INFO/job_scheduler] Job 60 completing (errored=False, cancelled=False)
[2017-06-29 11:48:15,827: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-29 11:48:16,749: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:16,749: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:16,749: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:16,749: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:16,750: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8903bef9-e38e-4e17-83d9-9a36775f04a8
[2017-06-29 11:48:16,750: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8903bef9-e38e-4e17-83d9-9a36775f04a8
[2017-06-29 11:48:16,773: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8903bef9-e38e-4e17-83d9-9a36775f04a8
[2017-06-29 11:48:16,773: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8903bef9-e38e-4e17-83d9-9a36775f04a8
[2017-06-29 11:48:16,773: INFO/job_scheduler] Job 63 finished 1 steps successfully
[2017-06-29 11:48:16,983: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->configured (intentional=True) 140668098587984
[2017-06-29 11:48:16,995: INFO/job] Job 63: StateChangeJob complete, setting state configured on lotus-32vm7.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-06-29 11:48:16,995: INFO/job] Job 63: not_deleted=True
[2017-06-29 11:48:16,995: INFO/job_scheduler] Job 63: complete_job: Updating cache
[2017-06-29 11:48:16,995: INFO/job_scheduler] Job 63: locked_item 140668098587984 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-06-29 11:48:16,998: INFO/job_scheduler] Job 63 completing (errored=False, cancelled=False)
[2017-06-29 11:48:17,018: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 1 tasked)
[2017-06-29 11:48:27,367: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:27,367: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:27,368: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:27,368: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:27,368: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3f3b2395-2a15-4568-ab34-65c5ddb5a877
[2017-06-29 11:48:27,368: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3f3b2395-2a15-4568-ab34-65c5ddb5a877
[2017-06-29 11:48:27,410: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3f3b2395-2a15-4568-ab34-65c5ddb5a877
[2017-06-29 11:48:27,410: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3f3b2395-2a15-4568-ab34-65c5ddb5a877
[2017-06-29 11:48:27,410: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_network {'ring1_ipaddr': u'10.128.1.226', 'ring1_name': u'eth1', 'ring0_name': u'eth0', 'ring1_prefix': 21}
[2017-06-29 11:48:27,436: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b0da672a-9f69-4890-899d-f28c2b8fab54
[2017-06-29 11:48:27,436: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b0da672a-9f69-4890-899d-f28c2b8fab54
[2017-06-29 11:48:27,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:27,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:27,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:27,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:27,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b0da672a-9f69-4890-899d-f28c2b8fab54
[2017-06-29 11:48:27,495: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b0da672a-9f69-4890-899d-f28c2b8fab54
[2017-06-29 11:48:27,511: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b0da672a-9f69-4890-899d-f28c2b8fab54
[2017-06-29 11:48:27,511: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b0da672a-9f69-4890-899d-f28c2b8fab54
[2017-06-29 11:48:27,541: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_corosync2_stage_1 {'mcast_port': 42897, 'pcs_password': 'LTnm40vvUwcX6LzIlCXL'}
[2017-06-29 11:48:27,574: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4184bb32-6e93-4350-b1e6-b56084e03d2b
[2017-06-29 11:48:27,574: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4184bb32-6e93-4350-b1e6-b56084e03d2b
[2017-06-29 11:48:40,749: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:40,749: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:40,750: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:40,750: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:40,750: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4184bb32-6e93-4350-b1e6-b56084e03d2b
[2017-06-29 11:48:40,750: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4184bb32-6e93-4350-b1e6-b56084e03d2b
[2017-06-29 11:48:40,755: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4184bb32-6e93-4350-b1e6-b56084e03d2b
[2017-06-29 11:48:40,755: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4184bb32-6e93-4350-b1e6-b56084e03d2b
[2017-06-29 11:48:40,756: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_corosync2_stage_2 {'ring1_name': u'eth1', 'create_cluster': False, 'mcast_port': 42897, 'new_node_fqdn': u'lotus-32vm7.lotus.hpdd.lab.intel.com', 'ring0_name': u'eth0', 'pcs_password': 'LTnm40vvUwcX6LzIlCXL'}
[2017-06-29 11:48:40,785: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3fb690e4-cea0-40f7-897d-898761a1fdea
[2017-06-29 11:48:40,785: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3fb690e4-cea0-40f7-897d-898761a1fdea
[2017-06-29 11:48:46,892: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:48:46,892: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:48:46,893: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:48:46,893: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 11:48:46,893: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3fb690e4-cea0-40f7-897d-898761a1fdea
[2017-06-29 11:48:46,893: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3fb690e4-cea0-40f7-897d-898761a1fdea
[2017-06-29 11:48:46,906: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3fb690e4-cea0-40f7-897d-898761a1fdea
[2017-06-29 11:48:46,906: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3fb690e4-cea0-40f7-897d-898761a1fdea
[2017-06-29 11:48:46,906: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-06-29 11:48:46.906548+00:00:
[2017-06-29 11:48:46,906: INFO/chroma_core.services.job_scheduler.job_scheduler_notify] Enqueuing notify lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration at 2017-06-29 11:48:46.906548+00:00:
[2017-06-29 11:48:46,906: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->42897
[2017-06-29 11:48:46,906: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .mcast_port None->42897
[2017-06-29 11:48:46,908: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-06-29 11:48:46,908: INFO/chroma_core.services.job_scheduler.job_scheduler_notify]   .network_interfaces []->[u'eth0', u'eth1']
[2017-06-29 11:48:46,921: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-29 11:48:46,922: INFO/job_scheduler]   Job 62 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped
[2017-06-29 11:48:46,922: INFO/job_scheduler]   Job 64 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-06-29 11:48:46,922: INFO/job_scheduler]   Job 67 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:48:46,922: INFO/job_scheduler]   Job 68 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:48:46,922: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 1 tasked)
[2017-06-29 11:48:46,924: INFO/job_scheduler] Job 62 finished 1 steps successfully
[2017-06-29 11:48:46,978: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->stopped (intentional=True) 140668098569040
[2017-06-29 11:48:46,987: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-06-29 11:48:47,008: INFO/job] Job 62: StateChangeJob complete, setting state stopped on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:48:47,008: INFO/job] Job 62: not_deleted=True
[2017-06-29 11:48:47,008: INFO/job_scheduler] Job 62: complete_job: Updating cache
[2017-06-29 11:48:47,008: INFO/job_scheduler] Job 62: locked_item 140668098569040 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-29 11:48:47,015: INFO/job_scheduler] Job 62 completing (errored=False, cancelled=False)
[2017-06-29 11:48:47,041: INFO/job_scheduler] run_next: 1 runnable jobs of (4 pending, 0 tasked)
[2017-06-29 11:48:47,069: INFO/job_scheduler] Job 64: RunJobThread.run
[2017-06-29 11:48:47,069: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_corosync2 {}
[2017-06-29 11:48:47,083: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 848bd0de-e693-4c3a-a883-1dae896200f2
[2017-06-29 11:48:47,083: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 848bd0de-e693-4c3a-a883-1dae896200f2
[2017-06-29 11:48:53,020: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-29 11:48:53,020: INFO/job_scheduler]   Job 64 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started
[2017-06-29 11:48:53,020: INFO/job_scheduler]   Job 67 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:48:53,021: INFO/job_scheduler]   Job 68 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:48:53,021: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 11:48:53,031: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com because of locks
[2017-06-29 11:48:53,032: INFO/job_scheduler]   Job 67 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:48:53,032: INFO/job_scheduler]   Job 69 writelock on lotus-32vm7.lotus.hpdd.lab.intel.com packages_installed->managed
[2017-06-29 11:48:53,032: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 11:48:53,057: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 5 (lotus-32vm8.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7fefc80fd690> to [4]
[2017-06-29 11:48:53,085: INFO/job_scheduler] _notify: Set ha_cluster_peers=[4] on lotus-32vm8.lotus.hpdd.lab.intel.com (ManagedHost-5) and saved
[2017-06-29 11:48:53,229: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 11:48:53,791: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 11:48:53,857: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 11:48:55,184: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:55,184: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:55,184: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:55,184: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:48:55,185: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 848bd0de-e693-4c3a-a883-1dae896200f2
[2017-06-29 11:48:55,185: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 848bd0de-e693-4c3a-a883-1dae896200f2
[2017-06-29 11:48:55,215: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 848bd0de-e693-4c3a-a883-1dae896200f2
[2017-06-29 11:48:55,215: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 848bd0de-e693-4c3a-a883-1dae896200f2
[2017-06-29 11:48:55,215: INFO/job_scheduler] Job 64 finished 1 steps successfully
[2017-06-29 11:48:55,267: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->started (intentional=True) 140668098569040
[2017-06-29 11:48:55,316: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 30
[2017-06-29 11:48:55,350: INFO/job] Job 64: StateChangeJob complete, setting state started on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:48:55,350: INFO/job] Job 64: not_deleted=True
[2017-06-29 11:48:55,350: INFO/job_scheduler] Job 64: complete_job: Updating cache
[2017-06-29 11:48:55,350: INFO/job_scheduler] Job 64: locked_item 140668098569040 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-29 11:48:55,355: INFO/job_scheduler] Job 64 completing (errored=False, cancelled=False)
[2017-06-29 11:48:55,372: INFO/job_scheduler] run_next: 1 runnable jobs of (3 pending, 0 tasked)
[2017-06-29 11:48:55,413: INFO/job_scheduler] Job 67: RunJobThread.run
[2017-06-29 11:48:55,414: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-06-29 11:48:55,463: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 71d57c84-147c-420c-9e97-27c342f3323c
[2017-06-29 11:48:55,463: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 71d57c84-147c-420c-9e97-27c342f3323c
[2017-06-29 11:49:00,376: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-29 11:49:00,396: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-29 11:49:02,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:49:02,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:49:02,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:49:02,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:49:02,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 71d57c84-147c-420c-9e97-27c342f3323c
[2017-06-29 11:49:02,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 71d57c84-147c-420c-9e97-27c342f3323c
[2017-06-29 11:49:02,752: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 71d57c84-147c-420c-9e97-27c342f3323c
[2017-06-29 11:49:02,752: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 71d57c84-147c-420c-9e97-27c342f3323c
[2017-06-29 11:49:02,753: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_pacemaker {}
[2017-06-29 11:49:02,769: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7f0f1cd1-a132-4322-a30f-7d789b99fac0
[2017-06-29 11:49:02,769: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7f0f1cd1-a132-4322-a30f-7d789b99fac0
[2017-06-29 11:49:03,061: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-29 11:49:03,061: INFO/job_scheduler]   Job 67 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:49:03,062: INFO/job_scheduler]   Job 68 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:49:03,062: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-29 11:49:03,789: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-29 11:49:03,816: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-29 11:49:03,867: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-29 11:49:03,868: INFO/job_scheduler]   Job 67 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:49:03,868: INFO/job_scheduler]   Job 68 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:49:03,868: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-29 11:49:12,898: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-29 11:49:12,939: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-29 11:49:13,148: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-29 11:49:13,149: INFO/job_scheduler]   Job 67 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:49:13,149: INFO/job_scheduler]   Job 68 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:49:13,149: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-29 11:49:22,945: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:49:22,945: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:49:22,946: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:49:22,946: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:49:22,946: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7f0f1cd1-a132-4322-a30f-7d789b99fac0
[2017-06-29 11:49:22,946: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7f0f1cd1-a132-4322-a30f-7d789b99fac0
[2017-06-29 11:49:22,955: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7f0f1cd1-a132-4322-a30f-7d789b99fac0
[2017-06-29 11:49:22,955: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7f0f1cd1-a132-4322-a30f-7d789b99fac0
[2017-06-29 11:49:22,957: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-06-29 11:49:22,970: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 52ecdd2a-6530-42d4-b071-34f9e08cb034
[2017-06-29 11:49:22,970: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 52ecdd2a-6530-42d4-b071-34f9e08cb034
[2017-06-29 11:49:23,109: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-29 11:49:23,109: INFO/job_scheduler]   Job 67 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:49:23,109: INFO/job_scheduler]   Job 68 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:49:23,110: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-29 11:49:23,903: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-29 11:49:23,948: INFO/job_scheduler] run_next: 0 runnable jobs of (2 pending, 1 tasked)
[2017-06-29 11:49:30,048: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:49:30,048: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:49:30,048: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:49:30,048: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:49:30,048: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 52ecdd2a-6530-42d4-b071-34f9e08cb034
[2017-06-29 11:49:30,048: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 52ecdd2a-6530-42d4-b071-34f9e08cb034
[2017-06-29 11:49:30,093: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 52ecdd2a-6530-42d4-b071-34f9e08cb034
[2017-06-29 11:49:30,093: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 52ecdd2a-6530-42d4-b071-34f9e08cb034
[2017-06-29 11:49:30,093: INFO/job_scheduler] Job 67 finished 3 steps successfully
[2017-06-29 11:49:30,147: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->stopped (intentional=True) 140668098585296
[2017-06-29 11:49:30,155: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-06-29 11:49:30,166: INFO/job] Job 67: StateChangeJob complete, setting state stopped on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-29 11:49:30,167: INFO/job] Job 67: not_deleted=True
[2017-06-29 11:49:30,169: INFO/job_scheduler] Job 67: complete_job: Updating cache
[2017-06-29 11:49:30,169: INFO/job_scheduler] Job 67: locked_item 140668098585296 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-29 11:49:30,179: INFO/job_scheduler] Job 67 completing (errored=False, cancelled=False)
[2017-06-29 11:49:30,201: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 0 tasked)
[2017-06-29 11:49:30,231: INFO/job_scheduler] Job 68: RunJobThread.run
[2017-06-29 11:49:30,232: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_pacemaker {}
[2017-06-29 11:49:30,293: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 652c73b7-c800-4938-a356-20d64daccf5b
[2017-06-29 11:49:30,293: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 652c73b7-c800-4938-a356-20d64daccf5b
[2017-06-29 11:49:33,125: INFO/job_scheduler] _notify: Buffering update to lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration because of locks
[2017-06-29 11:49:33,125: INFO/job_scheduler]   Job 68 readlock on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 11:49:33,125: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-06-29 11:49:37,403: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:49:37,403: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:49:37,403: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:49:37,403: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:49:37,403: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 652c73b7-c800-4938-a356-20d64daccf5b
[2017-06-29 11:49:37,403: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 652c73b7-c800-4938-a356-20d64daccf5b
[2017-06-29 11:49:37,413: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 652c73b7-c800-4938-a356-20d64daccf5b
[2017-06-29 11:49:37,413: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 652c73b7-c800-4938-a356-20d64daccf5b
[2017-06-29 11:49:37,413: INFO/job_scheduler] Job 68 finished 1 steps successfully
[2017-06-29 11:49:37,504: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->started (intentional=True) 140668098585296
[2017-06-29 11:49:37,545: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 30
[2017-06-29 11:49:37,559: INFO/job] Job 68: StateChangeJob complete, setting state started on lotus-32vm7.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-29 11:49:37,559: INFO/job] Job 68: not_deleted=True
[2017-06-29 11:49:37,559: INFO/job_scheduler] Job 68: complete_job: Updating cache
[2017-06-29 11:49:37,559: INFO/job_scheduler] Job 68: locked_item 140668098585296 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-29 11:49:37,562: INFO/job_scheduler] Job 68 completing (errored=False, cancelled=False)
[2017-06-29 11:49:37,583: INFO/job_scheduler] Replaying 2 buffered notifications for Corosync2Configuration-4
[2017-06-29 11:49:37,583: INFO/job_scheduler] _notify: Updating .mcast_port of item 4 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from None to 42897
[2017-06-29 11:49:37,591: INFO/job_scheduler] _notify: Set mcast_port=42897 on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-4) and saved
[2017-06-29 11:49:37,614: INFO/job_scheduler] _notify: Updating .network_interfaces of item 4 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from [] to [u'eth0', u'eth1']
[2017-06-29 11:49:37,644: INFO/job_scheduler] _notify: Set network_interfaces=[u'eth0', u'eth1'] on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-4) and saved
[2017-06-29 11:49:37,663: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 4 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-06-29 11:49:37,668: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-4) and saved
[2017-06-29 11:49:37,682: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-29 11:49:37,718: INFO/job_scheduler] Job 69: RunJobThread.run
[2017-06-29 11:49:37,720: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com initialise_block_device_drivers {}
[2017-06-29 11:49:37,734: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a70d95c9-3ce4-4da5-a407-cb443ce772b0
[2017-06-29 11:49:37,734: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a70d95c9-3ce4-4da5-a407-cb443ce772b0
[2017-06-29 11:50:04,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:50:04,123: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:50:04,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:50:04,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 11:50:04,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a70d95c9-3ce4-4da5-a407-cb443ce772b0
[2017-06-29 11:50:04,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a70d95c9-3ce4-4da5-a407-cb443ce772b0
[2017-06-29 11:50:04,130: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a70d95c9-3ce4-4da5-a407-cb443ce772b0
[2017-06-29 11:50:04,130: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a70d95c9-3ce4-4da5-a407-cb443ce772b0
[2017-06-29 11:50:04,132: INFO/job_scheduler] Job 69 finished 1 steps successfully
[2017-06-29 11:50:04,563: INFO/job] StatefulObject.set_state lotus-32vm7.lotus.hpdd.lab.intel.com unconfigured->managed (intentional=True) 140668098748304
[2017-06-29 11:50:04,568: INFO/job] Job 69: StateChangeJob complete, setting state managed on lotus-32vm7.lotus.hpdd.lab.intel.com
[2017-06-29 11:50:04,568: INFO/job] Job 69: not_deleted=True
[2017-06-29 11:50:04,568: INFO/job_scheduler] Job 69: complete_job: Updating cache
[2017-06-29 11:50:04,568: INFO/job_scheduler] Job 69: locked_item 140668098748304 <class 'chroma_core.models.host.ManagedHost'> True True
[2017-06-29 11:50:04,571: INFO/job_scheduler] Job 69 completing (errored=False, cancelled=False)
[2017-06-29 11:50:04,629: INFO/job_scheduler] Replaying 6 buffered notifications for ManagedHost-4
[2017-06-29 11:50:04,637: INFO/job_scheduler] _notify: Updating .immutable_state of item 4 (lotus-32vm7.lotus.hpdd.lab.intel.com) from True to False
[2017-06-29 11:50:04,642: INFO/job_scheduler] _notify: Set immutable_state=False on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-4) and saved
[2017-06-29 11:50:04,652: INFO/job_scheduler] _notify: Updating .boot_time of item 4 (lotus-32vm7.lotus.hpdd.lab.intel.com) from 2017-04-25 07:09:31+00:00 to 2017-06-29 04:44:14+00:00
[2017-06-29 11:50:04,665: INFO/job_scheduler] _notify: Set boot_time=2017-06-29 04:44:14+00:00 on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-4) and saved
[2017-06-29 11:50:04,688: INFO/job_scheduler] _notify: Updating .properties of item 4 (lotus-32vm7.lotus.hpdd.lab.intel.com) from {"python_patchlevel": 5, "kernel_version": "3.10.0-514.10.2.el7_lustre.x86_64", "zfs_installed": false, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} to {"python_patchlevel": 5, "kernel_version": "3.10.0-514.21.1.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"}
[2017-06-29 11:50:04,706: INFO/job_scheduler] _notify: Set properties={"python_patchlevel": 5, "kernel_version": "3.10.0-514.21.1.el7_lustre.x86_64", "zfs_installed": true, "distro_version": 7.3, "python_version_major_minor": 2.7, "distro": "CentOS Linux"} on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-4) and saved
[2017-06-29 11:50:04,742: INFO/job_scheduler] _notify: Updating .ha_cluster_peers of item 4 (lotus-32vm7.lotus.hpdd.lab.intel.com) from <django.db.models.fields.related.ManyRelatedManager object at 0x7fefd80dea50> to [5]
[2017-06-29 11:50:04,764: INFO/job_scheduler] _notify: Set ha_cluster_peers=[5] on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-4) and saved
[2017-06-29 11:50:04,947: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 11:53:43,859: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 11:53:43,905: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 11:58:15,573: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 11:58:15,593: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 11:59:12,119: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 11:59:12,144: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:03:55,681: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:03:55,704: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:08:27,335: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:08:27,350: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:09:23,722: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:09:23,748: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:14:07,753: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:14:07,768: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:15:03,163: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 12:15:03,164: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-29 12:15:03,187: INFO/command_plan] add_jobs: created Job 84 (Trigger plugin poll for linux plugins)
[2017-06-29 12:15:03,198: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-29 12:15:03,261: INFO/job_scheduler] Job 84: RunJobThread.run
[2017-06-29 12:15:03,302: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:15:03,335: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1a08f4d9-6823-44e1-87ea-3a8ea94f11af
[2017-06-29 12:15:03,335: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1a08f4d9-6823-44e1-87ea-3a8ea94f11af
[2017-06-29 12:15:03,343: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:15:03,364: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5c0d3d36-3a79-4749-aa1e-2e697a7fa2c3
[2017-06-29 12:15:03,364: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5c0d3d36-3a79-4749-aa1e-2e697a7fa2c3
[2017-06-29 12:15:03,366: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:15:03,391: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 103cb5a2-02b0-4ea0-8716-01d7d3381344
[2017-06-29 12:15:03,391: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 103cb5a2-02b0-4ea0-8716-01d7d3381344
[2017-06-29 12:15:03,401: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:15:03,401: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:15:03,401: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:15:03,401: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:15:03,402: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1a08f4d9-6823-44e1-87ea-3a8ea94f11af
[2017-06-29 12:15:03,402: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1a08f4d9-6823-44e1-87ea-3a8ea94f11af
[2017-06-29 12:15:03,424: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1a08f4d9-6823-44e1-87ea-3a8ea94f11af
[2017-06-29 12:15:03,424: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1a08f4d9-6823-44e1-87ea-3a8ea94f11af
[2017-06-29 12:15:03,424: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:15:03,433: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:15:03,433: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:15:03,433: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:15:03,433: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:15:03,434: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 103cb5a2-02b0-4ea0-8716-01d7d3381344
[2017-06-29 12:15:03,434: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 103cb5a2-02b0-4ea0-8716-01d7d3381344
[2017-06-29 12:15:03,435: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:15:03,435: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:15:03,435: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:15:03,435: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:15:03,435: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5c0d3d36-3a79-4749-aa1e-2e697a7fa2c3
[2017-06-29 12:15:03,435: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5c0d3d36-3a79-4749-aa1e-2e697a7fa2c3
[2017-06-29 12:15:03,450: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5c0d3d36-3a79-4749-aa1e-2e697a7fa2c3
[2017-06-29 12:15:03,450: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5c0d3d36-3a79-4749-aa1e-2e697a7fa2c3
[2017-06-29 12:15:03,450: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:15:03,464: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 103cb5a2-02b0-4ea0-8716-01d7d3381344
[2017-06-29 12:15:03,464: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 103cb5a2-02b0-4ea0-8716-01d7d3381344
[2017-06-29 12:15:03,464: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:15:03,464: INFO/job_scheduler] Job 84 finished 1 steps successfully
[2017-06-29 12:15:03,537: INFO/job_scheduler] Job 84: complete_job: Updating cache
[2017-06-29 12:15:03,537: INFO/job_scheduler] Job 84 completing (errored=False, cancelled=False)
[2017-06-29 12:15:03,590: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:15:40,469: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 12:15:40,469: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-29 12:15:40,494: INFO/command_plan] add_jobs: created Job 85 (Trigger plugin poll for linux plugins)
[2017-06-29 12:15:40,517: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-29 12:15:40,549: INFO/job_scheduler] Job 85: RunJobThread.run
[2017-06-29 12:15:40,564: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:15:40,613: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:15:40,616: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 05e714b0-4902-478c-99f8-67c8bfc269ca
[2017-06-29 12:15:40,616: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 05e714b0-4902-478c-99f8-67c8bfc269ca
[2017-06-29 12:15:40,634: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:15:40,651: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0d456e56-f91a-42d1-a5c6-5d24d647c684
[2017-06-29 12:15:40,651: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0d456e56-f91a-42d1-a5c6-5d24d647c684
[2017-06-29 12:15:40,693: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 253eaf52-1c5e-4db5-8031-d1ec57e904d7
[2017-06-29 12:15:40,693: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 253eaf52-1c5e-4db5-8031-d1ec57e904d7
[2017-06-29 12:15:40,731: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:15:40,731: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:15:40,731: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:15:40,731: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:15:40,731: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 05e714b0-4902-478c-99f8-67c8bfc269ca
[2017-06-29 12:15:40,731: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 05e714b0-4902-478c-99f8-67c8bfc269ca
[2017-06-29 12:15:40,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:15:40,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:15:40,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:15:40,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:15:40,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0d456e56-f91a-42d1-a5c6-5d24d647c684
[2017-06-29 12:15:40,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0d456e56-f91a-42d1-a5c6-5d24d647c684
[2017-06-29 12:15:40,734: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:15:40,734: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:15:40,734: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:15:40,734: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:15:40,735: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 253eaf52-1c5e-4db5-8031-d1ec57e904d7
[2017-06-29 12:15:40,735: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 253eaf52-1c5e-4db5-8031-d1ec57e904d7
[2017-06-29 12:15:40,744: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0d456e56-f91a-42d1-a5c6-5d24d647c684
[2017-06-29 12:15:40,744: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0d456e56-f91a-42d1-a5c6-5d24d647c684
[2017-06-29 12:15:40,744: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:15:40,745: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 05e714b0-4902-478c-99f8-67c8bfc269ca
[2017-06-29 12:15:40,745: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 05e714b0-4902-478c-99f8-67c8bfc269ca
[2017-06-29 12:15:40,745: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:15:40,762: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 253eaf52-1c5e-4db5-8031-d1ec57e904d7
[2017-06-29 12:15:40,762: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 253eaf52-1c5e-4db5-8031-d1ec57e904d7
[2017-06-29 12:15:40,762: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:15:40,762: INFO/job_scheduler] Job 85 finished 1 steps successfully
[2017-06-29 12:15:40,803: INFO/job_scheduler] Job 85: complete_job: Updating cache
[2017-06-29 12:15:40,803: INFO/job_scheduler] Job 85 completing (errored=False, cancelled=False)
[2017-06-29 12:15:40,856: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:15:45,772: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 12:15:45,774: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-29 12:15:45,785: INFO/command_plan] add_jobs: created Job 86 (Trigger plugin poll for linux plugins)
[2017-06-29 12:15:45,827: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-29 12:15:45,840: INFO/job_scheduler] Job 86: RunJobThread.run
[2017-06-29 12:15:45,871: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:15:45,882: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:15:45,882: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:15:45,887: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8a1d5534-ac8d-48fc-9d03-e22eb33109ab
[2017-06-29 12:15:45,887: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8a1d5534-ac8d-48fc-9d03-e22eb33109ab
[2017-06-29 12:15:45,901: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc daf902e8-e857-4836-8ba8-64a74465a0f3
[2017-06-29 12:15:45,901: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc daf902e8-e857-4836-8ba8-64a74465a0f3
[2017-06-29 12:15:45,921: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c0d95736-3159-422b-ac2c-b0538445d63d
[2017-06-29 12:15:45,921: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c0d95736-3159-422b-ac2c-b0538445d63d
[2017-06-29 12:15:45,971: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:15:45,971: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:15:45,971: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:15:45,971: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:15:45,971: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8a1d5534-ac8d-48fc-9d03-e22eb33109ab
[2017-06-29 12:15:45,971: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8a1d5534-ac8d-48fc-9d03-e22eb33109ab
[2017-06-29 12:15:45,977: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:15:45,977: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:15:45,977: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:15:45,977: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:15:45,977: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c0d95736-3159-422b-ac2c-b0538445d63d
[2017-06-29 12:15:45,977: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c0d95736-3159-422b-ac2c-b0538445d63d
[2017-06-29 12:15:45,982: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:15:45,982: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:15:45,982: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:15:45,982: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:15:45,982: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc daf902e8-e857-4836-8ba8-64a74465a0f3
[2017-06-29 12:15:45,982: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc daf902e8-e857-4836-8ba8-64a74465a0f3
[2017-06-29 12:15:45,985: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc daf902e8-e857-4836-8ba8-64a74465a0f3
[2017-06-29 12:15:45,985: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc daf902e8-e857-4836-8ba8-64a74465a0f3
[2017-06-29 12:15:45,986: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:15:45,987: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c0d95736-3159-422b-ac2c-b0538445d63d
[2017-06-29 12:15:45,987: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c0d95736-3159-422b-ac2c-b0538445d63d
[2017-06-29 12:15:45,987: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:15:46,018: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8a1d5534-ac8d-48fc-9d03-e22eb33109ab
[2017-06-29 12:15:46,018: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8a1d5534-ac8d-48fc-9d03-e22eb33109ab
[2017-06-29 12:15:46,018: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:15:46,018: INFO/job_scheduler] Job 86 finished 1 steps successfully
[2017-06-29 12:15:46,035: INFO/job_scheduler] Job 86: complete_job: Updating cache
[2017-06-29 12:15:46,035: INFO/job_scheduler] Job 86 completing (errored=False, cancelled=False)
[2017-06-29 12:15:46,095: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:15:46,615: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 12:15:46,615: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-29 12:15:46,635: INFO/command_plan] add_jobs: created Job 87 (Trigger plugin poll for linux plugins)
[2017-06-29 12:15:46,668: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-29 12:15:46,690: INFO/job_scheduler] Job 87: RunJobThread.run
[2017-06-29 12:15:46,713: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:15:46,739: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 353da698-0ba7-4c8f-b515-8427105c619d
[2017-06-29 12:15:46,739: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 353da698-0ba7-4c8f-b515-8427105c619d
[2017-06-29 12:15:46,741: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:15:46,767: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 162ca5cd-6527-4cef-8591-53416783d275
[2017-06-29 12:15:46,767: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 162ca5cd-6527-4cef-8591-53416783d275
[2017-06-29 12:15:46,768: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:15:46,787: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6c9199bc-2e40-48e2-9ee2-bcd70694df12
[2017-06-29 12:15:46,787: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6c9199bc-2e40-48e2-9ee2-bcd70694df12
[2017-06-29 12:15:46,848: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:15:46,848: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:15:46,848: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:15:46,848: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:15:46,848: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 353da698-0ba7-4c8f-b515-8427105c619d
[2017-06-29 12:15:46,848: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 353da698-0ba7-4c8f-b515-8427105c619d
[2017-06-29 12:15:46,851: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:15:46,851: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:15:46,851: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:15:46,851: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:15:46,851: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 162ca5cd-6527-4cef-8591-53416783d275
[2017-06-29 12:15:46,851: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 162ca5cd-6527-4cef-8591-53416783d275
[2017-06-29 12:15:46,863: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:15:46,863: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:15:46,864: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:15:46,864: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:15:46,864: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6c9199bc-2e40-48e2-9ee2-bcd70694df12
[2017-06-29 12:15:46,864: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6c9199bc-2e40-48e2-9ee2-bcd70694df12
[2017-06-29 12:15:46,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 353da698-0ba7-4c8f-b515-8427105c619d
[2017-06-29 12:15:46,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 353da698-0ba7-4c8f-b515-8427105c619d
[2017-06-29 12:15:46,868: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:15:46,890: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 162ca5cd-6527-4cef-8591-53416783d275
[2017-06-29 12:15:46,890: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 162ca5cd-6527-4cef-8591-53416783d275
[2017-06-29 12:15:46,890: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:15:46,902: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6c9199bc-2e40-48e2-9ee2-bcd70694df12
[2017-06-29 12:15:46,902: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6c9199bc-2e40-48e2-9ee2-bcd70694df12
[2017-06-29 12:15:46,902: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:15:46,903: INFO/job_scheduler] Job 87 finished 1 steps successfully
[2017-06-29 12:15:46,943: INFO/job_scheduler] Job 87: complete_job: Updating cache
[2017-06-29 12:15:46,943: INFO/job_scheduler] Job 87 completing (errored=False, cancelled=False)
[2017-06-29 12:15:47,018: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:16:36,753: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 12:16:36,754: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-29 12:16:36,767: INFO/command_plan] add_jobs: created Job 88 (Trigger plugin poll for linux plugins)
[2017-06-29 12:16:36,800: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-29 12:16:36,829: INFO/job_scheduler] Job 88: RunJobThread.run
[2017-06-29 12:16:36,839: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:16:36,862: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:16:36,875: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:16:36,879: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c17f458d-4505-4e04-894b-65f912a5e08d
[2017-06-29 12:16:36,879: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c17f458d-4505-4e04-894b-65f912a5e08d
[2017-06-29 12:16:36,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 41f617e8-3496-406e-b592-eba83e638d0e
[2017-06-29 12:16:36,898: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 41f617e8-3496-406e-b592-eba83e638d0e
[2017-06-29 12:16:36,913: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c81dcfd6-c3e5-4052-8c5a-fd1c9a446554
[2017-06-29 12:16:36,913: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c81dcfd6-c3e5-4052-8c5a-fd1c9a446554
[2017-06-29 12:16:36,996: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:16:36,996: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:16:36,996: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:16:36,996: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:16:36,996: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c17f458d-4505-4e04-894b-65f912a5e08d
[2017-06-29 12:16:36,996: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c17f458d-4505-4e04-894b-65f912a5e08d
[2017-06-29 12:16:37,000: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:16:37,000: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:16:37,000: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:16:37,000: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:16:37,000: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 41f617e8-3496-406e-b592-eba83e638d0e
[2017-06-29 12:16:37,000: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 41f617e8-3496-406e-b592-eba83e638d0e
[2017-06-29 12:16:37,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:16:37,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:16:37,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:16:37,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:16:37,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c81dcfd6-c3e5-4052-8c5a-fd1c9a446554
[2017-06-29 12:16:37,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c81dcfd6-c3e5-4052-8c5a-fd1c9a446554
[2017-06-29 12:16:37,006: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c17f458d-4505-4e04-894b-65f912a5e08d
[2017-06-29 12:16:37,006: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c17f458d-4505-4e04-894b-65f912a5e08d
[2017-06-29 12:16:37,007: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:16:37,036: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 41f617e8-3496-406e-b592-eba83e638d0e
[2017-06-29 12:16:37,036: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 41f617e8-3496-406e-b592-eba83e638d0e
[2017-06-29 12:16:37,037: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:16:37,052: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c81dcfd6-c3e5-4052-8c5a-fd1c9a446554
[2017-06-29 12:16:37,052: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c81dcfd6-c3e5-4052-8c5a-fd1c9a446554
[2017-06-29 12:16:37,052: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:16:37,053: INFO/job_scheduler] Job 88 finished 1 steps successfully
[2017-06-29 12:16:37,091: INFO/job_scheduler] Job 88: complete_job: Updating cache
[2017-06-29 12:16:37,091: INFO/job_scheduler] Job 88 completing (errored=False, cancelled=False)
[2017-06-29 12:16:37,133: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:16:41,136: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 12:16:41,136: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-29 12:16:41,165: INFO/command_plan] add_jobs: created Job 89 (Trigger plugin poll for linux plugins)
[2017-06-29 12:16:41,176: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-29 12:16:41,201: INFO/job_scheduler] Job 89: RunJobThread.run
[2017-06-29 12:16:41,212: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:16:41,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1e57625a-3f3f-4f89-b286-25e09250dd39
[2017-06-29 12:16:41,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1e57625a-3f3f-4f89-b286-25e09250dd39
[2017-06-29 12:16:41,273: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:16:41,287: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:16:41,294: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 579585ff-9aa1-4ce9-b807-6488498a2ffb
[2017-06-29 12:16:41,294: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 579585ff-9aa1-4ce9-b807-6488498a2ffb
[2017-06-29 12:16:41,308: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 467f8cca-3a74-4c1b-bb9f-48c0a0c9c7c3
[2017-06-29 12:16:41,308: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 467f8cca-3a74-4c1b-bb9f-48c0a0c9c7c3
[2017-06-29 12:16:41,377: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:16:41,377: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:16:41,377: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:16:41,377: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:16:41,378: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1e57625a-3f3f-4f89-b286-25e09250dd39
[2017-06-29 12:16:41,378: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1e57625a-3f3f-4f89-b286-25e09250dd39
[2017-06-29 12:16:41,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:16:41,380: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:16:41,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:16:41,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:16:41,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 467f8cca-3a74-4c1b-bb9f-48c0a0c9c7c3
[2017-06-29 12:16:41,381: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 467f8cca-3a74-4c1b-bb9f-48c0a0c9c7c3
[2017-06-29 12:16:41,383: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:16:41,383: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:16:41,383: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:16:41,383: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:16:41,383: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 579585ff-9aa1-4ce9-b807-6488498a2ffb
[2017-06-29 12:16:41,383: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 579585ff-9aa1-4ce9-b807-6488498a2ffb
[2017-06-29 12:16:41,389: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1e57625a-3f3f-4f89-b286-25e09250dd39
[2017-06-29 12:16:41,389: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1e57625a-3f3f-4f89-b286-25e09250dd39
[2017-06-29 12:16:41,390: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:16:41,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 579585ff-9aa1-4ce9-b807-6488498a2ffb
[2017-06-29 12:16:41,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 579585ff-9aa1-4ce9-b807-6488498a2ffb
[2017-06-29 12:16:41,421: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:16:41,424: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 467f8cca-3a74-4c1b-bb9f-48c0a0c9c7c3
[2017-06-29 12:16:41,424: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 467f8cca-3a74-4c1b-bb9f-48c0a0c9c7c3
[2017-06-29 12:16:41,424: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:16:41,425: INFO/job_scheduler] Job 89 finished 1 steps successfully
[2017-06-29 12:16:41,449: INFO/job_scheduler] Job 89: complete_job: Updating cache
[2017-06-29 12:16:41,449: INFO/job_scheduler] Job 89 completing (errored=False, cancelled=False)
[2017-06-29 12:16:41,502: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:16:48,419: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 12:16:48,425: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-29 12:16:48,555: INFO/command_plan] add_jobs: created Job 90 (Trigger plugin poll for linux plugins)
[2017-06-29 12:16:48,619: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-29 12:16:48,642: INFO/job_scheduler] Job 90: RunJobThread.run
[2017-06-29 12:16:48,665: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:16:48,675: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:16:48,679: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4e477466-0c64-4027-aa1c-fa9f17357f13
[2017-06-29 12:16:48,679: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4e477466-0c64-4027-aa1c-fa9f17357f13
[2017-06-29 12:16:48,693: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e5a121fd-078f-40dd-a236-7a7b0af1ef62
[2017-06-29 12:16:48,693: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e5a121fd-078f-40dd-a236-7a7b0af1ef62
[2017-06-29 12:16:48,694: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:16:48,709: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bd25a97c-394c-4cac-b14c-bed146df8962
[2017-06-29 12:16:48,709: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bd25a97c-394c-4cac-b14c-bed146df8962
[2017-06-29 12:16:48,746: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:16:48,746: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:16:48,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:16:48,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:16:48,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4e477466-0c64-4027-aa1c-fa9f17357f13
[2017-06-29 12:16:48,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4e477466-0c64-4027-aa1c-fa9f17357f13
[2017-06-29 12:16:48,760: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4e477466-0c64-4027-aa1c-fa9f17357f13
[2017-06-29 12:16:48,760: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4e477466-0c64-4027-aa1c-fa9f17357f13
[2017-06-29 12:16:48,760: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:16:48,761: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:16:48,761: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:16:48,761: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:16:48,761: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:16:48,761: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e5a121fd-078f-40dd-a236-7a7b0af1ef62
[2017-06-29 12:16:48,761: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e5a121fd-078f-40dd-a236-7a7b0af1ef62
[2017-06-29 12:16:48,768: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e5a121fd-078f-40dd-a236-7a7b0af1ef62
[2017-06-29 12:16:48,768: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e5a121fd-078f-40dd-a236-7a7b0af1ef62
[2017-06-29 12:16:48,768: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:16:48,771: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:16:48,771: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:16:48,772: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:16:48,772: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:16:48,772: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bd25a97c-394c-4cac-b14c-bed146df8962
[2017-06-29 12:16:48,772: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bd25a97c-394c-4cac-b14c-bed146df8962
[2017-06-29 12:16:48,777: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bd25a97c-394c-4cac-b14c-bed146df8962
[2017-06-29 12:16:48,777: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bd25a97c-394c-4cac-b14c-bed146df8962
[2017-06-29 12:16:48,777: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:16:48,778: INFO/job_scheduler] Job 90 finished 1 steps successfully
[2017-06-29 12:16:48,797: INFO/job_scheduler] Job 90: complete_job: Updating cache
[2017-06-29 12:16:48,797: INFO/job_scheduler] Job 90 completing (errored=False, cancelled=False)
[2017-06-29 12:16:48,854: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:17:01,334: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 12:17:01,336: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-29 12:17:01,361: INFO/command_plan] add_jobs: created Job 91 (Trigger plugin poll for linux plugins)
[2017-06-29 12:17:01,374: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-29 12:17:01,409: INFO/job_scheduler] Job 91: RunJobThread.run
[2017-06-29 12:17:01,450: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:17:01,475: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:17:01,478: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:17:01,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 51ef1e3a-ef7b-4ab4-ab5e-cc214026af67
[2017-06-29 12:17:01,494: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 51ef1e3a-ef7b-4ab4-ab5e-cc214026af67
[2017-06-29 12:17:01,507: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b53b7855-4d45-46bc-b0de-d4eea7a547c6
[2017-06-29 12:17:01,507: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b53b7855-4d45-46bc-b0de-d4eea7a547c6
[2017-06-29 12:17:01,520: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2f708f6c-1d3b-464a-b2e2-2cd3f68bda49
[2017-06-29 12:17:01,520: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2f708f6c-1d3b-464a-b2e2-2cd3f68bda49
[2017-06-29 12:17:01,574: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:17:01,574: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:17:01,574: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:17:01,574: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:17:01,575: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 51ef1e3a-ef7b-4ab4-ab5e-cc214026af67
[2017-06-29 12:17:01,575: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 51ef1e3a-ef7b-4ab4-ab5e-cc214026af67
[2017-06-29 12:17:01,577: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:17:01,577: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:17:01,577: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:17:01,577: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:17:01,578: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b53b7855-4d45-46bc-b0de-d4eea7a547c6
[2017-06-29 12:17:01,578: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b53b7855-4d45-46bc-b0de-d4eea7a547c6
[2017-06-29 12:17:01,578: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 51ef1e3a-ef7b-4ab4-ab5e-cc214026af67
[2017-06-29 12:17:01,578: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 51ef1e3a-ef7b-4ab4-ab5e-cc214026af67
[2017-06-29 12:17:01,578: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:17:01,579: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b53b7855-4d45-46bc-b0de-d4eea7a547c6
[2017-06-29 12:17:01,579: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b53b7855-4d45-46bc-b0de-d4eea7a547c6
[2017-06-29 12:17:01,580: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:17:01,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:17:01,612: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:17:01,613: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:17:01,613: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:17:01,613: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2f708f6c-1d3b-464a-b2e2-2cd3f68bda49
[2017-06-29 12:17:01,613: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2f708f6c-1d3b-464a-b2e2-2cd3f68bda49
[2017-06-29 12:17:01,636: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2f708f6c-1d3b-464a-b2e2-2cd3f68bda49
[2017-06-29 12:17:01,636: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2f708f6c-1d3b-464a-b2e2-2cd3f68bda49
[2017-06-29 12:17:01,636: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:17:01,636: INFO/job_scheduler] Job 91 finished 1 steps successfully
[2017-06-29 12:17:01,669: INFO/job_scheduler] Job 91: complete_job: Updating cache
[2017-06-29 12:17:01,669: INFO/job_scheduler] Job 91 completing (errored=False, cancelled=False)
[2017-06-29 12:17:01,711: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:17:08,269: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 12:17:08,269: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-29 12:17:08,285: INFO/command_plan] add_jobs: created Job 92 (Trigger plugin poll for linux plugins)
[2017-06-29 12:17:08,337: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-29 12:17:08,377: INFO/job_scheduler] Job 92: RunJobThread.run
[2017-06-29 12:17:08,391: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:17:08,417: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:17:08,420: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d2e254b1-6d4f-47cc-8b0c-41f1480763e1
[2017-06-29 12:17:08,420: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d2e254b1-6d4f-47cc-8b0c-41f1480763e1
[2017-06-29 12:17:08,448: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6dc7c40f-c7b9-4b50-9db7-5ec158b90e21
[2017-06-29 12:17:08,448: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 6dc7c40f-c7b9-4b50-9db7-5ec158b90e21
[2017-06-29 12:17:08,449: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:17:08,480: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c403c0f9-e47c-41d2-9f5c-97b5a69c08ee
[2017-06-29 12:17:08,480: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c403c0f9-e47c-41d2-9f5c-97b5a69c08ee
[2017-06-29 12:17:08,483: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:17:08,483: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:17:08,483: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:17:08,483: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:17:08,483: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d2e254b1-6d4f-47cc-8b0c-41f1480763e1
[2017-06-29 12:17:08,483: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d2e254b1-6d4f-47cc-8b0c-41f1480763e1
[2017-06-29 12:17:08,497: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d2e254b1-6d4f-47cc-8b0c-41f1480763e1
[2017-06-29 12:17:08,497: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d2e254b1-6d4f-47cc-8b0c-41f1480763e1
[2017-06-29 12:17:08,498: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:17:08,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:17:08,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:17:08,524: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:17:08,524: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:17:08,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c403c0f9-e47c-41d2-9f5c-97b5a69c08ee
[2017-06-29 12:17:08,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c403c0f9-e47c-41d2-9f5c-97b5a69c08ee
[2017-06-29 12:17:08,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:17:08,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:17:08,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:17:08,525: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:17:08,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6dc7c40f-c7b9-4b50-9db7-5ec158b90e21
[2017-06-29 12:17:08,526: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 6dc7c40f-c7b9-4b50-9db7-5ec158b90e21
[2017-06-29 12:17:08,548: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c403c0f9-e47c-41d2-9f5c-97b5a69c08ee
[2017-06-29 12:17:08,548: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c403c0f9-e47c-41d2-9f5c-97b5a69c08ee
[2017-06-29 12:17:08,548: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:17:08,575: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6dc7c40f-c7b9-4b50-9db7-5ec158b90e21
[2017-06-29 12:17:08,575: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 6dc7c40f-c7b9-4b50-9db7-5ec158b90e21
[2017-06-29 12:17:08,575: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:17:08,575: INFO/job_scheduler] Job 92 finished 1 steps successfully
[2017-06-29 12:17:08,603: INFO/job_scheduler] Job 92: complete_job: Updating cache
[2017-06-29 12:17:08,603: INFO/job_scheduler] Job 92 completing (errored=False, cancelled=False)
[2017-06-29 12:17:08,694: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:17:12,740: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 12:17:12,742: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-29 12:17:12,768: INFO/command_plan] add_jobs: created Job 93 (Trigger plugin poll for linux plugins)
[2017-06-29 12:17:12,812: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-29 12:17:12,822: INFO/job_scheduler] Job 93: RunJobThread.run
[2017-06-29 12:17:12,847: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:17:12,855: INFO/job] invoke_agent on agent lotus-32vm6.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:17:12,855: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:17:12,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8b8d14d1-3c01-489d-8cb7-05e1eefe2cc7
[2017-06-29 12:17:12,868: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8b8d14d1-3c01-489d-8cb7-05e1eefe2cc7
[2017-06-29 12:17:12,896: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c06a20a7-9af8-4d39-b570-49926cdf0c18
[2017-06-29 12:17:12,896: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc c06a20a7-9af8-4d39-b570-49926cdf0c18
[2017-06-29 12:17:12,911: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4ef21af3-bd12-4803-b863-4a92c5e1b5b4
[2017-06-29 12:17:12,911: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4ef21af3-bd12-4803-b863-4a92c5e1b5b4
[2017-06-29 12:17:13,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:17:13,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:17:13,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:17:13,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm6.lotus.hpdd.lab.intel.com/e17f77e7-2dd3-40d8-a5c0-e045090b0ec2
[2017-06-29 12:17:13,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c06a20a7-9af8-4d39-b570-49926cdf0c18
[2017-06-29 12:17:13,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc c06a20a7-9af8-4d39-b570-49926cdf0c18
[2017-06-29 12:17:13,012: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:17:13,012: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:17:13,012: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:17:13,012: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:17:13,012: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8b8d14d1-3c01-489d-8cb7-05e1eefe2cc7
[2017-06-29 12:17:13,012: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8b8d14d1-3c01-489d-8cb7-05e1eefe2cc7
[2017-06-29 12:17:13,013: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:17:13,013: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:17:13,013: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:17:13,013: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:17:13,013: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4ef21af3-bd12-4803-b863-4a92c5e1b5b4
[2017-06-29 12:17:13,013: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4ef21af3-bd12-4803-b863-4a92c5e1b5b4
[2017-06-29 12:17:13,013: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8b8d14d1-3c01-489d-8cb7-05e1eefe2cc7
[2017-06-29 12:17:13,013: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8b8d14d1-3c01-489d-8cb7-05e1eefe2cc7
[2017-06-29 12:17:13,013: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:17:13,034: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c06a20a7-9af8-4d39-b570-49926cdf0c18
[2017-06-29 12:17:13,034: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc c06a20a7-9af8-4d39-b570-49926cdf0c18
[2017-06-29 12:17:13,035: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:17:13,049: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4ef21af3-bd12-4803-b863-4a92c5e1b5b4
[2017-06-29 12:17:13,049: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4ef21af3-bd12-4803-b863-4a92c5e1b5b4
[2017-06-29 12:17:13,049: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:17:13,049: INFO/job_scheduler] Job 93 finished 1 steps successfully
[2017-06-29 12:17:13,092: INFO/job_scheduler] Job 93: complete_job: Updating cache
[2017-06-29 12:17:13,092: INFO/job_scheduler] Job 93 completing (errored=False, cancelled=False)
[2017-06-29 12:17:13,143: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:17:19,224: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 12:17:19,225: INFO/command_plan] add_jobs: done checking dependencies
[2017-06-29 12:17:19,241: INFO/command_plan] add_jobs: created Job 94 (Trigger plugin poll for linux plugins)
[2017-06-29 12:17:19,266: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-29 12:17:19,290: INFO/job_scheduler] Job 94: RunJobThread.run
[2017-06-29 12:17:19,304: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:17:19,319: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 89ae5db5-126e-4402-8382-8d06217652af
[2017-06-29 12:17:19,319: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 89ae5db5-126e-4402-8382-8d06217652af
[2017-06-29 12:17:19,320: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:17:19,338: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7affaae6-7299-4a60-8984-bd0d398b6129
[2017-06-29 12:17:19,338: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7affaae6-7299-4a60-8984-bd0d398b6129
[2017-06-29 12:17:19,343: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com trigger_plugin_update {'plugin_names': [u'linux']}
[2017-06-29 12:17:19,364: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b43f763a-2bdb-4045-b216-74f1120a3b24
[2017-06-29 12:17:19,364: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b43f763a-2bdb-4045-b216-74f1120a3b24
[2017-06-29 12:17:19,418: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:17:19,418: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:17:19,418: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:17:19,418: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:17:19,418: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 89ae5db5-126e-4402-8382-8d06217652af
[2017-06-29 12:17:19,418: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 89ae5db5-126e-4402-8382-8d06217652af
[2017-06-29 12:17:19,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:17:19,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:17:19,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:17:19,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/98006ad6-3d35-467b-a0eb-42c12500547a
[2017-06-29 12:17:19,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7affaae6-7299-4a60-8984-bd0d398b6129
[2017-06-29 12:17:19,421: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7affaae6-7299-4a60-8984-bd0d398b6129
[2017-06-29 12:17:19,441: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:17:19,441: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:17:19,442: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:17:19,442: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/5082dfbe-1df4-417c-b301-99e890afd857
[2017-06-29 12:17:19,442: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b43f763a-2bdb-4045-b216-74f1120a3b24
[2017-06-29 12:17:19,442: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b43f763a-2bdb-4045-b216-74f1120a3b24
[2017-06-29 12:17:19,452: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 89ae5db5-126e-4402-8382-8d06217652af
[2017-06-29 12:17:19,452: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 89ae5db5-126e-4402-8382-8d06217652af
[2017-06-29 12:17:19,453: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:17:19,471: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7affaae6-7299-4a60-8984-bd0d398b6129
[2017-06-29 12:17:19,471: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7affaae6-7299-4a60-8984-bd0d398b6129
[2017-06-29 12:17:19,471: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:17:19,482: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b43f763a-2bdb-4045-b216-74f1120a3b24
[2017-06-29 12:17:19,482: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b43f763a-2bdb-4045-b216-74f1120a3b24
[2017-06-29 12:17:19,482: INFO/job] Invalid result None fixed up on called to trigger_plugin_update with args {'plugin_names': [u'linux']}
[2017-06-29 12:17:19,483: INFO/job_scheduler] Job 94 finished 1 steps successfully
[2017-06-29 12:17:19,499: INFO/job_scheduler] Job 94: complete_job: Updating cache
[2017-06-29 12:17:19,500: INFO/job_scheduler] Job 94 completing (errored=False, cancelled=False)
[2017-06-29 12:17:19,526: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:18:39,238: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:18:39,252: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:18:46,348: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 12:18:46,354: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-2 transitioning from managed to removed
[2017-06-29 12:18:46,444: INFO/command_plan] Created command 37 (Remove host lotus-32vm5.lotus.hpdd.lab.intel.com from configuration) with 9 jobs
[2017-06-29 12:18:46,470: INFO/job_scheduler] run_next: 4 runnable jobs of (9 pending, 0 tasked)
[2017-06-29 12:18:46,546: INFO/job_scheduler] Job 96: RunJobThread.run
[2017-06-29 12:18:46,547: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_lnet {}
[2017-06-29 12:18:46,558: INFO/job_scheduler] Job 97: RunJobThread.run
[2017-06-29 12:18:46,559: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_pacemaker {}
[2017-06-29 12:18:46,579: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d8e87b65-7b18-4116-b83e-c1c08188572c
[2017-06-29 12:18:46,579: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d8e87b65-7b18-4116-b83e-c1c08188572c
[2017-06-29 12:18:46,610: INFO/job_scheduler] Job 98: RunJobThread.run
[2017-06-29 12:18:46,610: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unconfigure_ntp {}
[2017-06-29 12:18:46,615: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f6c7a253-4c19-498f-9010-2bfe832bb3cc
[2017-06-29 12:18:46,615: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc f6c7a253-4c19-498f-9010-2bfe832bb3cc
[2017-06-29 12:18:46,632: INFO/job_scheduler] Job 95: RunJobThread.run
[2017-06-29 12:18:46,633: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unconfigure_rsyslog {}
[2017-06-29 12:18:46,641: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 684aeba2-921b-45a6-9eba-651cac65d92d
[2017-06-29 12:18:46,641: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 684aeba2-921b-45a6-9eba-651cac65d92d
[2017-06-29 12:18:46,667: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1774c138-d722-4821-8427-36d45aba363b
[2017-06-29 12:18:46,667: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1774c138-d722-4821-8427-36d45aba363b
[2017-06-29 12:18:52,697: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:18:52,697: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:18:52,697: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:18:52,697: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:18:52,697: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f6c7a253-4c19-498f-9010-2bfe832bb3cc
[2017-06-29 12:18:52,697: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc f6c7a253-4c19-498f-9010-2bfe832bb3cc
[2017-06-29 12:18:52,730: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f6c7a253-4c19-498f-9010-2bfe832bb3cc
[2017-06-29 12:18:52,730: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc f6c7a253-4c19-498f-9010-2bfe832bb3cc
[2017-06-29 12:18:52,731: INFO/job_scheduler] Job 97 finished 1 steps successfully
[2017-06-29 12:18:52,732: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:18:52,732: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:18:52,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:18:52,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:18:52,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 684aeba2-921b-45a6-9eba-651cac65d92d
[2017-06-29 12:18:52,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 684aeba2-921b-45a6-9eba-651cac65d92d
[2017-06-29 12:18:52,759: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 684aeba2-921b-45a6-9eba-651cac65d92d
[2017-06-29 12:18:52,759: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 684aeba2-921b-45a6-9eba-651cac65d92d
[2017-06-29 12:18:52,759: INFO/job_scheduler] Job 98 finished 1 steps successfully
[2017-06-29 12:18:52,772: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration started->stopped (intentional=True) 140667830955472
[2017-06-29 12:18:52,780: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-06-29 12:18:52,786: INFO/job] Job 97: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration
[2017-06-29 12:18:52,787: INFO/job] Job 97: not_deleted=True
[2017-06-29 12:18:52,789: INFO/job_scheduler] Job 97: complete_job: Updating cache
[2017-06-29 12:18:52,790: INFO/job_scheduler] Job 97: locked_item 140667830955472 <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> True True
[2017-06-29 12:18:52,797: INFO/job_scheduler] Job 97 completing (errored=False, cancelled=False)
[2017-06-29 12:18:52,822: INFO/job_scheduler] run_next: 1 runnable jobs of (5 pending, 3 tasked)
[2017-06-29 12:18:52,828: INFO/job_scheduler] Job 100: RunJobThread.run
[2017-06-29 12:18:52,829: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com stop_corosync2 {}
[2017-06-29 12:18:52,848: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 12152585-5039-42db-aec6-1cee971ad58e
[2017-06-29 12:18:52,848: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 12152585-5039-42db-aec6-1cee971ad58e
[2017-06-29 12:18:52,898: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration configured->unconfigured (intentional=True) 140667830903696
[2017-06-29 12:18:52,902: INFO/job] Job 98: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration
[2017-06-29 12:18:52,904: INFO/job] Job 98: not_deleted=True
[2017-06-29 12:18:52,904: INFO/job_scheduler] Job 98: complete_job: Updating cache
[2017-06-29 12:18:52,904: INFO/job_scheduler] Job 98: locked_item 140667830903696 <class 'chroma_core.models.ntp.NTPConfiguration'> True True
[2017-06-29 12:18:52,907: INFO/job_scheduler] Job 98 completing (errored=False, cancelled=False)
[2017-06-29 12:18:52,922: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 3 tasked)
[2017-06-29 12:18:53,803: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:18:53,803: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:18:53,803: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:18:53,803: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:18:53,803: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1774c138-d722-4821-8427-36d45aba363b
[2017-06-29 12:18:53,803: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1774c138-d722-4821-8427-36d45aba363b
[2017-06-29 12:18:53,832: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1774c138-d722-4821-8427-36d45aba363b
[2017-06-29 12:18:53,832: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1774c138-d722-4821-8427-36d45aba363b
[2017-06-29 12:18:53,832: INFO/job_scheduler] Job 95 finished 1 steps successfully
[2017-06-29 12:18:53,906: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration configured->unconfigured (intentional=True) 140667830905360
[2017-06-29 12:18:53,909: INFO/job] Job 95: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration
[2017-06-29 12:18:53,910: INFO/job] Job 95: not_deleted=True
[2017-06-29 12:18:53,910: INFO/job_scheduler] Job 95: complete_job: Updating cache
[2017-06-29 12:18:53,910: INFO/job_scheduler] Job 95: locked_item 140667830905360 <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> True True
[2017-06-29 12:18:53,912: INFO/job_scheduler] Job 95 completing (errored=False, cancelled=False)
[2017-06-29 12:18:53,922: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-29 12:18:54,043: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-29 12:18:54,058: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-29 12:18:57,721: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:18:57,721: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:18:57,721: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:18:57,721: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:18:57,721: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d8e87b65-7b18-4116-b83e-c1c08188572c
[2017-06-29 12:18:57,721: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d8e87b65-7b18-4116-b83e-c1c08188572c
[2017-06-29 12:18:57,745: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d8e87b65-7b18-4116-b83e-c1c08188572c
[2017-06-29 12:18:57,745: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d8e87b65-7b18-4116-b83e-c1c08188572c
[2017-06-29 12:18:57,746: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-29 12:18:57,762: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 87198c5f-1cd0-4ce4-a0da-c7e250c7867e
[2017-06-29 12:18:57,762: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 87198c5f-1cd0-4ce4-a0da-c7e250c7867e
[2017-06-29 12:18:58,842: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:18:58,842: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:18:58,842: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:18:58,842: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:18:58,843: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 87198c5f-1cd0-4ce4-a0da-c7e250c7867e
[2017-06-29 12:18:58,843: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 87198c5f-1cd0-4ce4-a0da-c7e250c7867e
[2017-06-29 12:18:58,876: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 87198c5f-1cd0-4ce4-a0da-c7e250c7867e
[2017-06-29 12:18:58,876: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 87198c5f-1cd0-4ce4-a0da-c7e250c7867e
[2017-06-29 12:18:58,916: INFO/job_scheduler] Job 96 finished 2 steps successfully
[2017-06-29 12:18:58,930: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_up->lnet_down (intentional=True) 140667830483728
[2017-06-29 12:18:58,936: INFO/job] Job 96: StateChangeJob complete, setting state lnet_down on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-29 12:18:58,936: INFO/job] Job 96: not_deleted=True
[2017-06-29 12:18:58,936: INFO/job_scheduler] Job 96: complete_job: Updating cache
[2017-06-29 12:18:58,936: INFO/job_scheduler] Job 96: locked_item 140667830483728 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-29 12:18:58,940: INFO/job_scheduler] Job 96 completing (errored=False, cancelled=False)
[2017-06-29 12:18:58,956: INFO/job_scheduler] run_next: 1 runnable jobs of (4 pending, 1 tasked)
[2017-06-29 12:18:58,959: INFO/job_scheduler] Job 99: RunJobThread.run
[2017-06-29 12:18:58,960: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unload_lnet {}
[2017-06-29 12:18:58,974: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a67c4a4e-ac28-4702-afc7-dab9dbabbf0f
[2017-06-29 12:18:58,974: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc a67c4a4e-ac28-4702-afc7-dab9dbabbf0f
[2017-06-29 12:19:00,943: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:19:00,943: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:19:00,943: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:19:00,943: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:19:00,943: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 12152585-5039-42db-aec6-1cee971ad58e
[2017-06-29 12:19:00,943: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 12152585-5039-42db-aec6-1cee971ad58e
[2017-06-29 12:19:00,964: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 12152585-5039-42db-aec6-1cee971ad58e
[2017-06-29 12:19:00,964: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 12152585-5039-42db-aec6-1cee971ad58e
[2017-06-29 12:19:00,964: INFO/job_scheduler] Job 100 finished 1 steps successfully
[2017-06-29 12:19:01,021: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration started->stopped (intentional=True) 140667830022736
[2017-06-29 12:19:01,029: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-06-29 12:19:01,038: INFO/job] Job 100: StateChangeJob complete, setting state stopped on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 12:19:01,039: INFO/job] Job 100: not_deleted=True
[2017-06-29 12:19:01,040: INFO/job_scheduler] Job 100: complete_job: Updating cache
[2017-06-29 12:19:01,040: INFO/job_scheduler] Job 100: locked_item 140667830022736 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-29 12:19:01,060: INFO/job_scheduler] Job 100 completing (errored=False, cancelled=False)
[2017-06-29 12:19:01,079: INFO/job_scheduler] run_next: 1 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 12:19:01,084: INFO/job_scheduler] Job 101: RunJobThread.run
[2017-06-29 12:19:01,085: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unconfigure_corosync2 {'mcast_port': 40759, 'host_fqdn': u'lotus-32vm5.lotus.hpdd.lab.intel.com'}
[2017-06-29 12:19:01,108: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7f45d67c-a0a7-4729-88fe-e73a168d2b2e
[2017-06-29 12:19:01,108: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7f45d67c-a0a7-4729-88fe-e73a168d2b2e
[2017-06-29 12:19:01,112: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:19:01,112: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:19:01,112: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:19:01,112: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:19:01,113: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a67c4a4e-ac28-4702-afc7-dab9dbabbf0f
[2017-06-29 12:19:01,113: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc a67c4a4e-ac28-4702-afc7-dab9dbabbf0f
[2017-06-29 12:19:01,143: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a67c4a4e-ac28-4702-afc7-dab9dbabbf0f
[2017-06-29 12:19:01,143: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc a67c4a4e-ac28-4702-afc7-dab9dbabbf0f
[2017-06-29 12:19:01,144: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com device_plugin {'plugin': 'linux_network'}
[2017-06-29 12:19:01,167: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7cc48952-e82c-4e65-982a-7dc0fa781528
[2017-06-29 12:19:01,167: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7cc48952-e82c-4e65-982a-7dc0fa781528
[2017-06-29 12:19:02,236: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:19:02,236: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:19:02,236: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:19:02,236: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:19:02,236: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7cc48952-e82c-4e65-982a-7dc0fa781528
[2017-06-29 12:19:02,236: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7cc48952-e82c-4e65-982a-7dc0fa781528
[2017-06-29 12:19:02,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7cc48952-e82c-4e65-982a-7dc0fa781528
[2017-06-29 12:19:02,282: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7cc48952-e82c-4e65-982a-7dc0fa781528
[2017-06-29 12:19:02,387: INFO/job_scheduler] Job 99 finished 2 steps successfully
[2017-06-29 12:19:02,404: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_down->lnet_unloaded (intentional=True) 140667830483728
[2017-06-29 12:19:02,409: INFO/job] Job 99: StateChangeJob complete, setting state lnet_unloaded on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-29 12:19:02,409: INFO/job] Job 99: not_deleted=True
[2017-06-29 12:19:02,409: INFO/job_scheduler] Job 99: complete_job: Updating cache
[2017-06-29 12:19:02,409: INFO/job_scheduler] Job 99: locked_item 140667830483728 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-29 12:19:02,412: INFO/job_scheduler] Job 99 completing (errored=False, cancelled=False)
[2017-06-29 12:19:02,422: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 1 tasked)
[2017-06-29 12:19:02,449: INFO/job_scheduler] Job 102: RunJobThread.run
[2017-06-29 12:19:02,450: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com unconfigure_lnet {}
[2017-06-29 12:19:02,465: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 648ae5fa-1038-488f-b75a-75efc33273b8
[2017-06-29 12:19:02,465: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 648ae5fa-1038-488f-b75a-75efc33273b8
[2017-06-29 12:19:02,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:19:02,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:19:02,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:19:02,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:19:02,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 648ae5fa-1038-488f-b75a-75efc33273b8
[2017-06-29 12:19:02,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 648ae5fa-1038-488f-b75a-75efc33273b8
[2017-06-29 12:19:02,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 648ae5fa-1038-488f-b75a-75efc33273b8
[2017-06-29 12:19:02,539: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 648ae5fa-1038-488f-b75a-75efc33273b8
[2017-06-29 12:19:02,539: INFO/job] Invalid result None fixed up on called to unconfigure_lnet with args {}
[2017-06-29 12:19:02,539: INFO/job_scheduler] Job 102 finished 1 steps successfully
[2017-06-29 12:19:02,608: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration lnet_unloaded->unconfigured (intentional=True) 140667830483728
[2017-06-29 12:19:02,613: INFO/job] Job 102: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration
[2017-06-29 12:19:02,613: INFO/job] Job 102: not_deleted=True
[2017-06-29 12:19:02,613: INFO/job_scheduler] Job 102: complete_job: Updating cache
[2017-06-29 12:19:02,613: INFO/job_scheduler] Job 102: locked_item 140667830483728 <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> True True
[2017-06-29 12:19:02,626: INFO/job_scheduler] Job 102 completing (errored=False, cancelled=False)
[2017-06-29 12:19:02,646: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-06-29 12:19:04,085: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-06-29 12:19:04,106: INFO/job_scheduler] run_next: 0 runnable jobs of (1 pending, 1 tasked)
[2017-06-29 12:19:07,191: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:19:07,191: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:19:07,191: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:19:07,191: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:19:07,191: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7f45d67c-a0a7-4729-88fe-e73a168d2b2e
[2017-06-29 12:19:07,191: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7f45d67c-a0a7-4729-88fe-e73a168d2b2e
[2017-06-29 12:19:07,225: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7f45d67c-a0a7-4729-88fe-e73a168d2b2e
[2017-06-29 12:19:07,225: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7f45d67c-a0a7-4729-88fe-e73a168d2b2e
[2017-06-29 12:19:07,225: INFO/job_scheduler] Job 101 finished 1 steps successfully
[2017-06-29 12:19:07,320: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration stopped->unconfigured (intentional=True) 140667830022736
[2017-06-29 12:19:07,326: INFO/job] Job 101: StateChangeJob complete, setting state unconfigured on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration
[2017-06-29 12:19:07,326: INFO/job] Job 101: not_deleted=True
[2017-06-29 12:19:07,326: INFO/job_scheduler] Job 101: complete_job: Updating cache
[2017-06-29 12:19:07,326: INFO/job_scheduler] Job 101: locked_item 140667830022736 <class 'chroma_core.models.corosync2.Corosync2Configuration'> True True
[2017-06-29 12:19:07,330: INFO/job_scheduler] Job 101 completing (errored=False, cancelled=False)
[2017-06-29 12:19:07,345: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-29 12:19:07,367: INFO/job_scheduler] Job 103: RunJobThread.run
[2017-06-29 12:19:07,369: INFO/job] invoke_agent on agent lotus-32vm5.lotus.hpdd.lab.intel.com deregister_server {}
[2017-06-29 12:19:07,389: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fdee972b-0529-4a8d-8ffc-bdfb254d8c5a
[2017-06-29 12:19:07,389: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fdee972b-0529-4a8d-8ffc-bdfb254d8c5a
[2017-06-29 12:19:07,462: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:19:07,462: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:19:07,462: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:19:07,462: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm5.lotus.hpdd.lab.intel.com/4fa3b1e4-7b49-4dc3-bf70-0223c6eee629
[2017-06-29 12:19:07,462: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fdee972b-0529-4a8d-8ffc-bdfb254d8c5a
[2017-06-29 12:19:07,462: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fdee972b-0529-4a8d-8ffc-bdfb254d8c5a
[2017-06-29 12:19:07,506: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fdee972b-0529-4a8d-8ffc-bdfb254d8c5a
[2017-06-29 12:19:07,506: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fdee972b-0529-4a8d-8ffc-bdfb254d8c5a
[2017-06-29 12:19:07,533: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 12:19:07,534: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com LNet configuration unconfigured->removed (intentional=False) 140667830853008
[2017-06-29 12:19:07,548: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> 2
[2017-06-29 12:19:07,699: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 12:19:07,699: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->removed (intentional=False) 140667830914704
[2017-06-29 12:19:07,722: INFO/job] Lowered 2 alerts while deleting <class 'chroma_core.models.corosync2.Corosync2Configuration'> 2
[2017-06-29 12:19:07,757: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 12:19:07,757: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration unconfigured->removed (intentional=False) 140667830900816
[2017-06-29 12:19:07,771: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.ntp.NTPConfiguration'> 2
[2017-06-29 12:19:07,796: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 12:19:07,797: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->removed (intentional=False) 140667830956944
[2017-06-29 12:19:07,811: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> 2
[2017-06-29 12:19:07,848: INFO/job] Deleting dependent lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration for host lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 12:19:07,848: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Pacemaker configuration stopped->removed (intentional=False) 140667830956752
[2017-06-29 12:19:07,858: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> 2
[2017-06-29 12:19:08,466: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->removed (intentional=False) 140667830022736
[2017-06-29 12:19:08,479: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm5.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-06-29 12:19:08,506: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.corosync2.Corosync2Configuration'> 2
[2017-06-29 12:19:08,520: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com NTP configuration unconfigured->removed (intentional=False) 140667830903696
[2017-06-29 12:19:08,526: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.ntp.NTPConfiguration'> 2
[2017-06-29 12:19:08,579: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->removed (intentional=False) 140667830905360
[2017-06-29 12:19:08,584: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> 2
[2017-06-29 12:19:08,624: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.host.ManagedHost'> 2
[2017-06-29 12:19:08,631: INFO/job_scheduler] Job 103 finished 2 steps successfully
[2017-06-29 12:19:08,684: INFO/job] StatefulObject.set_state lotus-32vm5.lotus.hpdd.lab.intel.com managed->removed (intentional=True) 140668098733712
[2017-06-29 12:19:08,695: INFO/job] Job 103: StateChangeJob complete, setting state removed on lotus-32vm5.lotus.hpdd.lab.intel.com
[2017-06-29 12:19:08,695: INFO/job] Job 103: not_deleted=None
[2017-06-29 12:19:08,696: INFO/job_scheduler] Job 103: complete_job: Updating cache
[2017-06-29 12:19:08,696: INFO/job_scheduler] Job 103: locked_item 140668098733712 <class 'chroma_core.models.host.ManagedHost'> True None
[2017-06-29 12:19:08,696: INFO/job_scheduler] Job 103 completing (errored=False, cancelled=False)
[2017-06-29 12:19:08,754: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:19:40,479: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 12:19:40,482: INFO/command_plan] set_state: <class 'chroma_core.models.host.ManagedHost'>-3 transitioning from unconfigured to removed
[2017-06-29 12:19:40,524: INFO/command_plan] Created command 38 (Remove host lotus-32vm6.lotus.hpdd.lab.intel.com from configuration) with 1 jobs
[2017-06-29 12:19:40,579: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-29 12:19:40,647: INFO/job_scheduler] Job 104: RunJobThread.run
[2017-06-29 12:19:40,725: INFO/job] Deleting dependent lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration for host lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-29 12:19:40,725: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration unconfigured->removed (intentional=False) 140667830917840
[2017-06-29 12:19:40,749: INFO/job] AlertState: Raised <class 'chroma_core.models.pacemaker.PacemakerStoppedAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com Pacemaker configuration at severity 20
[2017-06-29 12:19:40,769: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.pacemaker.PacemakerConfiguration'> 3
[2017-06-29 12:19:40,949: INFO/job] Deleting dependent lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration for host lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-29 12:19:40,950: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com RSyslog configuration unconfigured->removed (intentional=False) 140667830916752
[2017-06-29 12:19:40,958: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.rsyslog.RSyslogConfiguration'> 3
[2017-06-29 12:19:40,977: INFO/job] Deleting dependent lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration for host lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-29 12:19:40,977: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com NTP configuration unconfigured->removed (intentional=False) 140667830917904
[2017-06-29 12:19:40,983: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.ntp.NTPConfiguration'> 3
[2017-06-29 12:19:41,008: INFO/job] Deleting dependent lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration for host lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-29 12:19:41,015: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration unconfigured->removed (intentional=False) 140667830917328
[2017-06-29 12:19:41,034: INFO/job] AlertState: Raised <class 'chroma_core.models.lnet_configuration.LNetOfflineAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com LNet configuration at severity 20
[2017-06-29 12:19:41,071: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.lnet_configuration.LNetConfiguration'> 3
[2017-06-29 12:19:41,102: INFO/job] Deleting dependent lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration for host lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-29 12:19:41,102: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration unconfigured->removed (intentional=False) 140667830916816
[2017-06-29 12:19:41,124: INFO/job] AlertState: Raised <class 'chroma_core.models.corosync_common.CorosyncStoppedAlert'> on lotus-32vm6.lotus.hpdd.lab.intel.com Corosync2 configuration at severity 20
[2017-06-29 12:19:41,140: INFO/job] Lowered 1 alerts while deleting <class 'chroma_core.models.corosync2.Corosync2Configuration'> 3
[2017-06-29 12:19:41,832: INFO/job] Lowered 0 alerts while deleting <class 'chroma_core.models.host.ManagedHost'> 3
[2017-06-29 12:19:41,840: INFO/job_scheduler] Job 104 finished 1 steps successfully
[2017-06-29 12:19:41,901: INFO/job] StatefulObject.set_state lotus-32vm6.lotus.hpdd.lab.intel.com unconfigured->removed (intentional=True) 140667830958928
[2017-06-29 12:19:41,907: INFO/job] Job 104: StateChangeJob complete, setting state removed on lotus-32vm6.lotus.hpdd.lab.intel.com
[2017-06-29 12:19:41,907: INFO/job] Job 104: not_deleted=None
[2017-06-29 12:19:41,907: INFO/job_scheduler] Job 104: complete_job: Updating cache
[2017-06-29 12:19:41,907: INFO/job_scheduler] Job 104: locked_item 140667830958928 <class 'chroma_core.models.host.ManagedHost'> True None
[2017-06-29 12:19:41,907: INFO/job_scheduler] Job 104 completing (errored=False, cancelled=False)
[2017-06-29 12:19:42,024: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:24:19,618: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:24:19,635: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:28:51,085: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:28:51,106: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:34:31,533: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:34:31,547: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:39:02,798: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:39:02,817: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:44:43,360: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:44:43,377: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:49:14,537: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:49:14,551: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:54:55,151: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:54:55,169: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:59:26,338: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 12:59:26,353: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 13:05:07,184: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 13:05:07,202: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 13:09:38,258: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 13:09:38,276: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 13:15:18,942: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 13:15:18,958: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 13:19:50,046: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 13:19:50,062: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 13:25:30,770: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 13:25:30,793: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 13:30:01,964: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 13:30:01,984: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 13:35:42,433: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 13:35:42,448: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 13:40:13,837: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 13:40:13,857: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 13:45:54,278: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 13:45:54,294: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 13:50:25,544: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 13:50:25,568: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 13:56:06,313: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 13:56:06,328: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 14:00:37,255: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 14:00:37,275: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 14:06:18,068: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 14:06:18,081: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 14:10:49,281: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 14:10:49,293: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 14:16:29,744: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 14:16:29,762: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 14:21:01,180: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 14:21:01,198: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 14:26:41,573: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 14:26:41,587: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 14:27:30,393: INFO/chroma_service] Stopping job_scheduler
[2017-06-29 14:27:30,393: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-06-29 14:27:30,393: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.stop
[2017-06-29 14:27:30,394: INFO/queue] Stopping ServiceQueue agent_action_runner_rx
[2017-06-29 14:27:30,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-06-29 14:27:30,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete
[2017-06-29 14:27:30,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-06-29 14:27:30,922: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.complete_all
[2017-06-29 14:27:30,922: INFO/job_scheduler] Stopping...
[2017-06-29 14:27:30,922: INFO/queue] Stopping ServiceQueue job_scheduler_notifications
[2017-06-29 14:27:30,923: INFO/job_scheduler] Joining...
[2017-06-29 14:27:31,796: INFO/job_scheduler] Complete.
[2017-06-29 14:27:31,797: INFO/chroma_service] Joining job_scheduler
[2017-06-29 14:27:31,797: INFO/job_scheduler] Cancelling outstanding jobs...
[2017-06-29 14:27:31,805: ERROR/chroma_service] Rogue thread still running, exiting hard
[2017-06-29 14:27:31,805: ERROR/chroma_service] ['Thread-85216', 'Thread-85219', 'Thread-85218', 'Thread-85217', 'Thread-85258', 'Thread-85220', 'MainThread', 'Thread-85227']
[2017-06-29 14:27:55,362: INFO/queue] Purged 0 messages from 'job_scheduler_notifications' queue
[2017-06-29 14:27:55,413: INFO/queue] Purged 1 messages from 'agent_action_runner_rx' queue
[2017-06-29 14:28:03,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 14:28:03,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 14:28:04,177: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 14:28:04,177: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 14:28:07,784: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 14:28:07,817: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 14:28:07,840: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 14:28:08,317: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 14:28:08,336: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 14:28:08,381: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 14:38:09,363: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 14:38:09,378: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 14:38:09,757: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 14:38:09,776: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 14:48:21,037: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 14:48:21,051: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 14:48:21,526: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 14:48:21,543: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 14:58:32,769: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 14:58:32,783: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 14:58:33,249: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 14:58:33,288: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 15:08:44,551: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 15:08:44,568: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 15:08:44,867: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 15:08:44,882: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 15:14:00,229: INFO/job] AlertState: Raised <class 'chroma_core.models.command.CommandRunningAlert'> on Command object at severity 20
[2017-06-29 15:14:00,233: INFO/command_plan] set_state: <class 'chroma_core.models.filesystem.ManagedFilesystem'>-1 transitioning from unavailable to available
[2017-06-29 15:14:00,611: INFO/command_plan] Created command 39 (Creating filesystem fs1) with 21 jobs
[2017-06-29 15:14:00,692: INFO/job_scheduler] run_next: 5 runnable jobs of (21 pending, 0 tasked)
[2017-06-29 15:14:00,830: INFO/job] create_parameters: host: 'lotus-32vm7.lotus.hpdd.lab.intel.com' active_volume_node: 'TargetVolumeInfo(host=<ManagedHost: lotus-32vm7.lotus.hpdd.lab.intel.com>, path=u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk3', device_type='linux')'
[2017-06-29 15:14:00,910: INFO/job] create_parameters: host: 'lotus-32vm7.lotus.hpdd.lab.intel.com' active_volume_node: 'TargetVolumeInfo(host=<ManagedHost: lotus-32vm7.lotus.hpdd.lab.intel.com>, path=u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', device_type='linux')'
[2017-06-29 15:14:01,024: INFO/job] create_parameters: host: 'lotus-32vm7.lotus.hpdd.lab.intel.com' active_volume_node: 'TargetVolumeInfo(host=<ManagedHost: lotus-32vm7.lotus.hpdd.lab.intel.com>, path=u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', device_type='linux')'
[2017-06-29 15:14:01,114: INFO/job] create_parameters: host: 'lotus-32vm7.lotus.hpdd.lab.intel.com' active_volume_node: 'TargetVolumeInfo(host=<ManagedHost: lotus-32vm7.lotus.hpdd.lab.intel.com>, path=u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk5', device_type='linux')'
[2017-06-29 15:14:01,151: INFO/job] create_parameters: host: 'lotus-32vm7.lotus.hpdd.lab.intel.com' active_volume_node: 'TargetVolumeInfo(host=<ManagedHost: lotus-32vm7.lotus.hpdd.lab.intel.com>, path=u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk1', device_type='linux')'
[2017-06-29 15:14:01,160: INFO/job_scheduler] Job 105: RunJobThread.run
[2017-06-29 15:14:01,163: INFO/job_scheduler] Job 106: RunJobThread.run
[2017-06-29 15:14:01,165: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', 'device_type': 'linux'}
[2017-06-29 15:14:01,173: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk3', 'device_type': 'linux'}
[2017-06-29 15:14:01,174: INFO/job_scheduler] Job 107: RunJobThread.run
[2017-06-29 15:14:01,182: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ea8a5327-8c64-4815-be76-b67168d3fb33
[2017-06-29 15:14:01,182: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ea8a5327-8c64-4815-be76-b67168d3fb33
[2017-06-29 15:14:01,191: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux'}
[2017-06-29 15:14:01,191: INFO/job_scheduler] Job 108: RunJobThread.run
[2017-06-29 15:14:01,195: INFO/job_scheduler] Job 109: RunJobThread.run
[2017-06-29 15:14:01,198: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk5', 'device_type': 'linux'}
[2017-06-29 15:14:01,198: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk1', 'device_type': 'linux'}
[2017-06-29 15:14:01,232: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ac534e84-7f0d-4150-bb9d-f4c9f52d97f7
[2017-06-29 15:14:01,232: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ac534e84-7f0d-4150-bb9d-f4c9f52d97f7
[2017-06-29 15:14:01,256: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 72e85d28-185d-4363-bf0c-b23777246861
[2017-06-29 15:14:01,256: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 72e85d28-185d-4363-bf0c-b23777246861
[2017-06-29 15:14:01,289: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3f6f5eb5-9a89-4188-a973-5e1a54d179c3
[2017-06-29 15:14:01,289: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3f6f5eb5-9a89-4188-a973-5e1a54d179c3
[2017-06-29 15:14:01,290: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:01,290: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:01,290: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:01,290: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:01,290: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ea8a5327-8c64-4815-be76-b67168d3fb33
[2017-06-29 15:14:01,290: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ea8a5327-8c64-4815-be76-b67168d3fb33
[2017-06-29 15:14:01,319: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ea8a5327-8c64-4815-be76-b67168d3fb33
[2017-06-29 15:14:01,319: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ea8a5327-8c64-4815-be76-b67168d3fb33
[2017-06-29 15:14:01,320: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com import_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', 'pacemaker_ha_operation': False, 'device_type': 'linux', 'validate_importable': False}
[2017-06-29 15:14:01,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fb3e4d87-152c-42b8-b227-ed21ac5f070a
[2017-06-29 15:14:01,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc fb3e4d87-152c-42b8-b227-ed21ac5f070a
[2017-06-29 15:14:01,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:01,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:01,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:01,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:01,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ac534e84-7f0d-4150-bb9d-f4c9f52d97f7
[2017-06-29 15:14:01,322: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ac534e84-7f0d-4150-bb9d-f4c9f52d97f7
[2017-06-29 15:14:01,338: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2be75df9-1e40-4123-a69e-397220f935a0
[2017-06-29 15:14:01,338: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2be75df9-1e40-4123-a69e-397220f935a0
[2017-06-29 15:14:01,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:01,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:01,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:01,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:01,351: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 72e85d28-185d-4363-bf0c-b23777246861
[2017-06-29 15:14:01,351: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 72e85d28-185d-4363-bf0c-b23777246861
[2017-06-29 15:14:01,352: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 72e85d28-185d-4363-bf0c-b23777246861
[2017-06-29 15:14:01,352: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 72e85d28-185d-4363-bf0c-b23777246861
[2017-06-29 15:14:01,356: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com import_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'pacemaker_ha_operation': False, 'device_type': 'linux', 'validate_importable': False}
[2017-06-29 15:14:01,365: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ac534e84-7f0d-4150-bb9d-f4c9f52d97f7
[2017-06-29 15:14:01,365: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ac534e84-7f0d-4150-bb9d-f4c9f52d97f7
[2017-06-29 15:14:01,365: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com import_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk3', 'pacemaker_ha_operation': False, 'device_type': 'linux', 'validate_importable': False}
[2017-06-29 15:14:01,382: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 93b488bf-7f0f-4c3a-9a7e-c13d8548ba47
[2017-06-29 15:14:01,382: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 93b488bf-7f0f-4c3a-9a7e-c13d8548ba47
[2017-06-29 15:14:01,406: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 00ad715d-e1c6-4155-99a1-85365166e110
[2017-06-29 15:14:01,406: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 00ad715d-e1c6-4155-99a1-85365166e110
[2017-06-29 15:14:01,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:01,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:01,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:01,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:01,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2be75df9-1e40-4123-a69e-397220f935a0
[2017-06-29 15:14:01,412: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2be75df9-1e40-4123-a69e-397220f935a0
[2017-06-29 15:14:01,414: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:01,414: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:01,415: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:01,415: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:01,415: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3f6f5eb5-9a89-4188-a973-5e1a54d179c3
[2017-06-29 15:14:01,415: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3f6f5eb5-9a89-4188-a973-5e1a54d179c3
[2017-06-29 15:14:01,415: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2be75df9-1e40-4123-a69e-397220f935a0
[2017-06-29 15:14:01,415: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2be75df9-1e40-4123-a69e-397220f935a0
[2017-06-29 15:14:01,416: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com check_block_device {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', 'device_type': 'linux'}
[2017-06-29 15:14:01,430: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3f6f5eb5-9a89-4188-a973-5e1a54d179c3
[2017-06-29 15:14:01,430: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3f6f5eb5-9a89-4188-a973-5e1a54d179c3
[2017-06-29 15:14:01,430: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com import_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk1', 'pacemaker_ha_operation': False, 'device_type': 'linux', 'validate_importable': False}
[2017-06-29 15:14:01,442: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0dd5dee2-ec7b-4c57-b28e-d6677619fb33
[2017-06-29 15:14:01,442: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0dd5dee2-ec7b-4c57-b28e-d6677619fb33
[2017-06-29 15:14:01,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 73239e2c-3564-4cbb-9f7f-6cc6f3bab7d7
[2017-06-29 15:14:01,472: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 73239e2c-3564-4cbb-9f7f-6cc6f3bab7d7
[2017-06-29 15:14:01,473: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:01,473: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:01,473: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:01,473: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:01,473: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fb3e4d87-152c-42b8-b227-ed21ac5f070a
[2017-06-29 15:14:01,473: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc fb3e4d87-152c-42b8-b227-ed21ac5f070a
[2017-06-29 15:14:01,481: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:01,481: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:01,481: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:01,481: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:01,481: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 93b488bf-7f0f-4c3a-9a7e-c13d8548ba47
[2017-06-29 15:14:01,481: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 93b488bf-7f0f-4c3a-9a7e-c13d8548ba47
[2017-06-29 15:14:01,503: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fb3e4d87-152c-42b8-b227-ed21ac5f070a
[2017-06-29 15:14:01,503: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc fb3e4d87-152c-42b8-b227-ed21ac5f070a
[2017-06-29 15:14:01,509: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com import_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk5', 'pacemaker_ha_operation': False, 'device_type': 'linux', 'validate_importable': False}
[2017-06-29 15:14:01,517: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 93b488bf-7f0f-4c3a-9a7e-c13d8548ba47
[2017-06-29 15:14:01,517: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 93b488bf-7f0f-4c3a-9a7e-c13d8548ba47
[2017-06-29 15:14:01,517: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com check_block_device {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux'}
[2017-06-29 15:14:01,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 056ffdee-311e-4e15-b3a9-226ea8db43aa
[2017-06-29 15:14:01,545: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 056ffdee-311e-4e15-b3a9-226ea8db43aa
[2017-06-29 15:14:01,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ec774201-44cf-4fcc-9c00-239b134fc41a
[2017-06-29 15:14:01,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ec774201-44cf-4fcc-9c00-239b134fc41a
[2017-06-29 15:14:01,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:01,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:01,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:01,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:01,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 00ad715d-e1c6-4155-99a1-85365166e110
[2017-06-29 15:14:01,563: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 00ad715d-e1c6-4155-99a1-85365166e110
[2017-06-29 15:14:01,564: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:01,564: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:01,564: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:01,564: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:01,564: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 73239e2c-3564-4cbb-9f7f-6cc6f3bab7d7
[2017-06-29 15:14:01,564: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 73239e2c-3564-4cbb-9f7f-6cc6f3bab7d7
[2017-06-29 15:14:01,596: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:01,596: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:01,597: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:01,597: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:01,597: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 056ffdee-311e-4e15-b3a9-226ea8db43aa
[2017-06-29 15:14:01,597: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 056ffdee-311e-4e15-b3a9-226ea8db43aa
[2017-06-29 15:14:01,603: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 00ad715d-e1c6-4155-99a1-85365166e110
[2017-06-29 15:14:01,603: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 00ad715d-e1c6-4155-99a1-85365166e110
[2017-06-29 15:14:01,604: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com check_block_device {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk3', 'device_type': 'linux'}
[2017-06-29 15:14:01,611: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 73239e2c-3564-4cbb-9f7f-6cc6f3bab7d7
[2017-06-29 15:14:01,611: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 73239e2c-3564-4cbb-9f7f-6cc6f3bab7d7
[2017-06-29 15:14:01,612: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com check_block_device {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk1', 'device_type': 'linux'}
[2017-06-29 15:14:01,619: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 056ffdee-311e-4e15-b3a9-226ea8db43aa
[2017-06-29 15:14:01,619: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 056ffdee-311e-4e15-b3a9-226ea8db43aa
[2017-06-29 15:14:01,620: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com check_block_device {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk5', 'device_type': 'linux'}
[2017-06-29 15:14:01,632: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc da1c308b-f45a-4090-ab55-20ef5af37d73
[2017-06-29 15:14:01,632: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc da1c308b-f45a-4090-ab55-20ef5af37d73
[2017-06-29 15:14:01,663: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 23c93051-0e62-46b1-8b30-87f5c20ecff0
[2017-06-29 15:14:01,663: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 23c93051-0e62-46b1-8b30-87f5c20ecff0
[2017-06-29 15:14:01,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d941ba52-90b4-4bb1-9a78-4b0a4ce8e98b
[2017-06-29 15:14:01,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d941ba52-90b4-4bb1-9a78-4b0a4ce8e98b
[2017-06-29 15:14:02,543: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:02,543: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:02,543: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:02,543: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:02,543: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0dd5dee2-ec7b-4c57-b28e-d6677619fb33
[2017-06-29 15:14:02,543: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0dd5dee2-ec7b-4c57-b28e-d6677619fb33
[2017-06-29 15:14:02,561: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0dd5dee2-ec7b-4c57-b28e-d6677619fb33
[2017-06-29 15:14:02,561: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0dd5dee2-ec7b-4c57-b28e-d6677619fb33
[2017-06-29 15:14:02,562: INFO/job] fs1-MDT0000 passed pre-format check, allowing subsequent reformats
[2017-06-29 15:14:02,667: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:02,667: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:02,667: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:02,667: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:02,667: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ec774201-44cf-4fcc-9c00-239b134fc41a
[2017-06-29 15:14:02,667: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ec774201-44cf-4fcc-9c00-239b134fc41a
[2017-06-29 15:14:02,669: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com format_target {'index': 0, 'mkfsoptions': '-I 512 -i 2048 -J size=2048', 'backfstype': 'ldiskfs', 'target_name': u'fs1-MDT0000', 'mgsnode': ((u'10.14.81.226@tcp0',), (u'10.14.81.227@tcp0',)), 'target_types': 'mdt', 'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', 'device_type': 'linux', 'failnode': [u'10.14.81.227@tcp0'], 'fsname': u'fs1'}
[2017-06-29 15:14:02,690: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ec774201-44cf-4fcc-9c00-239b134fc41a
[2017-06-29 15:14:02,690: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ec774201-44cf-4fcc-9c00-239b134fc41a
[2017-06-29 15:14:02,691: INFO/job] fs1-MDT0002 passed pre-format check, allowing subsequent reformats
[2017-06-29 15:14:02,721: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e5f294d8-310e-4913-af4b-3daef18aecc0
[2017-06-29 15:14:02,721: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc e5f294d8-310e-4913-af4b-3daef18aecc0
[2017-06-29 15:14:02,721: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:02,721: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:02,721: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:02,721: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:02,721: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc da1c308b-f45a-4090-ab55-20ef5af37d73
[2017-06-29 15:14:02,721: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc da1c308b-f45a-4090-ab55-20ef5af37d73
[2017-06-29 15:14:02,722: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:02,722: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:02,722: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:02,722: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:02,722: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 23c93051-0e62-46b1-8b30-87f5c20ecff0
[2017-06-29 15:14:02,722: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 23c93051-0e62-46b1-8b30-87f5c20ecff0
[2017-06-29 15:14:02,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:02,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:02,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:02,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:02,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d941ba52-90b4-4bb1-9a78-4b0a4ce8e98b
[2017-06-29 15:14:02,733: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d941ba52-90b4-4bb1-9a78-4b0a4ce8e98b
[2017-06-29 15:14:02,734: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 23c93051-0e62-46b1-8b30-87f5c20ecff0
[2017-06-29 15:14:02,734: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 23c93051-0e62-46b1-8b30-87f5c20ecff0
[2017-06-29 15:14:02,735: INFO/job] MGS passed pre-format check, allowing subsequent reformats
[2017-06-29 15:14:02,750: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc da1c308b-f45a-4090-ab55-20ef5af37d73
[2017-06-29 15:14:02,750: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc da1c308b-f45a-4090-ab55-20ef5af37d73
[2017-06-29 15:14:02,751: INFO/job] fs1-MDT0001 passed pre-format check, allowing subsequent reformats
[2017-06-29 15:14:02,760: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d941ba52-90b4-4bb1-9a78-4b0a4ce8e98b
[2017-06-29 15:14:02,760: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d941ba52-90b4-4bb1-9a78-4b0a4ce8e98b
[2017-06-29 15:14:02,761: INFO/job] fs1-OST0000 passed pre-format check, allowing subsequent reformats
[2017-06-29 15:14:02,788: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com format_target {'index': 2, 'mkfsoptions': '-I 512 -i 2048 -J size=2048', 'backfstype': 'ldiskfs', 'target_name': u'fs1-MDT0002', 'mgsnode': ((u'10.14.81.226@tcp0',), (u'10.14.81.227@tcp0',)), 'target_types': 'mdt', 'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux', 'failnode': [u'10.14.81.227@tcp0'], 'fsname': u'fs1'}
[2017-06-29 15:14:02,821: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dfcfefe7-47cd-4b0b-95b0-3d8d2e961a44
[2017-06-29 15:14:02,821: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc dfcfefe7-47cd-4b0b-95b0-3d8d2e961a44
[2017-06-29 15:14:02,869: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com format_target {'backfstype': 'ldiskfs', 'target_name': u'MGS', 'target_types': 'mgs', 'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk1', 'device_type': 'linux', 'failnode': [u'10.14.81.227@tcp0']}
[2017-06-29 15:14:02,880: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com format_target {'index': 1, 'mkfsoptions': '-I 512 -i 2048 -J size=2048', 'backfstype': 'ldiskfs', 'target_name': u'fs1-MDT0001', 'mgsnode': ((u'10.14.81.226@tcp0',), (u'10.14.81.227@tcp0',)), 'target_types': 'mdt', 'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk3', 'device_type': 'linux', 'failnode': [u'10.14.81.227@tcp0'], 'fsname': u'fs1'}
[2017-06-29 15:14:02,897: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com format_target {'index': 0, 'mkfsoptions': '-J size=2048', 'backfstype': 'ldiskfs', 'target_name': u'fs1-OST0000', 'mgsnode': ((u'10.14.81.226@tcp0',), (u'10.14.81.227@tcp0',)), 'target_types': 'ost', 'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk5', 'device_type': 'linux', 'failnode': [u'10.14.81.227@tcp0'], 'fsname': u'fs1'}
[2017-06-29 15:14:02,902: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8c3b5f00-e6c6-41bb-b316-f92418fa097f
[2017-06-29 15:14:02,902: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8c3b5f00-e6c6-41bb-b316-f92418fa097f
[2017-06-29 15:14:02,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 39382fa6-2662-4a76-8f4b-9a805ba1a6f4
[2017-06-29 15:14:02,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 39382fa6-2662-4a76-8f4b-9a805ba1a6f4
[2017-06-29 15:14:02,973: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 48117eab-f681-45b8-853a-078083237d92
[2017-06-29 15:14:02,973: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 48117eab-f681-45b8-853a-078083237d92
[2017-06-29 15:14:09,873: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:09,873: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:09,880: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:09,880: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:09,884: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e5f294d8-310e-4913-af4b-3daef18aecc0
[2017-06-29 15:14:09,884: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc e5f294d8-310e-4913-af4b-3daef18aecc0
[2017-06-29 15:14:09,914: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e5f294d8-310e-4913-af4b-3daef18aecc0
[2017-06-29 15:14:09,914: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc e5f294d8-310e-4913-af4b-3daef18aecc0
[2017-06-29 15:14:09,945: INFO/job] Updating mtm volume_nodes for target fs1-MDT0000
[2017-06-29 15:14:09,989: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:09,989: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:09,989: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:09,989: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:09,989: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8c3b5f00-e6c6-41bb-b316-f92418fa097f
[2017-06-29 15:14:09,989: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8c3b5f00-e6c6-41bb-b316-f92418fa097f
[2017-06-29 15:14:10,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:10,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:10,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:10,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:10,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dfcfefe7-47cd-4b0b-95b0-3d8d2e961a44
[2017-06-29 15:14:10,003: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc dfcfefe7-47cd-4b0b-95b0-3d8d2e961a44
[2017-06-29 15:14:10,005: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:10,005: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:10,005: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:10,005: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:10,005: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 39382fa6-2662-4a76-8f4b-9a805ba1a6f4
[2017-06-29 15:14:10,005: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 39382fa6-2662-4a76-8f4b-9a805ba1a6f4
[2017-06-29 15:14:10,015: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 39382fa6-2662-4a76-8f4b-9a805ba1a6f4
[2017-06-29 15:14:10,015: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 39382fa6-2662-4a76-8f4b-9a805ba1a6f4
[2017-06-29 15:14:10,027: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8c3b5f00-e6c6-41bb-b316-f92418fa097f
[2017-06-29 15:14:10,027: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8c3b5f00-e6c6-41bb-b316-f92418fa097f
[2017-06-29 15:14:10,044: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dfcfefe7-47cd-4b0b-95b0-3d8d2e961a44
[2017-06-29 15:14:10,044: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc dfcfefe7-47cd-4b0b-95b0-3d8d2e961a44
[2017-06-29 15:14:10,134: INFO/job] Updating mtm volume_nodes for target MGS
[2017-06-29 15:14:10,177: INFO/job] Updating mtm volume_nodes for target fs1-MDT0001
[2017-06-29 15:14:10,224: INFO/job] Updating mtm volume_nodes for target fs1-MDT0002
[2017-06-29 15:14:10,244: INFO/job_scheduler] Job 106 finished 5 steps successfully
[2017-06-29 15:14:10,314: INFO/job_scheduler] Job 109 finished 5 steps successfully
[2017-06-29 15:14:10,354: INFO/job_scheduler] Job 105 finished 5 steps successfully
[2017-06-29 15:14:10,388: INFO/job_scheduler] Job 107 finished 5 steps successfully
[2017-06-29 15:14:10,638: INFO/job] StatefulObject.set_state fs1-MDT0000 unformatted->formatted (intentional=True) 140206121077072
[2017-06-29 15:14:10,646: INFO/job] Job 106: StateChangeJob complete, setting state formatted on fs1-MDT0000
[2017-06-29 15:14:10,647: INFO/job] Job 106: not_deleted=True
[2017-06-29 15:14:10,654: INFO/job_scheduler] Job 106: complete_job: Updating cache
[2017-06-29 15:14:10,654: INFO/job_scheduler] Job 106: locked_item 140206121077072 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-29 15:14:10,657: INFO/job_scheduler] Job 106: locked_item 140206053683600 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-06-29 15:14:10,658: INFO/job_scheduler] Job 106: locked_item 140206053683792 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-06-29 15:14:10,660: INFO/job_scheduler] Job 106 completing (errored=False, cancelled=False)
[2017-06-29 15:14:10,705: INFO/job_scheduler] run_next: 0 runnable jobs of (16 pending, 4 tasked)
[2017-06-29 15:14:10,715: INFO/job] StatefulObject.set_state MGS unformatted->formatted (intentional=True) 140206121213392
[2017-06-29 15:14:10,720: INFO/job] Job 109: StateChangeJob complete, setting state formatted on MGS
[2017-06-29 15:14:10,720: INFO/job] Job 109: not_deleted=True
[2017-06-29 15:14:10,741: INFO/job_scheduler] Job 109: complete_job: Updating cache
[2017-06-29 15:14:10,741: INFO/job_scheduler] Job 109: locked_item 140206121213392 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-29 15:14:10,747: INFO/job_scheduler] Job 109: locked_item 140206121277904 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-06-29 15:14:10,751: INFO/job_scheduler] Job 109: locked_item 140206121138192 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-06-29 15:14:10,752: INFO/job_scheduler] Job 109 completing (errored=False, cancelled=False)
[2017-06-29 15:14:10,814: INFO/job_scheduler] run_next: 1 runnable jobs of (16 pending, 3 tasked)
[2017-06-29 15:14:10,859: INFO/job_scheduler] Job 110: RunJobThread.run
[2017-06-29 15:14:10,862: INFO/job_scheduler] Job 110 finished 1 steps successfully
[2017-06-29 15:14:10,880: INFO/job] StatefulObject.set_state fs1-MDT0001 unformatted->formatted (intentional=True) 140206121261328
[2017-06-29 15:14:10,888: INFO/job] Job 105: StateChangeJob complete, setting state formatted on fs1-MDT0001
[2017-06-29 15:14:10,888: INFO/job] Job 105: not_deleted=True
[2017-06-29 15:14:10,903: INFO/job_scheduler] Job 105: complete_job: Updating cache
[2017-06-29 15:14:10,903: INFO/job_scheduler] Job 105: locked_item 140206121261328 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-29 15:14:10,906: INFO/job_scheduler] Job 105: locked_item 140206053668752 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-06-29 15:14:10,908: INFO/job_scheduler] Job 105: locked_item 140206053668944 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-06-29 15:14:10,910: INFO/job_scheduler] Job 105 completing (errored=False, cancelled=False)
[2017-06-29 15:14:10,957: INFO/job_scheduler] run_next: 0 runnable jobs of (15 pending, 3 tasked)
[2017-06-29 15:14:11,023: INFO/job] StatefulObject.set_state fs1-MDT0002 unformatted->formatted (intentional=True) 140206121260368
[2017-06-29 15:14:11,034: INFO/job] Job 107: StateChangeJob complete, setting state formatted on fs1-MDT0002
[2017-06-29 15:14:11,034: INFO/job] Job 107: not_deleted=True
[2017-06-29 15:14:11,061: INFO/job_scheduler] Job 107: complete_job: Updating cache
[2017-06-29 15:14:11,061: INFO/job_scheduler] Job 107: locked_item 140206121260368 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-29 15:14:11,065: INFO/job_scheduler] Job 107: locked_item 140206053685328 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-06-29 15:14:11,074: INFO/job_scheduler] Job 107: locked_item 140206053685520 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-06-29 15:14:11,076: INFO/job_scheduler] Job 107 completing (errored=False, cancelled=False)
[2017-06-29 15:14:11,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:11,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:11,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:11,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:11,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 48117eab-f681-45b8-853a-078083237d92
[2017-06-29 15:14:11,085: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 48117eab-f681-45b8-853a-078083237d92
[2017-06-29 15:14:11,101: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 48117eab-f681-45b8-853a-078083237d92
[2017-06-29 15:14:11,101: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 48117eab-f681-45b8-853a-078083237d92
[2017-06-29 15:14:11,156: INFO/job] Updating mtm volume_nodes for target fs1-OST0000
[2017-06-29 15:14:11,164: INFO/job_scheduler] run_next: 0 runnable jobs of (15 pending, 2 tasked)
[2017-06-29 15:14:11,241: INFO/job] StatefulObject.set_state MGS formatted->registered (intentional=True) 140206121213392
[2017-06-29 15:14:11,252: INFO/job] Job 110: StateChangeJob complete, setting state registered on MGS
[2017-06-29 15:14:11,252: INFO/job] Job 110: not_deleted=True
[2017-06-29 15:14:11,252: INFO/job_scheduler] Job 110: complete_job: Updating cache
[2017-06-29 15:14:11,252: INFO/job_scheduler] Job 110: locked_item 140206121213392 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-29 15:14:11,263: INFO/job_scheduler] Job 110 completing (errored=False, cancelled=False)
[2017-06-29 15:14:11,281: INFO/job_scheduler] Job 108 finished 5 steps successfully
[2017-06-29 15:14:11,288: INFO/job_scheduler] run_next: 1 runnable jobs of (15 pending, 1 tasked)
[2017-06-29 15:14:11,382: INFO/job_scheduler] Job 111: RunJobThread.run
[2017-06-29 15:14:11,382: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-06-29 15:14:11,403: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0e52df60-06b6-49d6-9ae3-830dde3f1575
[2017-06-29 15:14:11,403: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0e52df60-06b6-49d6-9ae3-830dde3f1575
[2017-06-29 15:14:11,637: INFO/job] StatefulObject.set_state fs1-OST0000 unformatted->formatted (intentional=True) 140206121076176
[2017-06-29 15:14:11,645: INFO/job] Job 108: StateChangeJob complete, setting state formatted on fs1-OST0000
[2017-06-29 15:14:11,645: INFO/job] Job 108: not_deleted=True
[2017-06-29 15:14:11,681: INFO/job_scheduler] Job 108: complete_job: Updating cache
[2017-06-29 15:14:11,681: INFO/job_scheduler] Job 108: locked_item 140206121076176 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-29 15:14:11,685: INFO/job_scheduler] Job 108: locked_item 140206121231824 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-06-29 15:14:11,686: INFO/job_scheduler] Job 108: locked_item 140206121231184 <class 'chroma_core.models.target.ManagedTargetMount'> False True
[2017-06-29 15:14:11,687: INFO/job_scheduler] Job 108 completing (errored=False, cancelled=False)
[2017-06-29 15:14:11,782: INFO/job_scheduler] run_next: 0 runnable jobs of (14 pending, 1 tasked)
[2017-06-29 15:14:12,446: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:12,446: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:12,446: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:12,446: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:12,446: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0e52df60-06b6-49d6-9ae3-830dde3f1575
[2017-06-29 15:14:12,446: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0e52df60-06b6-49d6-9ae3-830dde3f1575
[2017-06-29 15:14:12,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0e52df60-06b6-49d6-9ae3-830dde3f1575
[2017-06-29 15:14:12,467: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0e52df60-06b6-49d6-9ae3-830dde3f1575
[2017-06-29 15:14:12,468: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-06-29 15:14:12,484: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2d797cf1-4094-4a31-8c16-c07aab79227c
[2017-06-29 15:14:12,484: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2d797cf1-4094-4a31-8c16-c07aab79227c
[2017-06-29 15:14:13,567: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:13,567: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:13,567: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:13,567: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:13,567: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2d797cf1-4094-4a31-8c16-c07aab79227c
[2017-06-29 15:14:13,567: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2d797cf1-4094-4a31-8c16-c07aab79227c
[2017-06-29 15:14:13,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2d797cf1-4094-4a31-8c16-c07aab79227c
[2017-06-29 15:14:13,599: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2d797cf1-4094-4a31-8c16-c07aab79227c
[2017-06-29 15:14:13,601: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_target_store {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk1', 'device_type': 'linux', 'mount_point': u'/mnt/MGS', 'uuid': u'33feb403-57f9-45c4-9cea-cdf03b325f39', 'backfstype': 'ldiskfs'}
[2017-06-29 15:14:13,623: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 724c86db-6a2a-4078-8e3e-b5f36b079ffa
[2017-06-29 15:14:13,623: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 724c86db-6a2a-4078-8e3e-b5f36b079ffa
[2017-06-29 15:14:13,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:13,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:13,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:13,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:13,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 724c86db-6a2a-4078-8e3e-b5f36b079ffa
[2017-06-29 15:14:13,691: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 724c86db-6a2a-4078-8e3e-b5f36b079ffa
[2017-06-29 15:14:13,740: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 724c86db-6a2a-4078-8e3e-b5f36b079ffa
[2017-06-29 15:14:13,740: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 724c86db-6a2a-4078-8e3e-b5f36b079ffa
[2017-06-29 15:14:13,742: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_target_store {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk1', 'device_type': 'linux', 'mount_point': u'/mnt/MGS', 'uuid': u'33feb403-57f9-45c4-9cea-cdf03b325f39', 'backfstype': 'ldiskfs'}
[2017-06-29 15:14:13,754: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7542e324-8096-41e2-aded-5da5d50dbf30
[2017-06-29 15:14:13,754: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7542e324-8096-41e2-aded-5da5d50dbf30
[2017-06-29 15:14:13,816: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:13,816: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:13,816: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:13,816: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:13,817: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7542e324-8096-41e2-aded-5da5d50dbf30
[2017-06-29 15:14:13,817: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7542e324-8096-41e2-aded-5da5d50dbf30
[2017-06-29 15:14:13,827: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7542e324-8096-41e2-aded-5da5d50dbf30
[2017-06-29 15:14:13,827: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7542e324-8096-41e2-aded-5da5d50dbf30
[2017-06-29 15:14:13,829: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk1', 'mount_point': u'/mnt/MGS', 'uuid': u'33feb403-57f9-45c4-9cea-cdf03b325f39', 'ha_label': u'MGS_cbff7c', 'primary': True}
[2017-06-29 15:14:13,843: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1a530501-ef07-4ed7-ad82-af20aab8b4e7
[2017-06-29 15:14:13,843: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1a530501-ef07-4ed7-ad82-af20aab8b4e7
[2017-06-29 15:14:16,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:16,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:16,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:16,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:16,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1a530501-ef07-4ed7-ad82-af20aab8b4e7
[2017-06-29 15:14:16,939: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1a530501-ef07-4ed7-ad82-af20aab8b4e7
[2017-06-29 15:14:16,964: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1a530501-ef07-4ed7-ad82-af20aab8b4e7
[2017-06-29 15:14:16,964: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1a530501-ef07-4ed7-ad82-af20aab8b4e7
[2017-06-29 15:14:16,966: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk1', 'mount_point': u'/mnt/MGS', 'uuid': u'33feb403-57f9-45c4-9cea-cdf03b325f39', 'ha_label': u'MGS_cbff7c', 'primary': False}
[2017-06-29 15:14:16,978: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 781c8677-00bb-4681-ae7c-fb96f26dc351
[2017-06-29 15:14:16,978: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 781c8677-00bb-4681-ae7c-fb96f26dc351
[2017-06-29 15:14:18,057: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:18,057: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:18,058: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:18,058: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:18,058: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 781c8677-00bb-4681-ae7c-fb96f26dc351
[2017-06-29 15:14:18,058: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 781c8677-00bb-4681-ae7c-fb96f26dc351
[2017-06-29 15:14:18,095: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 781c8677-00bb-4681-ae7c-fb96f26dc351
[2017-06-29 15:14:18,095: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 781c8677-00bb-4681-ae7c-fb96f26dc351
[2017-06-29 15:14:18,095: INFO/job_scheduler] Job 111 finished 6 steps successfully
[2017-06-29 15:14:18,138: INFO/job] StatefulObject.set_state MGS registered->unmounted (intentional=True) 140206121213392
[2017-06-29 15:14:18,150: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on MGS at severity 30
[2017-06-29 15:14:18,172: INFO/job] Job 111: StateChangeJob complete, setting state unmounted on MGS
[2017-06-29 15:14:18,172: INFO/job] Job 111: not_deleted=True
[2017-06-29 15:14:18,172: INFO/job_scheduler] Job 111: complete_job: Updating cache
[2017-06-29 15:14:18,172: INFO/job_scheduler] Job 111: locked_item 140206121213392 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-29 15:14:18,175: INFO/job_scheduler] Job 111 completing (errored=False, cancelled=False)
[2017-06-29 15:14:18,220: INFO/job_scheduler] run_next: 1 runnable jobs of (14 pending, 0 tasked)
[2017-06-29 15:14:18,297: INFO/job] create_parameters: host: 'lotus-32vm7.lotus.hpdd.lab.intel.com' active_volume_node: 'TargetVolumeInfo(host=<ManagedHost: lotus-32vm7.lotus.hpdd.lab.intel.com>, path=u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk1', device_type='linux')'
[2017-06-29 15:14:18,315: INFO/job_scheduler] Job 112: RunJobThread.run
[2017-06-29 15:14:18,321: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk1', 'device_type': 'linux'}
[2017-06-29 15:14:18,360: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ce007530-750e-4a8d-b1bc-8a2737eeac18
[2017-06-29 15:14:18,360: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ce007530-750e-4a8d-b1bc-8a2737eeac18
[2017-06-29 15:14:18,419: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:18,419: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:18,419: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:18,419: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:18,419: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ce007530-750e-4a8d-b1bc-8a2737eeac18
[2017-06-29 15:14:18,419: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ce007530-750e-4a8d-b1bc-8a2737eeac18
[2017-06-29 15:14:18,429: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ce007530-750e-4a8d-b1bc-8a2737eeac18
[2017-06-29 15:14:18,429: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ce007530-750e-4a8d-b1bc-8a2737eeac18
[2017-06-29 15:14:18,430: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com import_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk1', 'pacemaker_ha_operation': False, 'device_type': 'linux', 'validate_importable': True}
[2017-06-29 15:14:18,455: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 57b42ac7-8ef8-4834-9877-fd260b7f1d1e
[2017-06-29 15:14:18,455: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 57b42ac7-8ef8-4834-9877-fd260b7f1d1e
[2017-06-29 15:14:18,546: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:18,546: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:18,546: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:18,546: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:18,546: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 57b42ac7-8ef8-4834-9877-fd260b7f1d1e
[2017-06-29 15:14:18,546: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 57b42ac7-8ef8-4834-9877-fd260b7f1d1e
[2017-06-29 15:14:18,575: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 57b42ac7-8ef8-4834-9877-fd260b7f1d1e
[2017-06-29 15:14:18,575: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 57b42ac7-8ef8-4834-9877-fd260b7f1d1e
[2017-06-29 15:14:18,576: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_target {'ha_label': u'MGS_cbff7c'}
[2017-06-29 15:14:18,606: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3eff1e59-a751-4752-b574-0895f4afe5ce
[2017-06-29 15:14:18,606: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3eff1e59-a751-4752-b574-0895f4afe5ce
[2017-06-29 15:14:21,512: INFO/job_scheduler] run_next: 0 runnable jobs of (13 pending, 1 tasked)
[2017-06-29 15:14:24,761: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:24,761: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:24,762: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:24,762: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:24,762: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3eff1e59-a751-4752-b574-0895f4afe5ce
[2017-06-29 15:14:24,762: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3eff1e59-a751-4752-b574-0895f4afe5ce
[2017-06-29 15:14:24,777: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3eff1e59-a751-4752-b574-0895f4afe5ce
[2017-06-29 15:14:24,777: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3eff1e59-a751-4752-b574-0895f4afe5ce
[2017-06-29 15:14:24,778: INFO/job_scheduler] Job 112 finished 1 steps successfully
[2017-06-29 15:14:24,886: INFO/job] StatefulObject.set_state MGS unmounted->mounted (intentional=True) 140206121213392
[2017-06-29 15:14:24,924: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 20
[2017-06-29 15:14:24,931: INFO/job] Job 112: StateChangeJob complete, setting state mounted on MGS
[2017-06-29 15:14:24,931: INFO/job] Job 112: not_deleted=True
[2017-06-29 15:14:24,931: INFO/job_scheduler] Job 112: complete_job: Updating cache
[2017-06-29 15:14:24,931: INFO/job_scheduler] Job 112: locked_item 140206121213392 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-29 15:14:24,935: INFO/job_scheduler] Job 112 completing (errored=False, cancelled=False)
[2017-06-29 15:14:24,994: INFO/job_scheduler] run_next: 3 runnable jobs of (13 pending, 0 tasked)
[2017-06-29 15:14:25,281: INFO/job_scheduler] Job 113: RunJobThread.run
[2017-06-29 15:14:25,283: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com register_target {'device_path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'mount_point': u'/mnt/fs1-MDT0002', 'backfstype': 'ldiskfs'}
[2017-06-29 15:14:25,313: INFO/job_scheduler] Job 114: RunJobThread.run
[2017-06-29 15:14:25,315: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com register_target {'device_path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk3', 'mount_point': u'/mnt/fs1-MDT0001', 'backfstype': 'ldiskfs'}
[2017-06-29 15:14:25,325: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0325e620-99cf-4aca-bb0c-585b5fba35d3
[2017-06-29 15:14:25,325: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 0325e620-99cf-4aca-bb0c-585b5fba35d3
[2017-06-29 15:14:25,346: INFO/job_scheduler] Job 115: RunJobThread.run
[2017-06-29 15:14:25,351: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 582a0b78-633c-43f9-9a54-1a8e49bc1c2c
[2017-06-29 15:14:25,351: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 582a0b78-633c-43f9-9a54-1a8e49bc1c2c
[2017-06-29 15:14:25,354: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com register_target {'device_path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', 'mount_point': u'/mnt/fs1-MDT0000', 'backfstype': 'ldiskfs'}
[2017-06-29 15:14:25,393: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5009f94f-5fce-4bce-b5d6-46e5ca9f82a1
[2017-06-29 15:14:25,393: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5009f94f-5fce-4bce-b5d6-46e5ca9f82a1
[2017-06-29 15:14:26,398: INFO/job_scheduler] notify: Dropping update of 1 (MGS) because it has been updated since
[2017-06-29 15:14:26,399: INFO/job_scheduler] run_next: 0 runnable jobs of (10 pending, 3 tasked)
[2017-06-29 15:14:29,476: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:29,476: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:29,477: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:29,477: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:29,477: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 582a0b78-633c-43f9-9a54-1a8e49bc1c2c
[2017-06-29 15:14:29,477: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 582a0b78-633c-43f9-9a54-1a8e49bc1c2c
[2017-06-29 15:14:29,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 582a0b78-633c-43f9-9a54-1a8e49bc1c2c
[2017-06-29 15:14:29,478: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 582a0b78-633c-43f9-9a54-1a8e49bc1c2c
[2017-06-29 15:14:29,480: INFO/job_scheduler] Job 114 finished 2 steps successfully
[2017-06-29 15:14:29,484: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:29,484: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:29,484: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:29,484: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:29,485: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0325e620-99cf-4aca-bb0c-585b5fba35d3
[2017-06-29 15:14:29,485: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 0325e620-99cf-4aca-bb0c-585b5fba35d3
[2017-06-29 15:14:29,497: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0325e620-99cf-4aca-bb0c-585b5fba35d3
[2017-06-29 15:14:29,497: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 0325e620-99cf-4aca-bb0c-585b5fba35d3
[2017-06-29 15:14:29,499: INFO/job_scheduler] Job 113 finished 2 steps successfully
[2017-06-29 15:14:29,588: INFO/job] StatefulObject.set_state fs1-MDT0001 formatted->registered (intentional=True) 140206121261328
[2017-06-29 15:14:29,593: INFO/job] Job 114: StateChangeJob complete, setting state registered on fs1-MDT0001
[2017-06-29 15:14:29,593: INFO/job] Job 114: not_deleted=True
[2017-06-29 15:14:29,593: INFO/job_scheduler] Job 114: complete_job: Updating cache
[2017-06-29 15:14:29,593: INFO/job_scheduler] Job 114: locked_item 140206121261328 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-29 15:14:29,596: INFO/job_scheduler] Job 114 completing (errored=False, cancelled=False)
[2017-06-29 15:14:29,663: INFO/job_scheduler] run_next: 1 runnable jobs of (10 pending, 2 tasked)
[2017-06-29 15:14:29,778: INFO/job_scheduler] Job 116: RunJobThread.run
[2017-06-29 15:14:29,778: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-06-29 15:14:29,809: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b49f5ae5-83e1-4273-a61b-62808623105a
[2017-06-29 15:14:29,809: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc b49f5ae5-83e1-4273-a61b-62808623105a
[2017-06-29 15:14:29,937: INFO/job] StatefulObject.set_state fs1-MDT0002 formatted->registered (intentional=True) 140206121260368
[2017-06-29 15:14:29,943: INFO/job] Job 113: StateChangeJob complete, setting state registered on fs1-MDT0002
[2017-06-29 15:14:29,943: INFO/job] Job 113: not_deleted=True
[2017-06-29 15:14:29,943: INFO/job_scheduler] Job 113: complete_job: Updating cache
[2017-06-29 15:14:29,943: INFO/job_scheduler] Job 113: locked_item 140206121260368 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-29 15:14:29,947: INFO/job_scheduler] Job 113 completing (errored=False, cancelled=False)
[2017-06-29 15:14:30,001: INFO/job_scheduler] run_next: 1 runnable jobs of (9 pending, 2 tasked)
[2017-06-29 15:14:30,106: INFO/job_scheduler] Job 117: RunJobThread.run
[2017-06-29 15:14:30,107: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-06-29 15:14:30,135: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3f4d345d-c649-4eb0-9ec2-7a96253a47bd
[2017-06-29 15:14:30,135: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 3f4d345d-c649-4eb0-9ec2-7a96253a47bd
[2017-06-29 15:14:30,499: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:30,499: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:30,500: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:30,500: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:30,502: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5009f94f-5fce-4bce-b5d6-46e5ca9f82a1
[2017-06-29 15:14:30,502: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5009f94f-5fce-4bce-b5d6-46e5ca9f82a1
[2017-06-29 15:14:30,517: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5009f94f-5fce-4bce-b5d6-46e5ca9f82a1
[2017-06-29 15:14:30,517: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5009f94f-5fce-4bce-b5d6-46e5ca9f82a1
[2017-06-29 15:14:30,518: INFO/job_scheduler] Job 115 finished 2 steps successfully
[2017-06-29 15:14:30,617: INFO/job] StatefulObject.set_state fs1-MDT0000 formatted->registered (intentional=True) 140206121077072
[2017-06-29 15:14:30,622: INFO/job] Job 115: StateChangeJob complete, setting state registered on fs1-MDT0000
[2017-06-29 15:14:30,622: INFO/job] Job 115: not_deleted=True
[2017-06-29 15:14:30,622: INFO/job_scheduler] Job 115: complete_job: Updating cache
[2017-06-29 15:14:30,622: INFO/job_scheduler] Job 115: locked_item 140206121077072 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-29 15:14:30,625: INFO/job_scheduler] Job 115 completing (errored=False, cancelled=False)
[2017-06-29 15:14:30,650: INFO/job_scheduler] run_next: 1 runnable jobs of (8 pending, 2 tasked)
[2017-06-29 15:14:30,753: INFO/job_scheduler] Job 118: RunJobThread.run
[2017-06-29 15:14:30,753: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-06-29 15:14:30,766: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7af8ca14-1d06-4d14-a104-c1caed78142f
[2017-06-29 15:14:30,766: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7af8ca14-1d06-4d14-a104-c1caed78142f
[2017-06-29 15:14:30,878: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:30,878: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:30,878: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:30,878: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:30,879: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b49f5ae5-83e1-4273-a61b-62808623105a
[2017-06-29 15:14:30,879: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc b49f5ae5-83e1-4273-a61b-62808623105a
[2017-06-29 15:14:30,879: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b49f5ae5-83e1-4273-a61b-62808623105a
[2017-06-29 15:14:30,879: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc b49f5ae5-83e1-4273-a61b-62808623105a
[2017-06-29 15:14:30,880: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-06-29 15:14:30,897: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1635cba0-de38-450f-a7ff-ae56d893dedc
[2017-06-29 15:14:30,897: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1635cba0-de38-450f-a7ff-ae56d893dedc
[2017-06-29 15:14:31,205: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:31,205: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:31,205: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:31,205: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:31,206: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3f4d345d-c649-4eb0-9ec2-7a96253a47bd
[2017-06-29 15:14:31,206: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 3f4d345d-c649-4eb0-9ec2-7a96253a47bd
[2017-06-29 15:14:31,253: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3f4d345d-c649-4eb0-9ec2-7a96253a47bd
[2017-06-29 15:14:31,253: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 3f4d345d-c649-4eb0-9ec2-7a96253a47bd
[2017-06-29 15:14:31,254: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-06-29 15:14:31,279: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1b752a44-711d-43be-8e42-79955994289a
[2017-06-29 15:14:31,279: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1b752a44-711d-43be-8e42-79955994289a
[2017-06-29 15:14:31,534: INFO/job_scheduler] run_next: 0 runnable jobs of (7 pending, 3 tasked)
[2017-06-29 15:14:31,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:31,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:31,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:31,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:31,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7af8ca14-1d06-4d14-a104-c1caed78142f
[2017-06-29 15:14:31,867: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7af8ca14-1d06-4d14-a104-c1caed78142f
[2017-06-29 15:14:31,893: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7af8ca14-1d06-4d14-a104-c1caed78142f
[2017-06-29 15:14:31,893: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7af8ca14-1d06-4d14-a104-c1caed78142f
[2017-06-29 15:14:31,894: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com open_firewall {'persist': True, 'description': 'lustre', 'proto': 'tcp', 'port': 988, 'address': None}
[2017-06-29 15:14:31,921: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9906349f-5a99-4169-9dd0-cf9c8d3d16d6
[2017-06-29 15:14:31,921: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9906349f-5a99-4169-9dd0-cf9c8d3d16d6
[2017-06-29 15:14:31,989: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:31,989: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:31,989: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:31,989: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:31,989: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1635cba0-de38-450f-a7ff-ae56d893dedc
[2017-06-29 15:14:31,989: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1635cba0-de38-450f-a7ff-ae56d893dedc
[2017-06-29 15:14:32,029: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1635cba0-de38-450f-a7ff-ae56d893dedc
[2017-06-29 15:14:32,029: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1635cba0-de38-450f-a7ff-ae56d893dedc
[2017-06-29 15:14:32,031: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_target_store {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk3', 'device_type': 'linux', 'mount_point': u'/mnt/fs1-MDT0001', 'uuid': u'8142d370-6246-4f2f-bf75-5d25031ce00c', 'backfstype': 'ldiskfs'}
[2017-06-29 15:14:32,045: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bc4b0da2-e051-4c19-8ece-e2fc11d32230
[2017-06-29 15:14:32,045: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bc4b0da2-e051-4c19-8ece-e2fc11d32230
[2017-06-29 15:14:32,109: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:32,109: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:32,109: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:32,109: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:32,109: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bc4b0da2-e051-4c19-8ece-e2fc11d32230
[2017-06-29 15:14:32,109: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bc4b0da2-e051-4c19-8ece-e2fc11d32230
[2017-06-29 15:14:32,110: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bc4b0da2-e051-4c19-8ece-e2fc11d32230
[2017-06-29 15:14:32,110: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bc4b0da2-e051-4c19-8ece-e2fc11d32230
[2017-06-29 15:14:32,112: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_target_store {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk3', 'device_type': 'linux', 'mount_point': u'/mnt/fs1-MDT0001', 'uuid': u'8142d370-6246-4f2f-bf75-5d25031ce00c', 'backfstype': 'ldiskfs'}
[2017-06-29 15:14:32,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 945670db-b2a8-4236-abb0-04a6f13483d4
[2017-06-29 15:14:32,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 945670db-b2a8-4236-abb0-04a6f13483d4
[2017-06-29 15:14:32,180: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:32,180: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:32,180: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:32,180: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:32,180: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 945670db-b2a8-4236-abb0-04a6f13483d4
[2017-06-29 15:14:32,180: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 945670db-b2a8-4236-abb0-04a6f13483d4
[2017-06-29 15:14:32,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 945670db-b2a8-4236-abb0-04a6f13483d4
[2017-06-29 15:14:32,189: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 945670db-b2a8-4236-abb0-04a6f13483d4
[2017-06-29 15:14:32,191: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk3', 'mount_point': u'/mnt/fs1-MDT0001', 'uuid': u'8142d370-6246-4f2f-bf75-5d25031ce00c', 'ha_label': u'fs1-MDT0001_4bd883', 'primary': True}
[2017-06-29 15:14:32,209: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc abdd76f9-2295-4e59-9661-7a82ad815795
[2017-06-29 15:14:32,209: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc abdd76f9-2295-4e59-9661-7a82ad815795
[2017-06-29 15:14:32,348: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:32,348: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:32,349: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:32,349: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:32,349: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1b752a44-711d-43be-8e42-79955994289a
[2017-06-29 15:14:32,349: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1b752a44-711d-43be-8e42-79955994289a
[2017-06-29 15:14:32,394: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1b752a44-711d-43be-8e42-79955994289a
[2017-06-29 15:14:32,394: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1b752a44-711d-43be-8e42-79955994289a
[2017-06-29 15:14:32,395: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_target_store {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux', 'mount_point': u'/mnt/fs1-MDT0002', 'uuid': u'cfe56c95-5a12-4fad-bb51-8d735995b336', 'backfstype': 'ldiskfs'}
[2017-06-29 15:14:32,408: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2263652e-0fe2-45b5-a0f1-b9d26f59c9de
[2017-06-29 15:14:32,408: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2263652e-0fe2-45b5-a0f1-b9d26f59c9de
[2017-06-29 15:14:32,487: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:32,487: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:32,487: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:32,487: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:32,487: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2263652e-0fe2-45b5-a0f1-b9d26f59c9de
[2017-06-29 15:14:32,487: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2263652e-0fe2-45b5-a0f1-b9d26f59c9de
[2017-06-29 15:14:32,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2263652e-0fe2-45b5-a0f1-b9d26f59c9de
[2017-06-29 15:14:32,523: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2263652e-0fe2-45b5-a0f1-b9d26f59c9de
[2017-06-29 15:14:32,525: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_target_store {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux', 'mount_point': u'/mnt/fs1-MDT0002', 'uuid': u'cfe56c95-5a12-4fad-bb51-8d735995b336', 'backfstype': 'ldiskfs'}
[2017-06-29 15:14:32,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8844b08f-938f-455c-a8e3-a4158955e1a1
[2017-06-29 15:14:32,538: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 8844b08f-938f-455c-a8e3-a4158955e1a1
[2017-06-29 15:14:32,656: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:32,656: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:32,657: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:32,657: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:32,657: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8844b08f-938f-455c-a8e3-a4158955e1a1
[2017-06-29 15:14:32,657: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 8844b08f-938f-455c-a8e3-a4158955e1a1
[2017-06-29 15:14:32,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8844b08f-938f-455c-a8e3-a4158955e1a1
[2017-06-29 15:14:32,702: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 8844b08f-938f-455c-a8e3-a4158955e1a1
[2017-06-29 15:14:32,704: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'mount_point': u'/mnt/fs1-MDT0002', 'uuid': u'cfe56c95-5a12-4fad-bb51-8d735995b336', 'ha_label': u'fs1-MDT0002_9dce20', 'primary': True}
[2017-06-29 15:14:32,717: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d116066c-747a-405c-8bc7-45858be62f48
[2017-06-29 15:14:32,717: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc d116066c-747a-405c-8bc7-45858be62f48
[2017-06-29 15:14:32,979: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:32,979: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:32,979: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:32,979: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:32,979: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9906349f-5a99-4169-9dd0-cf9c8d3d16d6
[2017-06-29 15:14:32,979: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9906349f-5a99-4169-9dd0-cf9c8d3d16d6
[2017-06-29 15:14:32,997: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9906349f-5a99-4169-9dd0-cf9c8d3d16d6
[2017-06-29 15:14:32,997: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9906349f-5a99-4169-9dd0-cf9c8d3d16d6
[2017-06-29 15:14:32,998: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_target_store {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', 'device_type': 'linux', 'mount_point': u'/mnt/fs1-MDT0000', 'uuid': u'6a81860e-0e1f-4efe-8b63-c7b5b3564289', 'backfstype': 'ldiskfs'}
[2017-06-29 15:14:33,052: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 940b280e-39ea-495b-a753-e4c36805f220
[2017-06-29 15:14:33,052: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 940b280e-39ea-495b-a753-e4c36805f220
[2017-06-29 15:14:33,110: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:33,110: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:33,110: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:33,110: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:33,111: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 940b280e-39ea-495b-a753-e4c36805f220
[2017-06-29 15:14:33,111: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 940b280e-39ea-495b-a753-e4c36805f220
[2017-06-29 15:14:33,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 940b280e-39ea-495b-a753-e4c36805f220
[2017-06-29 15:14:33,124: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 940b280e-39ea-495b-a753-e4c36805f220
[2017-06-29 15:14:33,126: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_target_store {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', 'device_type': 'linux', 'mount_point': u'/mnt/fs1-MDT0000', 'uuid': u'6a81860e-0e1f-4efe-8b63-c7b5b3564289', 'backfstype': 'ldiskfs'}
[2017-06-29 15:14:33,171: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7abc82bc-aa94-4dd3-a7ac-93085475f980
[2017-06-29 15:14:33,171: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 7abc82bc-aa94-4dd3-a7ac-93085475f980
[2017-06-29 15:14:33,254: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:33,254: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:33,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:33,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:33,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7abc82bc-aa94-4dd3-a7ac-93085475f980
[2017-06-29 15:14:33,255: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 7abc82bc-aa94-4dd3-a7ac-93085475f980
[2017-06-29 15:14:33,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7abc82bc-aa94-4dd3-a7ac-93085475f980
[2017-06-29 15:14:33,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 7abc82bc-aa94-4dd3-a7ac-93085475f980
[2017-06-29 15:14:33,272: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', 'mount_point': u'/mnt/fs1-MDT0000', 'uuid': u'6a81860e-0e1f-4efe-8b63-c7b5b3564289', 'ha_label': u'fs1-MDT0000_b3ca91', 'primary': True}
[2017-06-29 15:14:33,296: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ca0b0dda-a7a4-4b0b-a8fd-03ec602f7824
[2017-06-29 15:14:33,296: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ca0b0dda-a7a4-4b0b-a8fd-03ec602f7824
[2017-06-29 15:14:35,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:35,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:35,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:35,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:35,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc abdd76f9-2295-4e59-9661-7a82ad815795
[2017-06-29 15:14:35,270: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc abdd76f9-2295-4e59-9661-7a82ad815795
[2017-06-29 15:14:35,279: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc abdd76f9-2295-4e59-9661-7a82ad815795
[2017-06-29 15:14:35,279: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc abdd76f9-2295-4e59-9661-7a82ad815795
[2017-06-29 15:14:35,281: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk3', 'mount_point': u'/mnt/fs1-MDT0001', 'uuid': u'8142d370-6246-4f2f-bf75-5d25031ce00c', 'ha_label': u'fs1-MDT0001_4bd883', 'primary': False}
[2017-06-29 15:14:35,293: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bc21a64f-d7a1-4877-b617-4726a3883331
[2017-06-29 15:14:35,293: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc bc21a64f-d7a1-4877-b617-4726a3883331
[2017-06-29 15:14:35,813: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:35,813: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:35,813: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:35,813: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:35,813: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d116066c-747a-405c-8bc7-45858be62f48
[2017-06-29 15:14:35,813: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc d116066c-747a-405c-8bc7-45858be62f48
[2017-06-29 15:14:35,834: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d116066c-747a-405c-8bc7-45858be62f48
[2017-06-29 15:14:35,834: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc d116066c-747a-405c-8bc7-45858be62f48
[2017-06-29 15:14:35,836: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'mount_point': u'/mnt/fs1-MDT0002', 'uuid': u'cfe56c95-5a12-4fad-bb51-8d735995b336', 'ha_label': u'fs1-MDT0002_9dce20', 'primary': False}
[2017-06-29 15:14:35,858: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4f46d2bf-4466-4367-9a1c-db72a1f16881
[2017-06-29 15:14:35,858: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 4f46d2bf-4466-4367-9a1c-db72a1f16881
[2017-06-29 15:14:36,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:36,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:36,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:36,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:36,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bc21a64f-d7a1-4877-b617-4726a3883331
[2017-06-29 15:14:36,350: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc bc21a64f-d7a1-4877-b617-4726a3883331
[2017-06-29 15:14:36,362: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bc21a64f-d7a1-4877-b617-4726a3883331
[2017-06-29 15:14:36,362: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc bc21a64f-d7a1-4877-b617-4726a3883331
[2017-06-29 15:14:36,362: INFO/job_scheduler] Job 116 finished 6 steps successfully
[2017-06-29 15:14:36,419: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:36,419: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:36,420: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:36,420: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:36,420: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ca0b0dda-a7a4-4b0b-a8fd-03ec602f7824
[2017-06-29 15:14:36,420: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ca0b0dda-a7a4-4b0b-a8fd-03ec602f7824
[2017-06-29 15:14:36,428: INFO/job] StatefulObject.set_state fs1-MDT0001 registered->unmounted (intentional=True) 140206121261328
[2017-06-29 15:14:36,434: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on fs1-MDT0001 at severity 30
[2017-06-29 15:14:36,437: INFO/job] Job 116: StateChangeJob complete, setting state unmounted on fs1-MDT0001
[2017-06-29 15:14:36,437: INFO/job] Job 116: not_deleted=True
[2017-06-29 15:14:36,438: INFO/job_scheduler] Job 116: complete_job: Updating cache
[2017-06-29 15:14:36,438: INFO/job_scheduler] Job 116: locked_item 140206121261328 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-29 15:14:36,444: INFO/job_scheduler] Job 116 completing (errored=False, cancelled=False)
[2017-06-29 15:14:36,473: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ca0b0dda-a7a4-4b0b-a8fd-03ec602f7824
[2017-06-29 15:14:36,473: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ca0b0dda-a7a4-4b0b-a8fd-03ec602f7824
[2017-06-29 15:14:36,475: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com configure_target_ha {'device': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', 'mount_point': u'/mnt/fs1-MDT0000', 'uuid': u'6a81860e-0e1f-4efe-8b63-c7b5b3564289', 'ha_label': u'fs1-MDT0000_b3ca91', 'primary': False}
[2017-06-29 15:14:36,483: INFO/job_scheduler] run_next: 1 runnable jobs of (7 pending, 2 tasked)
[2017-06-29 15:14:36,510: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2b1ed91e-a2c7-411e-a296-5ca794e3eefc
[2017-06-29 15:14:36,510: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 2b1ed91e-a2c7-411e-a296-5ca794e3eefc
[2017-06-29 15:14:36,622: INFO/job] create_parameters: host: 'lotus-32vm7.lotus.hpdd.lab.intel.com' active_volume_node: 'TargetVolumeInfo(host=<ManagedHost: lotus-32vm7.lotus.hpdd.lab.intel.com>, path=u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk3', device_type='linux')'
[2017-06-29 15:14:36,639: INFO/job_scheduler] Job 119: RunJobThread.run
[2017-06-29 15:14:36,654: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk3', 'device_type': 'linux'}
[2017-06-29 15:14:36,677: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1b0a2192-f37c-4d0b-8570-2f9e79b52c37
[2017-06-29 15:14:36,677: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 1b0a2192-f37c-4d0b-8570-2f9e79b52c37
[2017-06-29 15:14:36,746: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:36,746: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:36,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:36,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:36,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1b0a2192-f37c-4d0b-8570-2f9e79b52c37
[2017-06-29 15:14:36,747: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 1b0a2192-f37c-4d0b-8570-2f9e79b52c37
[2017-06-29 15:14:36,750: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1b0a2192-f37c-4d0b-8570-2f9e79b52c37
[2017-06-29 15:14:36,750: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 1b0a2192-f37c-4d0b-8570-2f9e79b52c37
[2017-06-29 15:14:36,752: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com import_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk3', 'pacemaker_ha_operation': False, 'device_type': 'linux', 'validate_importable': True}
[2017-06-29 15:14:36,779: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9582ce6a-2a9a-45fb-a40f-b33782dbfc44
[2017-06-29 15:14:36,779: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 9582ce6a-2a9a-45fb-a40f-b33782dbfc44
[2017-06-29 15:14:36,869: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:36,869: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:36,869: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:36,869: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:36,869: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9582ce6a-2a9a-45fb-a40f-b33782dbfc44
[2017-06-29 15:14:36,869: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 9582ce6a-2a9a-45fb-a40f-b33782dbfc44
[2017-06-29 15:14:36,899: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9582ce6a-2a9a-45fb-a40f-b33782dbfc44
[2017-06-29 15:14:36,899: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 9582ce6a-2a9a-45fb-a40f-b33782dbfc44
[2017-06-29 15:14:36,899: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_target {'ha_label': u'fs1-MDT0001_4bd883'}
[2017-06-29 15:14:36,932: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ae7822c7-d399-4fc1-bfcd-ba75dd0e8dd2
[2017-06-29 15:14:36,932: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc ae7822c7-d399-4fc1-bfcd-ba75dd0e8dd2
[2017-06-29 15:14:36,944: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:36,944: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:36,944: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:36,944: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:36,944: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4f46d2bf-4466-4367-9a1c-db72a1f16881
[2017-06-29 15:14:36,944: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 4f46d2bf-4466-4367-9a1c-db72a1f16881
[2017-06-29 15:14:36,974: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4f46d2bf-4466-4367-9a1c-db72a1f16881
[2017-06-29 15:14:36,974: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 4f46d2bf-4466-4367-9a1c-db72a1f16881
[2017-06-29 15:14:36,974: INFO/job_scheduler] Job 117 finished 6 steps successfully
[2017-06-29 15:14:37,136: INFO/job] StatefulObject.set_state fs1-MDT0002 registered->unmounted (intentional=True) 140206121260368
[2017-06-29 15:14:37,142: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on fs1-MDT0002 at severity 30
[2017-06-29 15:14:37,155: INFO/job] Job 117: StateChangeJob complete, setting state unmounted on fs1-MDT0002
[2017-06-29 15:14:37,155: INFO/job] Job 117: not_deleted=True
[2017-06-29 15:14:37,155: INFO/job_scheduler] Job 117: complete_job: Updating cache
[2017-06-29 15:14:37,155: INFO/job_scheduler] Job 117: locked_item 140206121260368 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-29 15:14:37,157: INFO/job_scheduler] Job 117 completing (errored=False, cancelled=False)
[2017-06-29 15:14:37,234: INFO/job_scheduler] run_next: 1 runnable jobs of (6 pending, 2 tasked)
[2017-06-29 15:14:37,331: INFO/job] create_parameters: host: 'lotus-32vm7.lotus.hpdd.lab.intel.com' active_volume_node: 'TargetVolumeInfo(host=<ManagedHost: lotus-32vm7.lotus.hpdd.lab.intel.com>, path=u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', device_type='linux')'
[2017-06-29 15:14:37,362: INFO/job_scheduler] Job 121: RunJobThread.run
[2017-06-29 15:14:37,369: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'device_type': 'linux'}
[2017-06-29 15:14:37,409: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5e218749-9664-4456-a81e-f4c4bce1e4dd
[2017-06-29 15:14:37,409: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5e218749-9664-4456-a81e-f4c4bce1e4dd
[2017-06-29 15:14:37,490: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:37,490: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:37,490: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:37,490: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:37,490: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5e218749-9664-4456-a81e-f4c4bce1e4dd
[2017-06-29 15:14:37,490: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5e218749-9664-4456-a81e-f4c4bce1e4dd
[2017-06-29 15:14:37,533: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5e218749-9664-4456-a81e-f4c4bce1e4dd
[2017-06-29 15:14:37,533: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5e218749-9664-4456-a81e-f4c4bce1e4dd
[2017-06-29 15:14:37,534: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com import_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk4', 'pacemaker_ha_operation': False, 'device_type': 'linux', 'validate_importable': True}
[2017-06-29 15:14:37,564: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 782f752e-1011-4761-a9f9-35a3e24ff2dd
[2017-06-29 15:14:37,564: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 782f752e-1011-4761-a9f9-35a3e24ff2dd
[2017-06-29 15:14:37,613: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:37,613: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:37,613: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:37,613: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:37,613: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2b1ed91e-a2c7-411e-a296-5ca794e3eefc
[2017-06-29 15:14:37,613: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 2b1ed91e-a2c7-411e-a296-5ca794e3eefc
[2017-06-29 15:14:37,621: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:37,621: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:37,621: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:37,621: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:37,622: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 782f752e-1011-4761-a9f9-35a3e24ff2dd
[2017-06-29 15:14:37,622: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 782f752e-1011-4761-a9f9-35a3e24ff2dd
[2017-06-29 15:14:37,639: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 782f752e-1011-4761-a9f9-35a3e24ff2dd
[2017-06-29 15:14:37,639: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 782f752e-1011-4761-a9f9-35a3e24ff2dd
[2017-06-29 15:14:37,639: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_target {'ha_label': u'fs1-MDT0002_9dce20'}
[2017-06-29 15:14:37,647: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2b1ed91e-a2c7-411e-a296-5ca794e3eefc
[2017-06-29 15:14:37,647: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 2b1ed91e-a2c7-411e-a296-5ca794e3eefc
[2017-06-29 15:14:37,647: INFO/job_scheduler] Job 118 finished 6 steps successfully
[2017-06-29 15:14:37,683: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc abfdb910-88d9-4d76-a604-7b6e329857a7
[2017-06-29 15:14:37,683: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc abfdb910-88d9-4d76-a604-7b6e329857a7
[2017-06-29 15:14:37,719: INFO/job] StatefulObject.set_state fs1-MDT0000 registered->unmounted (intentional=True) 140206121077072
[2017-06-29 15:14:37,725: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on fs1-MDT0000 at severity 30
[2017-06-29 15:14:37,747: INFO/job] Job 118: StateChangeJob complete, setting state unmounted on fs1-MDT0000
[2017-06-29 15:14:37,747: INFO/job] Job 118: not_deleted=True
[2017-06-29 15:14:37,747: INFO/job_scheduler] Job 118: complete_job: Updating cache
[2017-06-29 15:14:37,747: INFO/job_scheduler] Job 118: locked_item 140206121077072 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-29 15:14:37,750: INFO/job_scheduler] Job 118 completing (errored=False, cancelled=False)
[2017-06-29 15:14:37,812: INFO/job_scheduler] run_next: 1 runnable jobs of (5 pending, 2 tasked)
[2017-06-29 15:14:37,911: INFO/job] create_parameters: host: 'lotus-32vm7.lotus.hpdd.lab.intel.com' active_volume_node: 'TargetVolumeInfo(host=<ManagedHost: lotus-32vm7.lotus.hpdd.lab.intel.com>, path=u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', device_type='linux')'
[2017-06-29 15:14:37,949: INFO/job_scheduler] Job 120: RunJobThread.run
[2017-06-29 15:14:37,965: INFO/job] invoke_agent on agent lotus-32vm8.lotus.hpdd.lab.intel.com export_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', 'device_type': 'linux'}
[2017-06-29 15:14:38,021: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5982f390-ff15-4874-b7fc-39b40a2221b4
[2017-06-29 15:14:38,021: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 5982f390-ff15-4874-b7fc-39b40a2221b4
[2017-06-29 15:14:38,118: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:38,118: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:38,119: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:38,119: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm8.lotus.hpdd.lab.intel.com/984b947d-b3bd-414b-9dc4-a21864ff1b81
[2017-06-29 15:14:38,119: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5982f390-ff15-4874-b7fc-39b40a2221b4
[2017-06-29 15:14:38,119: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 5982f390-ff15-4874-b7fc-39b40a2221b4
[2017-06-29 15:14:38,140: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5982f390-ff15-4874-b7fc-39b40a2221b4
[2017-06-29 15:14:38,140: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 5982f390-ff15-4874-b7fc-39b40a2221b4
[2017-06-29 15:14:38,142: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com import_target {'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk2', 'pacemaker_ha_operation': False, 'device_type': 'linux', 'validate_importable': True}
[2017-06-29 15:14:38,171: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 79df790e-951a-4555-9758-59b700172afc
[2017-06-29 15:14:38,171: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 79df790e-951a-4555-9758-59b700172afc
[2017-06-29 15:14:38,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:38,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:38,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:38,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:38,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 79df790e-951a-4555-9758-59b700172afc
[2017-06-29 15:14:38,243: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc 79df790e-951a-4555-9758-59b700172afc
[2017-06-29 15:14:38,249: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 79df790e-951a-4555-9758-59b700172afc
[2017-06-29 15:14:38,249: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 79df790e-951a-4555-9758-59b700172afc
[2017-06-29 15:14:38,249: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com start_target {'ha_label': u'fs1-MDT0000_b3ca91'}
[2017-06-29 15:14:38,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc aae95992-2d30-449e-a8d3-70dbde69c6a6
[2017-06-29 15:14:38,269: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc aae95992-2d30-449e-a8d3-70dbde69c6a6
[2017-06-29 15:14:41,534: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 3 tasked)
[2017-06-29 15:14:41,555: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 3 tasked)
[2017-06-29 15:14:41,570: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 3 tasked)
[2017-06-29 15:14:41,588: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 3 tasked)
[2017-06-29 15:14:43,196: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:43,196: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:43,258: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:43,258: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:43,258: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ae7822c7-d399-4fc1-bfcd-ba75dd0e8dd2
[2017-06-29 15:14:43,258: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc ae7822c7-d399-4fc1-bfcd-ba75dd0e8dd2
[2017-06-29 15:14:43,307: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ae7822c7-d399-4fc1-bfcd-ba75dd0e8dd2
[2017-06-29 15:14:43,307: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc ae7822c7-d399-4fc1-bfcd-ba75dd0e8dd2
[2017-06-29 15:14:43,307: INFO/job_scheduler] Job 119 finished 1 steps successfully
[2017-06-29 15:14:43,395: INFO/job] StatefulObject.set_state fs1-MDT0001 unmounted->mounted (intentional=True) 140206121261328
[2017-06-29 15:14:43,428: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 20
[2017-06-29 15:14:43,437: INFO/job] Job 119: StateChangeJob complete, setting state mounted on fs1-MDT0001
[2017-06-29 15:14:43,438: INFO/job] Job 119: not_deleted=True
[2017-06-29 15:14:43,438: INFO/job_scheduler] Job 119: complete_job: Updating cache
[2017-06-29 15:14:43,438: INFO/job_scheduler] Job 119: locked_item 140206121261328 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-29 15:14:43,441: INFO/job_scheduler] Job 119 completing (errored=False, cancelled=False)
[2017-06-29 15:14:43,494: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-29 15:14:45,547: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-29 15:14:45,606: INFO/job_scheduler] notify: Dropping update of 3 (fs1-MDT0001) because it has been updated since
[2017-06-29 15:14:45,607: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-29 15:14:45,639: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-29 15:14:45,670: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 2 tasked)
[2017-06-29 15:14:45,788: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:45,788: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:45,788: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:45,788: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:45,788: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc abfdb910-88d9-4d76-a604-7b6e329857a7
[2017-06-29 15:14:45,788: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc abfdb910-88d9-4d76-a604-7b6e329857a7
[2017-06-29 15:14:45,822: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc abfdb910-88d9-4d76-a604-7b6e329857a7
[2017-06-29 15:14:45,822: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc abfdb910-88d9-4d76-a604-7b6e329857a7
[2017-06-29 15:14:45,823: INFO/job_scheduler] Job 121 finished 1 steps successfully
[2017-06-29 15:14:46,056: INFO/job] StatefulObject.set_state fs1-MDT0002 unmounted->mounted (intentional=True) 140206121260368
[2017-06-29 15:14:46,121: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 20
[2017-06-29 15:14:46,146: INFO/job] Job 121: StateChangeJob complete, setting state mounted on fs1-MDT0002
[2017-06-29 15:14:46,146: INFO/job] Job 121: not_deleted=True
[2017-06-29 15:14:46,146: INFO/job_scheduler] Job 121: complete_job: Updating cache
[2017-06-29 15:14:46,146: INFO/job_scheduler] Job 121: locked_item 140206121260368 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-29 15:14:46,152: INFO/job_scheduler] Job 121 completing (errored=False, cancelled=False)
[2017-06-29 15:14:46,181: INFO/job_scheduler] run_next: 0 runnable jobs of (4 pending, 1 tasked)
[2017-06-29 15:14:46,452: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:46,452: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:46,452: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:46,452: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: good session lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:14:46,452: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc aae95992-2d30-449e-a8d3-70dbde69c6a6
[2017-06-29 15:14:46,452: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: completing rpc aae95992-2d30-449e-a8d3-70dbde69c6a6
[2017-06-29 15:14:46,492: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc aae95992-2d30-449e-a8d3-70dbde69c6a6
[2017-06-29 15:14:46,492: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc aae95992-2d30-449e-a8d3-70dbde69c6a6
[2017-06-29 15:14:46,493: INFO/job_scheduler] Job 120 finished 1 steps successfully
[2017-06-29 15:14:46,906: INFO/job] StatefulObject.set_state fs1-MDT0000 unmounted->mounted (intentional=True) 140206121077072
[2017-06-29 15:14:46,924: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 20
[2017-06-29 15:14:46,940: INFO/job] Job 120: StateChangeJob complete, setting state mounted on fs1-MDT0000
[2017-06-29 15:14:46,941: INFO/job] Job 120: not_deleted=True
[2017-06-29 15:14:46,942: INFO/job_scheduler] Job 120: complete_job: Updating cache
[2017-06-29 15:14:46,951: INFO/job_scheduler] Job 120: locked_item 140206121077072 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-29 15:14:46,956: INFO/job_scheduler] Job 120 completing (errored=False, cancelled=False)
[2017-06-29 15:14:46,991: INFO/job_scheduler] run_next: 1 runnable jobs of (4 pending, 0 tasked)
[2017-06-29 15:14:47,107: INFO/job_scheduler] Job 122: RunJobThread.run
[2017-06-29 15:14:47,108: INFO/job] invoke_agent on agent lotus-32vm7.lotus.hpdd.lab.intel.com register_target {'device_path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk5', 'mount_point': u'/mnt/fs1-OST0000', 'backfstype': 'ldiskfs'}
[2017-06-29 15:14:47,130: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 995c4d2f-8d06-4203-8d87-e0e2f9ec63c6
[2017-06-29 15:14:47,130: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: starting wait for rpc 995c4d2f-8d06-4203-8d87-e0e2f9ec63c6
[2017-06-29 15:14:52,635: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:14:52,638: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:14:52,640: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:14:52,641: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:14:52,790: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:14:52,807: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:14:52,830: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:14:52,847: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:18:52,217: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:18:52,241: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:18:52,260: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:18:52,279: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:18:53,784: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:18:53,794: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:18:53,833: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:18:53,881: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:18:56,540: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:18:56,577: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:18:57,080: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:18:57,112: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:29:04,633: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:29:04,661: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:29:04,684: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:29:04,710: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:29:06,671: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:29:06,691: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:29:06,710: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:29:06,728: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:29:08,440: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:29:08,454: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:29:09,881: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:29:09,894: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:36:27,973: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:36:28,025: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:36:28,068: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:36:28,118: INFO/job_scheduler] run_next: 0 runnable jobs of (3 pending, 1 tasked)
[2017-06-29 15:37:25,619: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:37:25,619: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/28cc0cc1-c131-4c2f-97b3-3c9552813811
[2017-06-29 15:37:25,619: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 28cc0cc1-c131-4c2f-97b3-3c9552813811 because session terminated
[2017-06-29 15:37:25,619: WARNING/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: aborting session 28cc0cc1-c131-4c2f-97b3-3c9552813811 because session terminated
[2017-06-29 15:37:25,644: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 995c4d2f-8d06-4203-8d87-e0e2f9ec63c6
[2017-06-29 15:37:25,644: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger._complete: completed wait for rpc 995c4d2f-8d06-4203-8d87-e0e2f9ec63c6
[2017-06-29 15:37:25,644: ERROR/job_scheduler] Job 122 step 0 encountered an agent error: Communications error with lotus-32vm7.lotus.hpdd.lab.intel.com because session terminated
[2017-06-29 15:37:25,735: INFO/job_scheduler] Job 122: complete_job: Updating cache
[2017-06-29 15:37:25,735: INFO/job_scheduler] Job 122: locked_item 140206121076176 <class 'chroma_core.models.target.ManagedTarget'> False True
[2017-06-29 15:37:25,738: INFO/job_scheduler] Job 122 completing (errored=True, cancelled=False)
[2017-06-29 15:37:25,801: INFO/job_scheduler] run_next: 1 runnable jobs of (3 pending, 0 tasked)
[2017-06-29 15:37:25,834: WARNING/job] DependOn not satisfied: fs1-OST0000 in state formatted, not one of ['registered'] (preferred registered)
[2017-06-29 15:37:25,848: WARNING/job] DependOn not satisfied: fs1-OST0000 in state formatted, not one of ['registered'] (preferred registered)
[2017-06-29 15:37:25,849: WARNING/job_scheduler] Job 123: cancelling because of failed dependency
[2017-06-29 15:37:25,849: INFO/job_scheduler] Job 123 completing (errored=False, cancelled=True)
[2017-06-29 15:37:25,888: INFO/job_scheduler] run_next: 1 runnable jobs of (2 pending, 0 tasked)
[2017-06-29 15:37:25,902: WARNING/job] DependOn not satisfied: fs1-OST0000 in state formatted, not one of ['unmounted'] (preferred unmounted)
[2017-06-29 15:37:25,924: WARNING/job] DependOn not satisfied: fs1-OST0000 in state formatted, not one of ['unmounted'] (preferred unmounted)
[2017-06-29 15:37:25,924: WARNING/job_scheduler] Job 124: cancelling because of failed dependency
[2017-06-29 15:37:25,924: INFO/job_scheduler] Job 124 completing (errored=False, cancelled=True)
[2017-06-29 15:37:25,967: INFO/job_scheduler] run_next: 1 runnable jobs of (1 pending, 0 tasked)
[2017-06-29 15:37:26,011: WARNING/job] DependOn not satisfied: fs1-OST0000 in state formatted, not one of ['mounted'] (preferred mounted)
[2017-06-29 15:37:26,047: WARNING/job] DependOn not satisfied: fs1-OST0000 in state formatted, not one of ['mounted'] (preferred mounted)
[2017-06-29 15:37:26,050: WARNING/job_scheduler] Job 125: cancelling because of failed dependency
[2017-06-29 15:37:26,050: INFO/job_scheduler] Job 125 completing (errored=False, cancelled=True)
[2017-06-29 15:37:26,109: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 15:39:15,821: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 15:39:15,885: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 15:39:15,955: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 15:39:16,006: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 15:39:20,726: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 15:39:20,755: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 15:42:51,371: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 4 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from True to False
[2017-06-29 15:42:51,377: INFO/job_scheduler] _notify: Set corosync_reported_up=False on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-4) and saved
[2017-06-29 15:42:51,390: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 15:49:28,074: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 15:49:28,101: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 15:49:28,132: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 15:49:28,165: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 15:49:32,449: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 15:49:32,468: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 15:59:38,484: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 15:59:38,509: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 15:59:38,536: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 15:59:38,570: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 15:59:44,087: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 15:59:44,108: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:09:49,849: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:09:49,891: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:09:49,919: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:09:49,954: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:09:55,771: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:09:55,785: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:20:01,421: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:20:01,538: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:20:01,578: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:20:01,606: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:20:07,572: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:20:07,591: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:30:12,670: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:30:12,744: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:30:12,825: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:30:12,882: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:30:19,217: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:30:19,236: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:37:10,446: INFO/job_scheduler] _notify: Updating .corosync_reported_up of item 4 (lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration) from False to True
[2017-06-29 16:37:10,451: INFO/job_scheduler] _notify: Set corosync_reported_up=True on lotus-32vm7.lotus.hpdd.lab.intel.com Corosync2 configuration (Corosync2Configuration-4) and saved
[2017-06-29 16:37:10,459: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:37:13,533: INFO/job_scheduler] _notify: Updating .state of item 2 (fs1-MDT0000) from mounted to unmounted
[2017-06-29 16:37:13,533: INFO/job] StatefulObject.set_state fs1-MDT0000 mounted->unmounted (intentional=False) 140206121266192
[2017-06-29 16:37:13,539: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on fs1-MDT0000 at severity 40
[2017-06-29 16:37:13,539: INFO/job_scheduler] _notify: Updating .active_mount_id of item 2 (fs1-MDT0000) from 3 to None
[2017-06-29 16:37:13,542: INFO/job_scheduler] _notify: Set active_mount_id=None on fs1-MDT0000 (ManagedTarget-2) and saved
[2017-06-29 16:37:13,569: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:37:13,580: INFO/job_scheduler] _notify: Updating .state of item 3 (fs1-MDT0001) from mounted to unmounted
[2017-06-29 16:37:13,580: INFO/job] StatefulObject.set_state fs1-MDT0001 mounted->unmounted (intentional=False) 140206121203920
[2017-06-29 16:37:13,586: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on fs1-MDT0001 at severity 40
[2017-06-29 16:37:13,586: INFO/job_scheduler] _notify: Updating .active_mount_id of item 3 (fs1-MDT0001) from 5 to None
[2017-06-29 16:37:13,598: INFO/job_scheduler] _notify: Set active_mount_id=None on fs1-MDT0001 (ManagedTarget-3) and saved
[2017-06-29 16:37:13,618: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:37:13,622: INFO/job_scheduler] _notify: Updating .state of item 4 (fs1-MDT0002) from mounted to unmounted
[2017-06-29 16:37:13,622: INFO/job] StatefulObject.set_state fs1-MDT0002 mounted->unmounted (intentional=False) 140206121268368
[2017-06-29 16:37:13,626: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on fs1-MDT0002 at severity 40
[2017-06-29 16:37:13,626: INFO/job_scheduler] _notify: Updating .active_mount_id of item 4 (fs1-MDT0002) from 7 to None
[2017-06-29 16:37:13,628: INFO/job_scheduler] _notify: Set active_mount_id=None on fs1-MDT0002 (ManagedTarget-4) and saved
[2017-06-29 16:37:13,654: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:37:13,665: INFO/job_scheduler] _notify: Updating .state of item 1 (MGS) from mounted to unmounted
[2017-06-29 16:37:13,666: INFO/job] StatefulObject.set_state MGS mounted->unmounted (intentional=False) 140206121266448
[2017-06-29 16:37:13,671: INFO/job] AlertState: Raised <class 'chroma_core.models.target.TargetOfflineAlert'> on MGS at severity 40
[2017-06-29 16:37:13,671: INFO/job_scheduler] _notify: Updating .active_mount_id of item 1 (MGS) from 1 to None
[2017-06-29 16:37:13,687: INFO/job_scheduler] _notify: Set active_mount_id=None on MGS (ManagedTarget-1) and saved
[2017-06-29 16:37:13,710: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:37:35,068: INFO/job_scheduler] _notify: Updating .boot_time of item 4 (lotus-32vm7.lotus.hpdd.lab.intel.com) from 2017-06-29 04:44:14+00:00 to 2017-06-29 09:36:52+00:00
[2017-06-29 16:37:35,080: INFO/job_scheduler] _notify: Set boot_time=2017-06-29 09:36:52+00:00 on lotus-32vm7.lotus.hpdd.lab.intel.com (ManagedHost-4) and saved
[2017-06-29 16:37:35,085: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:37:35,130: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/ed1396f5-5fb1-45e4-a9b0-5619ea16d965
[2017-06-29 16:37:35,130: INFO/chroma_core.services.job_scheduler.agent_rpc] AgentRpcMessenger.on_rx: lotus-32vm7.lotus.hpdd.lab.intel.com/ed1396f5-5fb1-45e4-a9b0-5619ea16d965
[2017-06-29 16:37:39,439: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:37:39,465: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:37:39,594: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:37:39,662: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:37:39,720: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:37:39,770: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:37:39,807: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:38:04,763: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:38:04,785: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:38:04,810: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:38:04,818: INFO/job_scheduler] _notify: Updating .state of item 1 (MGS) from unmounted to mounted
[2017-06-29 16:38:04,818: INFO/job] StatefulObject.set_state MGS unmounted->mounted (intentional=False) 140206121278544
[2017-06-29 16:38:04,845: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 20
[2017-06-29 16:38:04,851: INFO/job_scheduler] _notify: Updating .active_mount_id of item 1 (MGS) from None to 1
[2017-06-29 16:38:04,855: INFO/job_scheduler] _notify: Set active_mount_id=1 on MGS (ManagedTarget-1) and saved
[2017-06-29 16:38:04,871: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:38:07,598: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:38:07,647: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:38:07,673: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:38:07,698: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:38:50,803: INFO/job_scheduler] _notify: Updating .state of item 2 (fs1-MDT0000) from unmounted to mounted
[2017-06-29 16:38:50,803: INFO/job] StatefulObject.set_state fs1-MDT0000 unmounted->mounted (intentional=False) 140206255419536
[2017-06-29 16:38:50,825: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 20
[2017-06-29 16:38:50,831: INFO/job_scheduler] _notify: Updating .active_mount_id of item 2 (fs1-MDT0000) from None to 3
[2017-06-29 16:38:50,833: INFO/job_scheduler] _notify: Set active_mount_id=3 on fs1-MDT0000 (ManagedTarget-2) and saved
[2017-06-29 16:38:50,850: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:38:50,868: INFO/job_scheduler] _notify: Updating .state of item 3 (fs1-MDT0001) from unmounted to mounted
[2017-06-29 16:38:50,868: INFO/job] StatefulObject.set_state fs1-MDT0001 unmounted->mounted (intentional=False) 140206053702672
[2017-06-29 16:38:50,932: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 20
[2017-06-29 16:38:50,947: INFO/job_scheduler] _notify: Updating .active_mount_id of item 3 (fs1-MDT0001) from None to 5
[2017-06-29 16:38:50,951: INFO/job_scheduler] _notify: Set active_mount_id=5 on fs1-MDT0001 (ManagedTarget-3) and saved
[2017-06-29 16:38:50,990: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:38:51,106: INFO/job_scheduler] _notify: Updating .state of item 4 (fs1-MDT0002) from unmounted to mounted
[2017-06-29 16:38:51,106: INFO/job] StatefulObject.set_state fs1-MDT0002 unmounted->mounted (intentional=False) 140206053703504
[2017-06-29 16:38:51,156: INFO/job] AlertState: Raised <class 'chroma_core.models.event.AlertEvent'> on lotus-32vm7.lotus.hpdd.lab.intel.com at severity 20
[2017-06-29 16:38:51,171: INFO/job_scheduler] _notify: Updating .active_mount_id of item 4 (fs1-MDT0002) from None to 7
[2017-06-29 16:38:51,177: INFO/job_scheduler] _notify: Set active_mount_id=7 on fs1-MDT0002 (ManagedTarget-4) and saved
[2017-06-29 16:38:51,211: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:38:51,264: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:38:53,906: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:38:53,942: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:38:53,977: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:38:54,008: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:40:25,112: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:40:25,143: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:40:25,166: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:40:25,199: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:40:31,084: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:40:31,104: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:47:39,761: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:47:39,802: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:47:39,837: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:47:39,863: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:47:40,994: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:47:41,028: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:50:36,005: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:50:36,043: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:50:36,083: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:50:36,146: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:50:42,893: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:50:42,918: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:57:51,906: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:57:51,945: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:57:51,994: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:57:52,045: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:57:52,989: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 16:57:53,008: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 17:00:48,575: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 17:00:48,612: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 17:00:48,648: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 17:00:48,686: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 17:00:54,751: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 17:00:54,791: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 17:08:04,139: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 17:08:04,182: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 17:08:04,224: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 17:08:04,249: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 17:08:05,065: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 17:08:05,082: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 17:10:58,976: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 17:10:59,012: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 17:10:59,042: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 17:10:59,073: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 17:11:06,490: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 17:11:06,521: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 17:18:16,229: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 17:18:16,269: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 17:18:16,416: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 17:18:16,444: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 17:18:17,144: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 17:18:17,163: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 17:21:10,428: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 17:21:10,477: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 17:21:10,516: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 17:21:10,551: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 17:21:18,316: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)
[2017-06-29 17:21:18,359: INFO/job_scheduler] run_next: 0 runnable jobs of (0 pending, 0 tasked)